[{"data":1,"prerenderedAt":4089},["ShallowReactive",2],{"/blog/forward-deployed-engineer-vs-technical-consultant":3,"related-/blog/forward-deployed-engineer-vs-technical-consultant":877},{"id":4,"title":5,"authorId":6,"body":7,"category":826,"created":827,"description":828,"extension":829,"faqs":830,"featurePriority":851,"head":852,"hideCta":853,"landingPath":852,"meta":854,"navigation":865,"ogImage":852,"path":866,"robots":852,"schemaOrg":852,"seo":867,"sitemap":868,"stem":869,"tags":870,"__hash__":876},"blog/blog/1058.forward-deployed-engineer-vs-technical-consultant.md","Forward Deployed Engineer vs Technical Consultant","salome-koshadze",{"type":8,"value":9,"toc":808},"minimark",[10,18,131,134,139,173,178,181,192,198,207,211,214,251,254,261,265,268,277,286,289,294,297,306,309,313,316,319,322,325,333,340,355,358,362,365,494,497,511,514,524,529,532,621,624,628,631,635,638,641,644,648,651,655,675,678,682,685,688,713,716,720,723,727,733,737,740,760,763,766,770],[11,12,13,17],"p",{},[14,15,16],"strong",{},"Short answer:"," A forward deployed engineer (FDE) and a technical consultant can do almost identical work at a client site. They differ in the business model: an FDE works for a product vendor, where deployment helps sell and keep the software, and a technical consultant works in a services model, where the client pays for delivery, time, or both.",[19,20,21,36],"table",{},[22,23,24],"thead",{},[25,26,27,30,33],"tr",{},[28,29],"th",{},[28,31,32],{},"Forward deployed engineer",[28,34,35],{},"Technical consultant",[37,38,39,53,66,79,92,105,118],"tbody",{},[25,40,41,47,50],{},[42,43,44],"td",{},[14,45,46],{},"Employer",[42,48,49],{},"A product vendor",[42,51,52],{},"A services firm or a vendor's professional services team",[25,54,55,60,63],{},[42,56,57],{},[14,58,59],{},"What the customer mainly buys",[42,61,62],{},"Software; deployment work is bundled or priced to win the software deal",[42,64,65],{},"Delivery, priced by time, by project, or a mix",[25,67,68,73,76],{},[42,69,70],{},[14,71,72],{},"Measured on (in postings)",[42,74,75],{},"Production adoption, workflow impact, product feedback",[42,77,78],{},"Delivery against scope; firms track utilization",[25,80,81,86,89],{},[42,82,83],{},[14,84,85],{},"Deployment risk",[42,87,88],{},"Carried partly by the vendor, which needs the software to renew",[42,90,91],{},"Allocated through the contract, scope, and change requests",[25,93,94,99,102],{},[42,95,96],{},[14,97,98],{},"Where the lessons go",[42,100,101],{},"Into the vendor's product",[42,103,104],{},"Into the firm's playbooks",[25,106,107,112,115],{},[42,108,109],{},[14,110,111],{},"Travel (posting data)",[42,113,114],{},"Median stated maximum of 40% across 52 FDE postings with a figure",[42,116,117],{},"50% on average in the Deloitte postings we read",[25,119,120,125,128],{},[42,121,122],{},[14,123,124],{},"Pay (posting examples)",[42,126,127],{},"Base plus equity; some use on-target earnings",[42,129,130],{},"Base salary range",[11,132,133],{},"Contracts can mix both models, so each row shows the typical case.",[11,135,136],{},[14,137,138],{},"Key takeaways:",[140,141,142,149,155,161,167],"ul",{},[143,144,145,148],"li",{},[14,146,147],{},"Both roles can write production code."," FDE and consulting postings both describe taking systems from prototype to production.",[143,150,151,154],{},[14,152,153],{},"The commercial model is the clearest test."," Ask what the customer pays for and who carries the cost when a deployment stalls.",[143,156,157,160],{},[14,158,159],{},"Incentives follow the model."," FDE postings name adoption and product feedback as success measures; consulting firms report utilization (Accenture: 92%).",[143,162,163,166],{},[14,164,165],{},"Selected postings pay FDEs more."," OpenAI posts $185,000 to $300,000 plus equity for an FDE, Deloitte $86,700 to $170,900 for a technical consultant, with no matching by level or city.",[143,168,169,172],{},[14,170,171],{},"The title no longer decides it."," Accenture, McKinsey, Deloitte, and others now run, fund, or train FDE teams with the AI labs.",[174,175,177],"h2",{"id":176},"is-a-forward-deployed-engineer-just-a-consultant","Is a forward deployed engineer just a consultant?",[11,179,180],{},"Partly. The job content overlaps heavily, and the case for \"rebranded consulting\" is stronger than most FDE articles admit.",[11,182,183,184,191],{},"Andreessen Horowitz made that case directly. In a ",[185,186,190],"a",{"href":187,"rel":188},"https://a16z.com/forward-deployed-job-titles/",[189],"nofollow","January 2026 essay",", a16z's Tom Hollands described Palantir taking its solutions and integration engineers, \"typically among the lower-status roles in any engineering org,\" and giving them a new name. His verdict:",[193,194,195],"blockquote",{},[11,196,197],{},"\"The title gave the work prestige.\"",[11,199,200,201,206],{},"The standard rebuttal is that consultants advise and FDEs build. Gergely Orosz ",[185,202,205],{"href":203,"rel":204},"https://newsletter.pragmaticengineer.com/p/forward-deployed-engineers",[189],"writes in The Pragmatic Engineer"," that consultants \"make one-off recommendations, whereas FDEs generally work with customers, long-term.\" That fits a strategy consultant. A technical consultant, who ships code, fits it poorly.",[174,208,210],{"id":209},"do-both-roles-write-production-code","Do both roles write production code?",[11,212,213],{},"Yes. Consulting firms hire engineers to build and ship systems, and FDE postings ask for the same skills.",[140,215,216,228,240],{},[143,217,218,221,222,227],{},[14,219,220],{},"Thoughtworks",", a technology consultancy, posted a ",[185,223,226],{"href":224,"rel":225},"https://www.thoughtworks.com/careers/jobs/8200470?gh_jid=8200470",[189],"Senior AI Engineer role"," in Singapore in October 2026: \"Design, build and deploy GenAI applications using techniques such RAG, Agents, Multi Agent Systems etc., taking ideas from prototype to production.\"",[143,229,230,233,234,239],{},[14,231,232],{},"Deloitte's"," ",[185,235,238],{"href":236,"rel":237},"https://apply.deloitte.com/en_US/careers/JobDetail/Patient-Services-Salesforce-Technical-Consultant/354976",[189],"Patient Services Salesforce Technical Consultant"," posting asks for \"1+ full life-cycle Salesforce implementation.\"",[143,241,242,233,245,250],{},[14,243,244],{},"OpenAI's",[185,246,249],{"href":247,"rel":248},"https://jobs.ashbyhq.com/openai/967f94aa-1706-4dba-ac89-bfbc2c38b688",[189],"FDE posting"," asks candidates to \"Write and review production-grade code across frontend and backend.\"",[11,252,253],{},"Read side by side, the build work is hard to tell apart. Travel is close too: Deloitte asks for 50% on average and OpenAI for \"up to 50%.\"",[11,255,256,257,260],{},"Picture both at one client (an illustrative composite). A regional insurer wants an AI agent to handle first notice of loss: read the claim email, check the policy, open the case in the claims system. Two badges show up in the project room on Monday, one from a large services firm and one from the AI vendor whose model the agent runs on. Both write code, both spend much of the week on site, and both sit in the same meeting with the insurer's security team. ",[14,258,259],{},"One person's hours will likely appear on next month's invoice. The other's are folded into a software contract."," That split decides most of what follows.",[174,262,264],{"id":263},"what-does-the-customer-pay-for","What does the customer pay for?",[11,266,267],{},"The consultant's time is usually the product, and the FDE's time is usually a cost the vendor carries to sell and keep software customers. Public filings from the two archetypes show the gap.",[11,269,270,271,276],{},"Accenture's ",[185,272,275],{"href":273,"rel":274},"https://www.sec.gov/Archives/edgar/data/1467373/000146737325000217/acn-20250831.htm",[189],"annual report on Form 10-K"," tracks a number that product companies rarely mention: \"Utilization for fiscal 2025 was 92%.\" When people are the revenue, the share of their time billed to clients is a headline metric.",[11,278,279,280,285],{},"Palantir, which popularized the FDE role, reports the other shape. Its ",[185,281,284],{"href":282,"rel":283},"https://investors.palantir.com/files/2025%20FY%20PLTR%2010-K.pdf",[189],"2025 Form 10-K"," says the company earns revenue \"from the sale of subscriptions to access our software platforms,\" with professional services alongside. On early-stage work, the same filing reads: \"We conduct pilots and bootcamps with customers, generally at our own expense and without a guarantee of future returns.\"",[11,287,288],{},"Put the two filings side by side and the difference in model shows in every line:",[290,291],"nuxt-picture",{"src":292,":alt":293},"/blog/forward-deployed-engineer-vs-technical-consultant/1.svg","{\"Accenture versus Palantir financial metrics for the latest fiscal year\":{\" Accenture\":{\"$69\":{\"67 billion revenue, about 779,000 people, about $89,000 revenue per person, 31\":{\"9% gross margin, earned from consulting and managed services contracts\":{\" Palantir\":\"$4.48 billion revenue, 4,429 full-time employees, about $1.01 million revenue per person, 82% gross margin, earned from software subscriptions with services bundled.\"}}},\"provider\":\"none\",\"loading\":\"lazy\",\"width\":746,\"height\":678}}}",[11,295,296],{},"Both margins are GAAP gross margins from the companies' own filings, and revenue per person is our calculation. The comparison is still crude: Palantir's revenue comes from software sales, and most Accenture staff are not technical consultants. What it shows is the incentive each employer works under.",[11,298,299,300,305],{},"For the people in the project room, that incentive reaches the schedule. A firm running at 92% utilization needs its consultants billing again soon after an engagement ends, and the wider services market shows how hard that is. The 509 professional services organizations in SPI Research's 2026 benchmark, with over 245,000 consultants between them, averaged 66.4% billable utilization in 2025, down from 68.9% in 2024 and the lowest SPI has recorded, according to ",[185,301,304],{"href":302,"rel":303},"https://www.certinia.com/blog/analyzing-the-2026-spi-research-professional-services-maturity-benchmark-report/",[189],"Certinia's summary",".",[11,307,308],{},"So if the insurer's test environment is down for a week, the consulting firm can usually bill the time or raise a change request. Under a Palantir-style model, where pilots run at the vendor's expense, the vendor carries that week. Other FDE contracts split the risk differently. Who pays for the time also shapes what each engineer is asked to deliver.",[174,310,312],{"id":311},"what-is-each-role-measured-on","What is each role measured on?",[11,314,315],{},"FDE postings measure production adoption and product feedback, and consulting postings describe delivery of an agreed scope. The difference shows in the verbs.",[11,317,318],{},"OpenAI's FDE posting states: \"You will measure success through production adoption, measurable workflow impact, and eval-driven feedback that changes product and model roadmaps.\" It asks FDEs to \"Codify working patterns into tools, playbooks, or building blocks that others can use.\"",[11,320,321],{},"A Deloitte Engineering Technology Strategy and Transformation consultant posting lists \"Develop project plans, facilitate meetings with client and project leads, maintain project documentation\" and \"communicate recommendations.\" The engagement has a shape set by a statement of work, and Accenture's 10-K notes that \"many of our consulting contracts are less than 12 months in duration\" and can typically be ended \"with as little as 30 days' notice.\"",[11,323,324],{},"Condensed, the two scorecards look like this:",[290,326],{"src":327,":height":328,":width":329,"alt":330,"loading":331,"provider":332},"/blog/forward-deployed-engineer-vs-technical-consultant/2.svg","371","826","What each role is measured on. A forward deployed engineer is measured on production adoption and workflow impact, eval-driven feedback that shapes product roadmaps, and codifying patterns into reusable components. A technical consultant is measured on delivery of an agreed statement of work, project plans, documentation and recommendations, and time-boxed client engagements and handovers.","lazy","none",[11,334,335,336,339],{},"At the insurer, both badges leave the claims connector, its tests, and a handover with the insurer's IT team. The customer's code stays with the customer either way. What differs is where the ",[14,337,338],{},"lesson"," goes:",[341,342,343,349],"ol",{},[143,344,345,348],{},[14,346,347],{},"The consultant's lesson"," (\"agents keep failing at policy lookup in legacy claims systems\") goes into the firm's playbooks and the consultant's next engagement.",[143,350,351,354],{},[14,352,353],{},"The FDE's lesson"," is expected to reach the vendor's product team as a report, an eval, or a reusable component, so the next insurer needs less custom work.",[11,356,357],{},"Neither outcome is automatic. Consultants often write excellent handovers, and plenty of FDE field notes never reach a roadmap. Each employer rewards a different one, and pay follows the same split.",[174,359,361],{"id":360},"how-do-salary-travel-and-career-paths-compare","How do salary, travel, and career paths compare?",[11,363,364],{},"FDE postings list higher pay than the technical consultant postings we read, and travel is similar. These are selected postings from October 2026, not matched by level or city, so read the table as a market signal only.",[19,366,367,380],{},[22,368,369],{},[25,370,371,374,377],{},[28,372,373],{},"Role and source",[28,375,376],{},"Posted or reported pay",[28,378,379],{},"Travel",[37,381,382,395,409,422,435,455,468,481],{},[25,383,384,389,392],{},[42,385,386],{},[14,387,388],{},"OpenAI FDE, San Francisco",[42,390,391],{},"$185,000 to $300,000, plus equity",[42,393,394],{},"Up to 50%",[25,396,397,403,406],{},[42,398,399,402],{},[14,400,401],{},"Databricks Sr. FDE, U.S."," (October 7 snapshot)",[42,404,405],{},"$182,000 to $250,208 base",[42,407,408],{},"20%",[25,410,411,416,419],{},[42,412,413],{},[14,414,415],{},"Palantir FDSE, U.S.",[42,417,418],{},"$135,000 to $200,000 base",[42,420,421],{},"25% to 75%",[25,423,424,429,432],{},[42,425,426],{},[14,427,428],{},"Harvey Forward Deployed Finance Engineer",[42,430,431],{},"$254,600 to $350,000 on-target earnings, 75/25 split, plus equity",[42,433,434],{},"2 to 3+ trips a month",[25,436,437,449,452],{},[42,438,439,442,443,448],{},[14,440,441],{},"U.S. FDE median total compensation"," (",[185,444,447],{"href":445,"rel":446},"https://www.levels.fyi/t/software-engineer/title/forward-deployed-engineer/locations/united-states",[189],"Levels.fyi",")",[42,450,451],{},"$216,250 (self-reported, as of October 12, 2026; 25th to 75th percentile $178,750 to $290,000)",[42,453,454],{},"n/a",[25,456,457,462,465],{},[42,458,459],{},[14,460,461],{},"Deloitte Technical Consultant",[42,463,464],{},"$86,700 to $170,900",[42,466,467],{},"50%",[25,469,470,476,479],{},[42,471,472,475],{},[14,473,474],{},"Computer systems analysts, U.S. median"," (BLS, May 2025)",[42,477,478],{},"$105,850",[42,480,454],{},[25,482,483,488,491],{},[42,484,485,475],{},[14,486,487],{},"Management analysts, U.S. median",[42,489,490],{},"$101,860",[42,492,493],{},"\"Travel frequently\"",[11,495,496],{},"Two details matter more than the medians:",[140,498,499,505],{},[143,500,501,504],{},[14,502,503],{},"Equity."," OpenAI and Harvey list it on FDE roles, so an FDE at a growing vendor holds a stake in the product their field work improves. At a services firm, where revenue is billed time, upside tends to come through bonus and promotion.",[143,506,507,510],{},[14,508,509],{},"On-target earnings."," Harvey's 75/25 split between fixed and variable pay is a structure more common in sales than in engineering, and worth checking in any FDE offer.",[11,512,513],{},"Variable pay is one of several places where FDE roles drift toward sales. The sales engineer comparison shows where that line sits:",[515,516],"article-cheatsheet-card",{"cta":517,"description":518,"href":519,"image":520,"imageAlt":521,"label":522,"title":523},"Open the comparison","Pre-sale demos or post-sale production code compared: pay, travel, the hand-off on one deal, and when to hire each.","/compare/fde-vs-sales-engineer","/misc/fde-vs-sales-engineer-poster.png","FDE vs Sales Engineer comparison preview","","FDE vs Sales Engineer",[525,526,528],"h3",{"id":527},"how-much-do-forward-deployed-engineers-travel","How much do forward deployed engineers travel?",[11,530,531],{},"Most FDE postings mention travel, and the typical stated maximum sits between 25% and 50%. We counted it across 102 FDE postings collected on October 7, 2026 (method at the end of the article):",[19,533,534,544],{},[22,535,536],{},[25,537,538,541],{},[28,539,540],{},"Travel in FDE postings (n = 102)",[28,542,543],{},"Postings",[37,545,546,556,566,574,582,592,602,612],{},[25,547,548,553],{},[42,549,550],{},[14,551,552],{},"No travel mentioned",[42,554,555],{},"33",[25,557,558,563],{},[42,559,560],{},[14,561,562],{},"Travel mentioned",[42,564,565],{},"69",[25,567,568,571],{},[42,569,570],{},"Of those, with a percentage",[42,572,573],{},"52",[25,575,576,579],{},[42,577,578],{},"Of those, wording only (for example \"Expect regular travel for customer onsites\")",[42,580,581],{},"17",[25,583,584,589],{},[42,585,586],{},[14,587,588],{},"Median stated maximum",[42,590,591],{},"40%",[25,593,594,599],{},[42,595,596],{},[14,597,598],{},"25th to 75th percentile of stated maximum",[42,600,601],{},"25% to 50%",[25,603,604,609],{},[42,605,606],{},[14,607,608],{},"Stated maximum of 50% or more",[42,610,611],{},"20 (3 of them at 75%)",[25,613,614,619],{},[42,615,616],{},[14,617,618],{},"Stated maximum of 25% or less",[42,620,581],{},[11,622,623],{},"Some of the 33 silent postings will still involve travel, so the real share of FDE roles that travel is likely higher than 69 in 102. Where a figure is named, FDE travel lands close to Deloitte's 50%.",[525,625,627],{"id":626},"where-each-career-leads","Where each career leads",[11,629,630],{},"The ladders run in different directions. The consulting ladder is visible in Deloitte's own posting titles, from Consultant to Senior Consultant and on into manager roles, with progress tied to client delivery. The FDE path often bends back into the product: The Pragmatic Engineer reports that until about 2016 Palantir had more FDEs than regular software engineers, and that many moved into product engineering after Foundry launched.",[174,632,634],{"id":633},"can-a-technical-consultant-become-a-forward-deployed-engineer","Can a technical consultant become a forward deployed engineer?",[11,636,637],{},"Yes, and in 2026 it happened at company scale. The skills transfer well; the usual gaps are production ownership and code depth.",[11,639,640],{},"Client discovery, working inside someone else's systems, and getting through a security review are daily FDE tasks that consultants already know. OpenAI's posting asks for \"5+ years of engineering or technical deployment experience that includes customer-facing work,\" which many technical consultants meet. The adjustment is owning a system after go-live and writing for a product team that will read the field report.",[11,642,643],{},"The largest example of the move is a whole consulting firm changing sides, part of a wider shift in how consultancies sell this work.",[174,645,647],{"id":646},"why-do-consulting-firms-now-use-the-fde-label","Why do consulting firms now use the FDE label?",[11,649,650],{},"AI vendors need more deployment engineers than they employ, so the largest consultancies now supply, partner on, or train FDEs. \"Consultant or FDE\" has become a question about the engagement, whatever logo the employer has. Three moves in 2026 show it:",[290,652],{"src":653,":alt":654},"/blog/forward-deployed-engineer-vs-technical-consultant/3.svg","{\"Three 2026 moves by AI vendors and consulting firms\":{\" March 2026\":{\"Accenture and Microsoft launched a forward deployed engineering practice, with Microsoft supplying the platform and Accenture handling workflows and change management\":{\" May 2026\":{\"OpenAI launched a majority-owned deployment unit with more than $4 billion invested and agreed to acquire Tomoro, adding about 150 forward deployed engineers\":{\" October 2026\":\"Anthropic committed $100 million to train 10,000 Frontier Deployed Engineers by the end of 2027, with consulting-firm staff in the first cohorts.\"},\"provider\":\"none\",\"loading\":\"lazy\",\"width\":864,\"height\":484}}}}}",[11,656,657,658,663,664,669,670,305],{},"The names behind each move are the same firms that sell consulting hours. Accenture's practice with Microsoft ",[185,659,662],{"href":660,"rel":661},"https://newsroom.accenture.com/news/2026/accenture-launches-microsoft-forward-deployed-engineering-practice-to-help-organizations-scale-ai-across-the-enterprise",[189],"promises \"thousands of AI-skilled engineers\"",". Bain & Company, Capgemini, and McKinsey are among the 19 investors in ",[185,665,668],{"href":666,"rel":667},"https://openai.com/index/openai-launches-the-deployment-company/",[189],"OpenAI's Deployment Company",". Engineers from Accenture, Bain, Capgemini, Deloitte, and McKinsey sit in the first cohorts of ",[185,671,674],{"href":672,"rel":673},"https://www.anthropic.com/news/claude-frontier-academy",[189],"Anthropic's Frontier Academy",[11,676,677],{},"The Tomoro deal is the clearest case. Tomoro, \"an applied AI consulting and engineering firm,\" brings about 150 engineers into an FDE unit by changing their employer. Their skills do not change overnight; who pays them and what their work feeds into does. The reverse holds too: an \"FDE practice\" inside a consulting firm that bills clients for engineer time runs on a services model. That is a different contract, and a legitimate one.",[174,679,681],{"id":680},"does-the-blocker-tell-you-which-one-you-hired","Does the blocker tell you which one you hired?",[11,683,684],{},"No. In week two at the insurer, both badges hit the same wall: the claims workflow runs through a third-party policy portal the insurer licenses but does not own, with no API and every login tied to the adjuster's single sign-on and MFA prompt.",[11,686,687],{},"Both engineers may reach the same architecture decision:",[140,689,690,696,702],{},[143,691,692,695],{},[14,693,694],{},"An API"," where one exists.",[143,697,698,701],{},[14,699,700],{},"A headless workflow"," for unattended jobs with service credentials.",[143,703,704,707,708,712],{},[14,705,706],{},"The adjuster's own browser session",", with the agent acting inside it, when identity and human takeover matter. Webfuse, our product, supports this pattern; the ",[185,709,711],{"href":710},"/blog/how-to-connect-an-ai-agent-to-the-live-web","live web architectures guide"," compares all of them.",[11,714,715],{},"The commercial difference remains. The consultant delivers the workaround within an engagement; the FDE is also expected to turn it into a reusable product capability. That leaves the buyer's question: which of the two should you bring in?",[174,717,719],{"id":718},"which-one-should-you-hire","Which one should you hire?",[11,721,722],{},"Hire a technical consultant when you want scoped delivery you pay for directly, and an FDE when the vendor's product is the core of the system and you want the vendor to share deployment risk. Four questions sort most offers, from either side of the table:",[290,724],{"src":725,":alt":726},"/blog/forward-deployed-engineer-vs-technical-consultant/4.svg","{\"Four questions that separate a technical consultant from a forward deployed engineer\":{\" Who pays if the pilot fails\":{\"the client typically pays for hours, while the vendor often carries more risk\":{\" What the contract measures\":{\"delivered scope and billable hours, or production adoption and workflow impact\":{\" Where the lesson goes\":{\"into the firm's own playbooks, or potentially into the product\":{\" How the bonus is set\":\"utilization and billing targets, or often equity and outcome metrics.\"},\"provider\":\"none\",\"loading\":\"lazy\",\"width\":826,\"height\":576}}}}}}}",[11,728,729,732],{},[14,730,731],{},"Then ask for proof."," A vendor that says field work feeds the product should be able to name a feature that started at a customer site. If nobody can, the FDE you are hiring will likely work like a consultant, whatever the title says.",[174,734,736],{"id":735},"the-bottom-line","The bottom line",[11,738,739],{},"FDEs and technical consultants build the same kind of systems, often in the same project rooms, and the 2026 deals have blurred the employers too. What separates them is the contract around the work:",[140,741,742,748,754],{},[143,743,744,747],{},[14,745,746],{},"Who pays for the time:"," the client, hour by hour, or the vendor, out of software revenue.",[143,749,750,753],{},[14,751,752],{},"What counts as success:"," delivered scope, or adoption and a better product.",[143,755,756,759],{},[14,757,758],{},"Where the lesson goes:"," a firm's playbook, or the next release.",[11,761,762],{},"Pay currently favors the FDE side in the postings we read, and consultants have a clear path across. Before choosing either role, or hiring one, read the contract before the title.",[11,764,765],{},"Back in the project room, both badges can do excellent work on the same claims agent. After go-live, the consultant moves to the next engagement. The FDE, if the model works as intended, sends the policy-portal lesson home so the next insurer starts with it solved.",[174,767,769],{"id":768},"methodology-and-sources","Methodology and sources",[140,771,772,778,784,790,796,802],{},[143,773,774,777],{},[14,775,776],{},"Business model figures:"," Accenture's fiscal 2025 8-K earnings exhibit and 10-K (fiscal year ended August 31, 2025); Palantir's 2025 10-K (year ended December 31, 2025). Both gross margins are GAAP figures as reported. Revenue per person is our calculation.",[143,779,780,783],{},[14,781,782],{},"Postings:"," Deloitte, OpenAI, Palantir, Harvey, and Thoughtworks postings were read on October 12, 2026. Pay ranges are posted ranges, not salaries.",[143,785,786,789],{},[14,787,788],{},"Travel dataset:"," we pulled public job-board feeds from Greenhouse, Ashby, and Lever for 54 company career boards on October 7, 2026 (Databricks figures come from the same pull). A posting counted as FDE when the title contained \"forward deployed\" or \"FDE\" plus \"engineer\"; manager, lead, architect, strategist, intern, and pre-sales titles were excluded. Titles were deduplicated on company plus exact title, and companies with more than 13 FDE titles were randomly sampled down to 13. Travel was read only from sentences containing \"travel\"; where a posting gave several figures, we used the highest. The sample leans toward AI and developer-tool companies and is not a random sample of the market.",[143,791,792,795],{},[14,793,794],{},"Self-reported and survey data:"," Levels.fyi U.S. median and percentiles read on October 12, 2026; BLS medians from the May 2025 release; the SPI Research utilization figure via Certinia's and Deltek's summaries of the 2026 benchmark (the full report is paid).",[143,797,798,801],{},[14,799,800],{},"Analysis vs fact:"," the insurer scenario is an illustrative composite, not a client engagement. Statements on how the roles differ in general are our reading of the sources above.",[143,803,804,807],{},[14,805,806],{},"Disclosure:"," Webfuse is our product and supports one of the architecture options named in the blocker section.",{"title":522,"searchDepth":809,"depth":809,"links":810},2,[811,812,813,814,815,820,821,822,823,824,825],{"id":176,"depth":809,"text":177},{"id":209,"depth":809,"text":210},{"id":263,"depth":809,"text":264},{"id":311,"depth":809,"text":312},{"id":360,"depth":809,"text":361,"children":816},[817,819],{"id":527,"depth":818,"text":528},3,{"id":626,"depth":818,"text":627},{"id":633,"depth":809,"text":634},{"id":646,"depth":809,"text":647},{"id":680,"depth":809,"text":681},{"id":718,"depth":809,"text":719},{"id":735,"depth":809,"text":736},{"id":768,"depth":809,"text":769},"enterprise-ai","2026-10-12","Forward deployed engineer vs technical consultant compared with job postings and public filings: who pays for the engineer's hours, what each is measured on, pay, career paths, and why consulting firms now sell FDEs.","md",[831,834,837,840,843,846,848],{"question":832,"answer":833},"Is a forward deployed engineer the same as a consultant?","Not quite, though the daily work can be almost identical. Both may embed at a client site, write code, and sit through security reviews. The difference is commercial: an FDE works for a product vendor, where deployment supports software adoption and product learning, and a technical consultant works in a services model, where the client pays for scoped delivery, time, or both.",{"question":835,"answer":836},"Do forward deployed engineers and technical consultants both write code?","Yes. OpenAI's FDE postings ask for production-grade code across frontend and backend. Consulting firms post similar work: a Thoughtworks Senior AI Engineer role in October 2026 asked candidates to design, build, and deploy GenAI applications from prototype to production. The contract behind the code is what separates the two roles.",{"question":838,"answer":839},"Who pays for a forward deployed engineer?","Usually the vendor, which recovers the cost through software revenue. Palantir, which popularized the role, earns revenue mainly from software subscriptions and says in its 2025 annual report that it runs pilots and bootcamps mostly at its own expense. Contracts vary, and some vendors charge for deployment services, but the FDE's salary is mostly a cost of selling and keeping software customers.",{"question":841,"answer":842},"Are forward deployed engineers measured on billable hours?","Not in the postings we read. OpenAI's FDE posting measures success through production adoption, measurable workflow impact, and feedback that changes product and model roadmaps. Consulting firms track billed time closely: Accenture reported 92% utilization for fiscal 2025. An FDE role measured mainly on hours billed to a client is working in a services model, whatever the title says.",{"question":844,"answer":845},"Do forward deployed engineers earn more than technical consultants?","In the postings we compared, yes. OpenAI posted $185,000 to $300,000 plus equity for a San Francisco FDE, and Databricks posted $182,000 to $250,208 base for senior FDEs, against $86,700 to $170,900 for a Deloitte Technical Consultant. These are selected postings, not matched by level or city, so treat them as a market signal only.",{"question":634,"answer":847},"Yes, and the move has happened at scale. In May 2026 OpenAI agreed to acquire Tomoro, an applied AI consulting and engineering firm, to bring about 150 forward deployed engineers and deployment specialists into its new Deployment Company. Consultants already have client discovery and delivery experience. The usual gaps are production ownership and code depth: OpenAI asks for 5+ years of engineering or technical deployment experience that includes customer-facing work.",{"question":849,"answer":850},"Is an FDE at a consulting firm really an FDE?","It depends on the contract. Accenture and Microsoft launched a forward deployed engineering practice in March 2026, and Anthropic trains engineers from Accenture, Deloitte, McKinsey, and other partners as Frontier Deployed Engineers. If the client pays for the engineer's time and nothing flows back into a product, the work runs on a services model. The title alone does not settle it.",0,null,false,{"shortTitle":855,"relatedLinks":856},"FDE vs Technical Consultant",[857,861],{"text":858,"href":859,"description":860},"Forward Deployed Engineer vs Software Engineer (2026)","/blog/forward-deployed-engineer-vs-software-engineer","Pay, daily rhythm, customer time, and code ownership compared for FDEs and product engineers.",{"text":862,"href":863,"description":864},"Forward Deployed Engineering (FDE): The Complete 2026 Guide","/blog/forward-deployed-engineering-fde-guide","The operating loop, the artifacts an FDE leaves behind, and how to tell a real engagement from a rebranded services contract.",true,"/blog/forward-deployed-engineer-vs-technical-consultant",{"title":5,"description":828},{"loc":866},"blog/1058.forward-deployed-engineer-vs-technical-consultant",[871,872,873,874,875],"forward-deployed-engineering","fde","consulting","careers","ai-agents","9ge-KNv9iJogUa0s40_nTwFag5THQGAEKdnSG43Q1Vw",[878,3595],{"id":879,"title":880,"authorId":881,"body":882,"category":875,"created":3575,"description":3576,"extension":829,"faqs":852,"featurePriority":852,"head":852,"hideCta":865,"landingPath":852,"meta":3577,"navigation":865,"ogImage":852,"path":3585,"robots":852,"schemaOrg":852,"seo":3586,"sitemap":3587,"stem":3588,"tags":3589,"__hash__":3594},"blog/blog/1013.serialising-web-ui-state-for-llms-the-complete-guide.md","Serialising Web UI State for LLMs: The Complete Guide","thassilo-schiepanski",{"type":8,"value":883,"toc":3558},[884,890,913,917,920,924,927,930,939,943,947,954,973,977,980,983,987,994,998,1003,1006,1023,1027,1045,1049,1061,1075,1079,1093,1097,1100,1104,1122,1125,1129,1139,1144,1218,1221,1224,1229,1300,1305,1424,1429,1538,1542,1553,1558,1860,1865,2019,2028,2032,2043,2048,2052,2056,2059,2062,2065,2172,2176,2179,2183,2186,2189,2194,2418,2423,2675,2680,2756,2761,2768,2777,2781,2784,2787,2790,2795,2804,2808,3148,3168,3221,3225,3228,3260,3409,3413,3416,3419,3423,3434,3437,3448,3454,3554],[11,885,886,887,305],{},"LLMs have become a core component of AI web browsing agents. Previously, automating a web journey required formal models of both web browsing and web-based user interfaces (UIs). An LLM can serve as a plug-in reasoning backend: given serialised web UI state, it is expected to suggest input actions that progress a specified web browsing task. Serialised UI state is, more ergonomically, referred to as a ",[14,888,889],{},"snapshot",[11,891,892,893,898,899,904,905,898,909,912],{},"Snapshots and action suggestions are the two key artefacts along the agent loop. Action suggestions are elicited from the web agent LLM backend. Suggestions can be acted out via automated web browser environments, such as ",[185,894,897],{"href":895,"rel":896},"https://playwright.dev",[189],"Playwright"," or ",[185,900,903],{"href":901,"rel":902},"https://www.webfuse.com",[189],"Webfuse",", for example, ",[906,907,908],"em",{},"clicking",[906,910,911],{},"typing",". The snapshot, on the other hand, is an artefact created by the agent harness, that is, the individual application component between the LLM and the web browser.",[290,914],{"src":915,":alt":916},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/1.svg","{\"Figurative depiction of two key artefacts along the agent loop\":\"the snapshot towards the LLM and the input action towards the browser\",\"provider\":\"none\",\"loading\":\"lazy\",\"width\":460}",[11,918,919],{},"Creating snapshots for LLMs is a central challenge of developing efficient web agents. It is a trade-off between representation utility – how well models can interpret the such encoded UI – and LLM token costs. Snapshots are the major cost factor of an agentic web journey, accumulated over multiple rounds of the loop.",[174,921,923],{"id":922},"gui-snapshots","GUI Snapshots",[11,925,926],{},"Screenshots, i.e., GUI snapshots, resemble what humans see when viewing a web page at a given point in time. Early LLM-based web agents were primarily premised on GUI snapshots because of this evident equivalence. By design, the GUI surfaces only the UI context designated to be visible to the user.",[11,928,929],{},"The document object model (DOM) is a web browser's model of a web application UI's runtime state. With few exceptions, the DOM is the single source of truth regarding the UI. Most snapshot representations are thus, technically speaking, derivatives of a raw DOM snapshot. For the cases not reflected in the DOM, GUI snapshots are a first-class base representation.",[11,931,932,933,938],{},"Rendering a GUI snapshot from the DOM is, moreover, non-trivial; no client-side library manages to create GUI snapshots in reasonable quality or time – ",[185,934,937],{"href":935,"rel":936},"https://html2canvas.hertzen.com",[189],"HTML2Canvas"," demonstrates the limits.",[290,940],{"src":941,"alt":942,"loading":331},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/2.png","The same web-based user interface serialised via browser-native screenshot API and the client-side library HTML2Canvas showing significant geometric differences",[525,944,946],{"id":945},"vision-based-targeting","Vision-Based Targeting",[11,948,949,950,953],{},"The snapshot representation dictates how elements can be targeted in the live UI. Unambiguous targeting based on a visual representation must be either geometric (absolute), e.g., via point coordinates, or semantic (relative), e.g., via cues, such as ",[906,951,952],{},"\"Button with Text 'Shop'\"",". On its own, semantic targeting introduces a circular problem: which element has this text?",[11,955,956,957,960,961,964,965,968,969,305],{},"Consider the web browsing task: ",[906,958,959],{},"\"Navigate to the Apple online shop\"",". In combination with an action, in this case ",[906,962,963],{},"click",", the agent could target the ",[906,966,967],{},"Shop"," button via a point coordinate, e.g., ",[970,971,972],"code",{},"(725, 750)",[290,974],{"src":975,"alt":976,"loading":331},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/3.png","A GUI snapshot (screenshot) visualising point coordinate targeting",[11,978,979],{},"When the UI shifts between snapshot and action time, geometric targeting suggestions might have become obsolete.",[11,981,982],{},"Depending on the web browser environment, screenshots might be constrained to the GUI viewport. Scrolling is a secondary task that is imperative with viewport-contrained GUI snapshots.",[525,984,986],{"id":985},"the-problem-with-gui-snapshots","The Problem with GUI Snapshots",[11,988,989,990,993],{},"Image data is downsampled to make it economically viable as input to LLMs. It reduces LLM input tokens to a few thousand for real-world-representative full-page screenshots. A common downsampling approach is averaging ",[906,991,992],{},"n×n","-tiles of pixels, as known from image downscaling. Side effects are: pixel-precise targeting does not map back to the live UI, and small geometric features are not preserved (e.g., small labels). Downsampling effects can be accounted for under certain circumstances, e.g., by providing a snapshot format that downsamples to its identity. However, GUI aspect ratio and detail might be beyond the agent's control.",[290,995],{"src":996,"alt":997,"loading":331},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/4.png","A GUI snapshot (screenshot) compared to its figuratively downsampled counterpart",[999,1000,1002],"h4",{"id":1001},"grounded-gui-snapshots","Grounded GUI Snapshots",[11,1004,1005],{},"Geometric targeting remains unreliable with certain models and responsive UIs. To rely on semantic targeting instead, the agent must link relevant elements in the live UI to semantics in the screenshot. Relevance usually equates with actionability, but ultimately depends on the snapshot's purpose.",[11,1007,1008,1011,1012,1022],{},[906,1009,1010],{},"Set-of-Mark"," prompting is the go-to technique to create grounded GUI snapshots",[1013,1014,1015],"sup",{},[185,1016,1021],{"href":1017,"ariaDescribedBy":1018,"dataFootnoteRef":522,"id":1020},"#user-content-fn-1",[1019],"footnote-label","user-content-fnref-1","1",". In this context, relevant elements are parsed from the live DOM and indexed with a unique numeric identifier. Each indexed element in the live UI is then enhanced with a coloured bounding box associated with the respective identifier.",[290,1024],{"src":1025,"alt":1026,"loading":331},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/5.png","A GUI snapshot (screenshot) visualising grounded cue targeting",[11,1028,1029,1036,1037,305],{},[185,1030,1033],{"href":1031,"rel":1032},"https://browser-use.com",[189],[906,1034,1035],{},"Browser Use"," agents implement Set-of-Mark-grounded GUI snapshots",[1013,1038,1039],{},[185,1040,1044],{"href":1041,"ariaDescribedBy":1042,"dataFootnoteRef":522,"id":1043},"#user-content-fn-2",[1019],"user-content-fnref-2","2",[174,1046,1048],{"id":1047},"dom-snapshots","DOM Snapshots",[11,1050,1051,1052,1060],{},"The DOM is parsed from HTML. In its latest state it can be serialised back to HTML – including all runtime mutations. Research has supported LLMs' capabilities to interpret HTML, and even to navigate such encoded UIs",[1013,1053,1054],{},[185,1055,1059],{"href":1056,"ariaDescribedBy":1057,"dataFootnoteRef":522,"id":1058},"#user-content-fn-3",[1019],"user-content-fnref-3","3",". For that reason, DOM snapshots are a compelling alternative to GUI snapshots. LLMs excel at code generation and review, so these capabilities are no surprise.",[11,1062,1063,1064,1067,1068,1071,1072,1074],{},"Cross-origin ",[970,1065,1066],{},"iframe"," elements and ",[970,1069,1070],{},"canvas"," elements are exceptions to the DOM-is-source-of-truth assumption: ",[970,1073,1070],{},"-drawn GUI is not reflected in the DOM, nor are external, self-contained DOM subtrees if no explicit policy allows it. Structurally, both are leaves in the serialisable DOM tree.",[525,1076,1078],{"id":1077},"dom-based-targeting","DOM-Based Targeting",[11,1080,1081,1082,1085,1086,1089,1090,305],{},"Programmatically, DOM elements can be targeted by different relative means. CSS selectors abstract targeting with a standalone language. Whilst CSS selectors can resolve to more than one element in the DOM – compare semantics of the HTML ",[970,1083,1084],{},"class"," and ",[970,1087,1088],{},"id"," attributes – every element has at least one unique CSS selector, constructed from nth-child node paths, e.g., ",[970,1091,1092],{},"main .section:nth-child(2) > .row:nth-of-type(5) > form button",[290,1094],{"src":1095,"alt":1096,"loading":331},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/6.png","A DOM snapshot (HTML) visualising CSS selector targeting",[11,1098,1099],{},"A raw DOM snapshot is complete, beyond the GUI that renders in the spatially limited browser viewport. For a human-like visual browsing experience, scrolling can be abstracted by an indirect scroll-target-into-view mechanism.",[525,1101,1103],{"id":1102},"the-problem-with-dom-snapshots","The Problem with DOM Snapshots",[11,1105,1106,1107,1112,1113,1121],{},"DOM snapshots of real-world web pages may come at a size of several megabytes, which converts to millions of LLM input tokens – we estimate a raw snapshot of ",[185,1108,1111],{"href":1109,"rel":1110},"https://edition.cnn.com",[189],"cnn.com"," at around 1.5M tokens. At this size, DOM snapshots over-utilise the context window of many LLMs, or even exceed it (such as of ",[185,1114,1117,1120],{"href":1115,"rel":1116},"https://openai.com/index/gpt-4o-system-card/",[189],[906,1118,1119],{},"GPT-4o"," (OpenAI)","). Ultimately, raw DOM snapshots out-cost GUI snapshots.",[11,1123,1124],{},"Using DOM snapshots with LLM-based web agents requires size-reducing transformations, similar to image downsampling implemented behind LLM APIs.",[999,1126,1128],{"id":1127},"extracted-dom-snapshots","Extracted DOM Snapshots",[11,1130,1131,1132,1135,1136,1138],{},"The naïve approach to use information from the DOM as a snapshot is element extraction: relevant elements are copied from the live DOM to an initially empty virtual DOM. For small top-",[906,1133,1134],{},"k"," extractions – the ",[906,1137,1134],{}," highest-scoring elements – a snapshot of the virtual DOM comes at a fraction of the original DOM's size.",[11,1140,1141],{},[14,1142,1143],{},"Top-3 DOM Snapshot:",[1145,1146,1150],"pre",{"className":1147,"code":1148,"language":1149,"meta":522,"style":522},"language-html shiki shiki-themes github-dark github-dark","\u003Cbutton type=\"submit\">Submit\u003C/button>\n\u003Cbutton>More\u003C/button>\n\u003Ca href=\"/more\">Find out more\u003C/a>\n","html",[970,1151,1152,1184,1197],{"__ignoreMap":522},[1153,1154,1157,1161,1165,1169,1172,1176,1179,1181],"span",{"class":1155,"line":1156},"line",1,[1153,1158,1160],{"class":1159},"suv1-","\u003C",[1153,1162,1164],{"class":1163},"sxg3X","button",[1153,1166,1168],{"class":1167},"sFR8T"," type",[1153,1170,1171],{"class":1159},"=",[1153,1173,1175],{"class":1174},"s4wv1","\"submit\"",[1153,1177,1178],{"class":1159},">Submit\u003C/",[1153,1180,1164],{"class":1163},[1153,1182,1183],{"class":1159},">\n",[1153,1185,1186,1188,1190,1193,1195],{"class":1155,"line":809},[1153,1187,1160],{"class":1159},[1153,1189,1164],{"class":1163},[1153,1191,1192],{"class":1159},">More\u003C/",[1153,1194,1164],{"class":1163},[1153,1196,1183],{"class":1159},[1153,1198,1199,1201,1203,1206,1208,1211,1214,1216],{"class":1155,"line":818},[1153,1200,1160],{"class":1159},[1153,1202,185],{"class":1163},[1153,1204,1205],{"class":1167}," href",[1153,1207,1171],{"class":1159},[1153,1209,1210],{"class":1174},"\"/more\"",[1153,1212,1213],{"class":1159},">Find out more\u003C/",[1153,1215,185],{"class":1163},[1153,1217,1183],{"class":1159},[11,1219,1220],{},"If done manually, element extraction presupposes correct assumptions about which elements and element properties count as relevant. Alternatively, element relevance scoring can be outsourced to an LLM, at the cost of another inference step.",[11,1222,1223],{},"Extraction disposes of deep DOM hierarchy, which might be salient to the web agent LLM backend. From the below extracted DOM snapshot alone, the original content relationship is lost:",[11,1225,1226],{},[14,1227,1228],{},"Extracted DOM Snapshot:",[1145,1230,1232],{"className":1147,"code":1231,"language":1149,"meta":522,"style":522},"\u003Cstrong>MacBook Air\u003C/strong>\n\u003Cbutton>Add to cart\u003C/button>\n\u003Cp>Out of stock\u003C/p>\n\u003Ch2>MacBook Pro\u003C/h2>\n\u003Cbutton>Add to cart\u003C/button>\n",[970,1233,1234,1247,1260,1273,1287],{"__ignoreMap":522},[1153,1235,1236,1238,1240,1243,1245],{"class":1155,"line":1156},[1153,1237,1160],{"class":1159},[1153,1239,14],{"class":1163},[1153,1241,1242],{"class":1159},">MacBook Air\u003C/",[1153,1244,14],{"class":1163},[1153,1246,1183],{"class":1159},[1153,1248,1249,1251,1253,1256,1258],{"class":1155,"line":809},[1153,1250,1160],{"class":1159},[1153,1252,1164],{"class":1163},[1153,1254,1255],{"class":1159},">Add to cart\u003C/",[1153,1257,1164],{"class":1163},[1153,1259,1183],{"class":1159},[1153,1261,1262,1264,1266,1269,1271],{"class":1155,"line":818},[1153,1263,1160],{"class":1159},[1153,1265,11],{"class":1163},[1153,1267,1268],{"class":1159},">Out of stock\u003C/",[1153,1270,11],{"class":1163},[1153,1272,1183],{"class":1159},[1153,1274,1276,1278,1280,1283,1285],{"class":1155,"line":1275},4,[1153,1277,1160],{"class":1159},[1153,1279,174],{"class":1163},[1153,1281,1282],{"class":1159},">MacBook Pro\u003C/",[1153,1284,174],{"class":1163},[1153,1286,1183],{"class":1159},[1153,1288,1290,1292,1294,1296,1298],{"class":1155,"line":1289},5,[1153,1291,1160],{"class":1159},[1153,1293,1164],{"class":1163},[1153,1295,1255],{"class":1159},[1153,1297,1164],{"class":1163},[1153,1299,1183],{"class":1159},[11,1301,1302],{},[14,1303,1304],{},"Original DOM Snapshot – Candidate A:",[1145,1306,1308],{"className":1147,"code":1307,"language":1149,"meta":522,"style":522},"\u003Csection class=\"product\">\n  \u003Cstrong>MacBook Air\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n  \u003Cp>Out of stock\u003C/p>  👈\n\u003C/section>\n\u003Csection class=\"product\">\n  \u003Cstrong>MacBook Pro\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n\u003C/section>\n",[970,1309,1310,1327,1340,1352,1365,1374,1389,1402,1415],{"__ignoreMap":522},[1153,1311,1312,1314,1317,1320,1322,1325],{"class":1155,"line":1156},[1153,1313,1160],{"class":1159},[1153,1315,1316],{"class":1163},"section",[1153,1318,1319],{"class":1167}," class",[1153,1321,1171],{"class":1159},[1153,1323,1324],{"class":1174},"\"product\"",[1153,1326,1183],{"class":1159},[1153,1328,1329,1332,1334,1336,1338],{"class":1155,"line":809},[1153,1330,1331],{"class":1159},"  \u003C",[1153,1333,14],{"class":1163},[1153,1335,1242],{"class":1159},[1153,1337,14],{"class":1163},[1153,1339,1183],{"class":1159},[1153,1341,1342,1344,1346,1348,1350],{"class":1155,"line":818},[1153,1343,1331],{"class":1159},[1153,1345,1164],{"class":1163},[1153,1347,1255],{"class":1159},[1153,1349,1164],{"class":1163},[1153,1351,1183],{"class":1159},[1153,1353,1354,1356,1358,1360,1362],{"class":1155,"line":1275},[1153,1355,1331],{"class":1159},[1153,1357,11],{"class":1163},[1153,1359,1268],{"class":1159},[1153,1361,11],{"class":1163},[1153,1363,1364],{"class":1159},">  👈\n",[1153,1366,1367,1370,1372],{"class":1155,"line":1289},[1153,1368,1369],{"class":1159},"\u003C/",[1153,1371,1316],{"class":1163},[1153,1373,1183],{"class":1159},[1153,1375,1377,1379,1381,1383,1385,1387],{"class":1155,"line":1376},6,[1153,1378,1160],{"class":1159},[1153,1380,1316],{"class":1163},[1153,1382,1319],{"class":1167},[1153,1384,1171],{"class":1159},[1153,1386,1324],{"class":1174},[1153,1388,1183],{"class":1159},[1153,1390,1392,1394,1396,1398,1400],{"class":1155,"line":1391},7,[1153,1393,1331],{"class":1159},[1153,1395,14],{"class":1163},[1153,1397,1282],{"class":1159},[1153,1399,14],{"class":1163},[1153,1401,1183],{"class":1159},[1153,1403,1405,1407,1409,1411,1413],{"class":1155,"line":1404},8,[1153,1406,1331],{"class":1159},[1153,1408,1164],{"class":1163},[1153,1410,1255],{"class":1159},[1153,1412,1164],{"class":1163},[1153,1414,1183],{"class":1159},[1153,1416,1418,1420,1422],{"class":1155,"line":1417},9,[1153,1419,1369],{"class":1159},[1153,1421,1316],{"class":1163},[1153,1423,1183],{"class":1159},[11,1425,1426],{},[14,1427,1428],{},"Original DOM Snapshot – Candidate B:",[1145,1430,1432],{"className":1147,"code":1431,"language":1149,"meta":522,"style":522},"\u003Csection class=\"product\">\n  \u003Cstrong>MacBook Air\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n\u003C/section>\n\u003Csection class=\"product\">\n  \u003Cp>Out of stock\u003C/p>  👈\n  \u003Cstrong>MacBook Pro\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n\u003C/section>\n",[970,1433,1434,1448,1460,1472,1480,1494,1506,1518,1530],{"__ignoreMap":522},[1153,1435,1436,1438,1440,1442,1444,1446],{"class":1155,"line":1156},[1153,1437,1160],{"class":1159},[1153,1439,1316],{"class":1163},[1153,1441,1319],{"class":1167},[1153,1443,1171],{"class":1159},[1153,1445,1324],{"class":1174},[1153,1447,1183],{"class":1159},[1153,1449,1450,1452,1454,1456,1458],{"class":1155,"line":809},[1153,1451,1331],{"class":1159},[1153,1453,14],{"class":1163},[1153,1455,1242],{"class":1159},[1153,1457,14],{"class":1163},[1153,1459,1183],{"class":1159},[1153,1461,1462,1464,1466,1468,1470],{"class":1155,"line":818},[1153,1463,1331],{"class":1159},[1153,1465,1164],{"class":1163},[1153,1467,1255],{"class":1159},[1153,1469,1164],{"class":1163},[1153,1471,1183],{"class":1159},[1153,1473,1474,1476,1478],{"class":1155,"line":1275},[1153,1475,1369],{"class":1159},[1153,1477,1316],{"class":1163},[1153,1479,1183],{"class":1159},[1153,1481,1482,1484,1486,1488,1490,1492],{"class":1155,"line":1289},[1153,1483,1160],{"class":1159},[1153,1485,1316],{"class":1163},[1153,1487,1319],{"class":1167},[1153,1489,1171],{"class":1159},[1153,1491,1324],{"class":1174},[1153,1493,1183],{"class":1159},[1153,1495,1496,1498,1500,1502,1504],{"class":1155,"line":1376},[1153,1497,1331],{"class":1159},[1153,1499,11],{"class":1163},[1153,1501,1268],{"class":1159},[1153,1503,11],{"class":1163},[1153,1505,1364],{"class":1159},[1153,1507,1508,1510,1512,1514,1516],{"class":1155,"line":1391},[1153,1509,1331],{"class":1159},[1153,1511,14],{"class":1163},[1153,1513,1282],{"class":1159},[1153,1515,14],{"class":1163},[1153,1517,1183],{"class":1159},[1153,1519,1520,1522,1524,1526,1528],{"class":1155,"line":1404},[1153,1521,1331],{"class":1159},[1153,1523,1164],{"class":1163},[1153,1525,1255],{"class":1159},[1153,1527,1164],{"class":1163},[1153,1529,1183],{"class":1159},[1153,1531,1532,1534,1536],{"class":1155,"line":1417},[1153,1533,1369],{"class":1159},[1153,1535,1316],{"class":1163},[1153,1537,1183],{"class":1159},[999,1539,1541],{"id":1540},"downsampled-dom-snapshots","Downsampled DOM Snapshots",[11,1543,1544,1545,305],{},"Downsampling means reducing data point count whilst preserving the overall data shape and most of the relevant features. Picture an image being downsized: the depicted object remains recognisable to a large degree. The concept of image downsampling can be transferred to the DOM by locally 'averaging' DOM subtrees – trading fidelity for size",[1013,1546,1547],{},[185,1548,1552],{"href":1549,"ariaDescribedBy":1550,"dataFootnoteRef":522,"id":1551},"#user-content-fn-4",[1019],"user-content-fnref-4","4",[11,1554,1555],{},[14,1556,1557],{},"Original DOM Snapshot:",[1145,1559,1561],{"className":1147,"code":1560,"language":1149,"meta":522,"style":522},"\u003Csection class=\"container\" tabindex=\"3\" required=\"true\" type=\"example\">\n  \u003Cdiv class=\"mx-auto\" data-topic=\"products\" required=\"false\">\n    \u003Ch1>Our Pizza\u003C/h1>\n    \u003Cdiv>\n      \u003Cdiv class=\"shadow-lg\">\n        \u003Ch2>Margherita\u003C/h2>\n        \u003Cp>\n         A simple classic: mozzarella, tomatoes and basil.\n         An everyday choice!\n        \u003C/p>\n        \u003Cbutton type=\"button\">Add\u003C/button>\n      \u003C/div>\n      \u003Cdiv class=\"shadow-lg\">\n        \u003Ch2>Capricciosa\u003C/h2>\n        \u003Cp>\n          A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n          A true favourite!\n        \u003C/p>\n        \u003Cbutton type=\"button\">Add\u003C/button>\n      \u003C/div>\n    \u003C/div>\n  \u003C/div>\n\u003C/section>\n",[970,1562,1563,1601,1632,1647,1655,1671,1685,1693,1698,1703,1713,1734,1744,1759,1773,1782,1788,1794,1803,1822,1831,1841,1851],{"__ignoreMap":522},[1153,1564,1565,1567,1569,1571,1573,1576,1579,1581,1584,1587,1589,1592,1594,1596,1599],{"class":1155,"line":1156},[1153,1566,1160],{"class":1159},[1153,1568,1316],{"class":1163},[1153,1570,1319],{"class":1167},[1153,1572,1171],{"class":1159},[1153,1574,1575],{"class":1174},"\"container\"",[1153,1577,1578],{"class":1167}," tabindex",[1153,1580,1171],{"class":1159},[1153,1582,1583],{"class":1174},"\"3\"",[1153,1585,1586],{"class":1167}," required",[1153,1588,1171],{"class":1159},[1153,1590,1591],{"class":1174},"\"true\"",[1153,1593,1168],{"class":1167},[1153,1595,1171],{"class":1159},[1153,1597,1598],{"class":1174},"\"example\"",[1153,1600,1183],{"class":1159},[1153,1602,1603,1605,1608,1610,1612,1615,1618,1620,1623,1625,1627,1630],{"class":1155,"line":809},[1153,1604,1331],{"class":1159},[1153,1606,1607],{"class":1163},"div",[1153,1609,1319],{"class":1167},[1153,1611,1171],{"class":1159},[1153,1613,1614],{"class":1174},"\"mx-auto\"",[1153,1616,1617],{"class":1167}," data-topic",[1153,1619,1171],{"class":1159},[1153,1621,1622],{"class":1174},"\"products\"",[1153,1624,1586],{"class":1167},[1153,1626,1171],{"class":1159},[1153,1628,1629],{"class":1174},"\"false\"",[1153,1631,1183],{"class":1159},[1153,1633,1634,1637,1640,1643,1645],{"class":1155,"line":818},[1153,1635,1636],{"class":1159},"    \u003C",[1153,1638,1639],{"class":1163},"h1",[1153,1641,1642],{"class":1159},">Our Pizza\u003C/",[1153,1644,1639],{"class":1163},[1153,1646,1183],{"class":1159},[1153,1648,1649,1651,1653],{"class":1155,"line":1275},[1153,1650,1636],{"class":1159},[1153,1652,1607],{"class":1163},[1153,1654,1183],{"class":1159},[1153,1656,1657,1660,1662,1664,1666,1669],{"class":1155,"line":1289},[1153,1658,1659],{"class":1159},"      \u003C",[1153,1661,1607],{"class":1163},[1153,1663,1319],{"class":1167},[1153,1665,1171],{"class":1159},[1153,1667,1668],{"class":1174},"\"shadow-lg\"",[1153,1670,1183],{"class":1159},[1153,1672,1673,1676,1678,1681,1683],{"class":1155,"line":1376},[1153,1674,1675],{"class":1159},"        \u003C",[1153,1677,174],{"class":1163},[1153,1679,1680],{"class":1159},">Margherita\u003C/",[1153,1682,174],{"class":1163},[1153,1684,1183],{"class":1159},[1153,1686,1687,1689,1691],{"class":1155,"line":1391},[1153,1688,1675],{"class":1159},[1153,1690,11],{"class":1163},[1153,1692,1183],{"class":1159},[1153,1694,1695],{"class":1155,"line":1404},[1153,1696,1697],{"class":1159},"         A simple classic: mozzarella, tomatoes and basil.\n",[1153,1699,1700],{"class":1155,"line":1417},[1153,1701,1702],{"class":1159},"         An everyday choice!\n",[1153,1704,1706,1709,1711],{"class":1155,"line":1705},10,[1153,1707,1708],{"class":1159},"        \u003C/",[1153,1710,11],{"class":1163},[1153,1712,1183],{"class":1159},[1153,1714,1716,1718,1720,1722,1724,1727,1730,1732],{"class":1155,"line":1715},11,[1153,1717,1675],{"class":1159},[1153,1719,1164],{"class":1163},[1153,1721,1168],{"class":1167},[1153,1723,1171],{"class":1159},[1153,1725,1726],{"class":1174},"\"button\"",[1153,1728,1729],{"class":1159},">Add\u003C/",[1153,1731,1164],{"class":1163},[1153,1733,1183],{"class":1159},[1153,1735,1737,1740,1742],{"class":1155,"line":1736},12,[1153,1738,1739],{"class":1159},"      \u003C/",[1153,1741,1607],{"class":1163},[1153,1743,1183],{"class":1159},[1153,1745,1747,1749,1751,1753,1755,1757],{"class":1155,"line":1746},13,[1153,1748,1659],{"class":1159},[1153,1750,1607],{"class":1163},[1153,1752,1319],{"class":1167},[1153,1754,1171],{"class":1159},[1153,1756,1668],{"class":1174},[1153,1758,1183],{"class":1159},[1153,1760,1762,1764,1766,1769,1771],{"class":1155,"line":1761},14,[1153,1763,1675],{"class":1159},[1153,1765,174],{"class":1163},[1153,1767,1768],{"class":1159},">Capricciosa\u003C/",[1153,1770,174],{"class":1163},[1153,1772,1183],{"class":1159},[1153,1774,1776,1778,1780],{"class":1155,"line":1775},15,[1153,1777,1675],{"class":1159},[1153,1779,11],{"class":1163},[1153,1781,1183],{"class":1159},[1153,1783,1785],{"class":1155,"line":1784},16,[1153,1786,1787],{"class":1159},"          A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[1153,1789,1791],{"class":1155,"line":1790},17,[1153,1792,1793],{"class":1159},"          A true favourite!\n",[1153,1795,1797,1799,1801],{"class":1155,"line":1796},18,[1153,1798,1708],{"class":1159},[1153,1800,11],{"class":1163},[1153,1802,1183],{"class":1159},[1153,1804,1806,1808,1810,1812,1814,1816,1818,1820],{"class":1155,"line":1805},19,[1153,1807,1675],{"class":1159},[1153,1809,1164],{"class":1163},[1153,1811,1168],{"class":1167},[1153,1813,1171],{"class":1159},[1153,1815,1726],{"class":1174},[1153,1817,1729],{"class":1159},[1153,1819,1164],{"class":1163},[1153,1821,1183],{"class":1159},[1153,1823,1825,1827,1829],{"class":1155,"line":1824},20,[1153,1826,1739],{"class":1159},[1153,1828,1607],{"class":1163},[1153,1830,1183],{"class":1159},[1153,1832,1834,1837,1839],{"class":1155,"line":1833},21,[1153,1835,1836],{"class":1159},"    \u003C/",[1153,1838,1607],{"class":1163},[1153,1840,1183],{"class":1159},[1153,1842,1844,1847,1849],{"class":1155,"line":1843},22,[1153,1845,1846],{"class":1159},"  \u003C/",[1153,1848,1607],{"class":1163},[1153,1850,1183],{"class":1159},[1153,1852,1854,1856,1858],{"class":1155,"line":1853},23,[1153,1855,1369],{"class":1159},[1153,1857,1316],{"class":1163},[1153,1859,1183],{"class":1159},[11,1861,1862],{},[14,1863,1864],{},"Downsampled DOM Snapshot:",[1145,1866,1868],{"className":1147,"code":1867,"language":1149,"meta":522,"style":522},"\u003Csection class=\"container\" required=\"true\" type=\"example\">\n  \u003Cdiv class=\"mx-auto\" required=\"false\">\n    Our Pizza\n    \u003Cdiv>\n      Margherita\n      A simple classic: mozzarella, tomatoes and basil.\n      An everyday choice!\n      \u003Cbutton type=\"button\">Add\u003C/button>\n      Capricciosa\n      A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n      A true favourite!\n      \u003Cbutton type=\"button\">Add\u003C/button>\n    \u003C/div>\n  \u003C/div>\n\u003C/section>\n",[970,1869,1870,1896,1916,1921,1929,1934,1939,1944,1962,1967,1972,1977,1995,2003,2011],{"__ignoreMap":522},[1153,1871,1872,1874,1876,1878,1880,1882,1884,1886,1888,1890,1892,1894],{"class":1155,"line":1156},[1153,1873,1160],{"class":1159},[1153,1875,1316],{"class":1163},[1153,1877,1319],{"class":1167},[1153,1879,1171],{"class":1159},[1153,1881,1575],{"class":1174},[1153,1883,1586],{"class":1167},[1153,1885,1171],{"class":1159},[1153,1887,1591],{"class":1174},[1153,1889,1168],{"class":1167},[1153,1891,1171],{"class":1159},[1153,1893,1598],{"class":1174},[1153,1895,1183],{"class":1159},[1153,1897,1898,1900,1902,1904,1906,1908,1910,1912,1914],{"class":1155,"line":809},[1153,1899,1331],{"class":1159},[1153,1901,1607],{"class":1163},[1153,1903,1319],{"class":1167},[1153,1905,1171],{"class":1159},[1153,1907,1614],{"class":1174},[1153,1909,1586],{"class":1167},[1153,1911,1171],{"class":1159},[1153,1913,1629],{"class":1174},[1153,1915,1183],{"class":1159},[1153,1917,1918],{"class":1155,"line":818},[1153,1919,1920],{"class":1159},"    Our Pizza\n",[1153,1922,1923,1925,1927],{"class":1155,"line":1275},[1153,1924,1636],{"class":1159},[1153,1926,1607],{"class":1163},[1153,1928,1183],{"class":1159},[1153,1930,1931],{"class":1155,"line":1289},[1153,1932,1933],{"class":1159},"      Margherita\n",[1153,1935,1936],{"class":1155,"line":1376},[1153,1937,1938],{"class":1159},"      A simple classic: mozzarella, tomatoes and basil.\n",[1153,1940,1941],{"class":1155,"line":1391},[1153,1942,1943],{"class":1159},"      An everyday choice!\n",[1153,1945,1946,1948,1950,1952,1954,1956,1958,1960],{"class":1155,"line":1404},[1153,1947,1659],{"class":1159},[1153,1949,1164],{"class":1163},[1153,1951,1168],{"class":1167},[1153,1953,1171],{"class":1159},[1153,1955,1726],{"class":1174},[1153,1957,1729],{"class":1159},[1153,1959,1164],{"class":1163},[1153,1961,1183],{"class":1159},[1153,1963,1964],{"class":1155,"line":1417},[1153,1965,1966],{"class":1159},"      Capricciosa\n",[1153,1968,1969],{"class":1155,"line":1705},[1153,1970,1971],{"class":1159},"      A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[1153,1973,1974],{"class":1155,"line":1715},[1153,1975,1976],{"class":1159},"      A true favourite!\n",[1153,1978,1979,1981,1983,1985,1987,1989,1991,1993],{"class":1155,"line":1736},[1153,1980,1659],{"class":1159},[1153,1982,1164],{"class":1163},[1153,1984,1168],{"class":1167},[1153,1986,1171],{"class":1159},[1153,1988,1726],{"class":1174},[1153,1990,1729],{"class":1159},[1153,1992,1164],{"class":1163},[1153,1994,1183],{"class":1159},[1153,1996,1997,1999,2001],{"class":1155,"line":1746},[1153,1998,1836],{"class":1159},[1153,2000,1607],{"class":1163},[1153,2002,1183],{"class":1159},[1153,2004,2005,2007,2009],{"class":1155,"line":1761},[1153,2006,1846],{"class":1159},[1153,2008,1607],{"class":1163},[1153,2010,1183],{"class":1159},[1153,2012,2013,2015,2017],{"class":1155,"line":1775},[1153,2014,1369],{"class":1159},[1153,2016,1316],{"class":1163},[1153,2018,1183],{"class":1159},[11,2020,2021,2022,305],{},"When downsampling is defined over different types of nodes – for instance, elements, attributes, and text – it becomes a powerful technique to control DOM size in a snapshot pipeline. Adaptive implementations can moreover produce snapshots that fall below a hard token limit. Our evaluation revealed that moderate downsampling ratios comfortably fit the mean DOM snapshot of real-world-representative web pages within GPT-4o's context window – leaving headroom for an agent system prompt and a trace of reasoning and action history",[1013,2023,2024],{},[185,2025,1552],{"href":1549,"ariaDescribedBy":2026,"dataFootnoteRef":522,"id":2027},[1019],"user-content-fnref-4-2",[525,2029,2031],{"id":2030},"the-problem-with-dom-based-targeting","The Problem with DOM-Based Targeting",[11,2033,2034,2035,2038,2039,2042],{},"When DOM snapshots are transformation artefacts, relative CSS selectors like nth-child paths might not map back to the live DOM. There are two straightforward ways to account for breaking CSS selectors: (1) assigning unique IDs to relevant elements in the live UI that persist transformations (e.g., via a unique attribute ",[970,2036,2037],{},"UNIQUE-ID=\"27\"",", which can be selected via ",[970,2040,2041],{},"[UNIQUE-ID=\"27\"]","), and (2) maintaining a source map between elements in the live DOM and the transformed DOM. The latter approach does favourably not affect the snapshot directly.",[11,2044,2045],{},[14,2046,2047],{},"DOM Snapshot with Inlined IDs:",[290,2049],{"src":2050,"alt":2051,"loading":331},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/7.png","A DOM snapshot (HTML) visualising unique ID (attribute) targeting",[525,2053,2055],{"id":2054},"the-dom-based-snapshot-pipeline","The DOM-Based Snapshot Pipeline",[11,2057,2058],{},"LLMs are able to reason over HTML. Yet, HTML carries noise, such as obfuscated scripts or hidden subtrees. DOM snapshots should for that reason be seen as a base representation, rather than a target representation.",[11,2060,2061],{},"The concept of DOM downsampling can be utilised as a universal size control in a snapshot pipeline – ideally exposed as quality, the inverse property of downsampling. Moreover, the DOM can be pre-transformed by deterministic techniques, such as pruning hidden subtrees to surface only the tangible UI – a characteristic that is natural with GUI snapshots.",[11,2063,2064],{},"The pipeline approach traceably goes from the raw DOM to an abstract snapshot representation. For example:",[341,2066,2067,2090,2096,2154,2160,2166],{},[143,2068,2069,2072,2073,2078,2079,2082,2083,1085,2086,2089],{},[14,2070,2071],{},"Scope Subtree."," Scope in to a task-relevant subtree. An information retrieval task on ",[185,2074,2077],{"href":2075,"rel":2076},"https://www.wikipedia.org",[189],"wikipedia.org"," can scope into the ",[970,2080,2081],{},"main"," element – ",[970,2084,2085],{},"header",[970,2087,2088],{},"footer"," are irrelevant.",[143,2091,2092,2095],{},[14,2093,2094],{},"Clone."," Clone the live DOM.",[143,2097,2098,2101,2102],{},[14,2099,2100],{},"Filter."," Apply arbitrary filters to the cloned DOM (in memory). For example:\n",[140,2103,2104,2142,2148],{},[143,2105,2106,2109,2110],{},[14,2107,2108],{},"Prune Subtrees"," E.g.:\n",[140,2111,2112,2118,2124],{},[143,2113,2114,2117],{},[14,2115,2116],{},"Non-interactive Nodes",", e.g., hidden elements.",[143,2119,2120,2123],{},[14,2121,2122],{},"Irrelevant Nodes",", e.g., comment nodes.",[143,2125,2126,2129,2130,2132,2133,2135,2136,2139,2140,305],{},[14,2127,2128],{},"Redundant Nodes",", e.g., ",[970,2131,185],{}," in the ",[970,2134,2088],{}," that has a same-",[970,2137,2138],{},"href"," equivalent in the ",[970,2141,2085],{},[143,2143,2144,2147],{},[14,2145,2146],{},"Sanitise."," E.g., sensitive information from content-editables.",[143,2149,2150,2153],{},[14,2151,2152],{},"Clean."," E.g., remove stop words from text nodes.",[143,2155,2156,2159],{},[14,2157,2158],{},"Serialise."," Serialise the cloned DOM to HTML.",[143,2161,2162,2165],{},[14,2163,2164],{},"Downsample."," Downsample the HTML to control size.",[143,2167,2168,2171],{},[14,2169,2170],{},"Translate."," Translate the downsampled HTML to any DOM-based representation, e.g., the accessibility tree.",[290,2173],{"src":2174,"alt":2175,"loading":331,"provider":332},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/8.svg","Example of a DOM snapshot pipeline",[11,2177,2178],{},"A filtered DOM drastically increases stability of repeatedly run web journeys: if only a single button routes to a target page, the agent perceives an unambiguous action target element. The rule of thumb is: the less actionable options a snapshot transcribes, the better – given all relevant options are covered.",[999,2180,2182],{"id":2181},"arbitrary-snapshot-representations","Arbitrary Snapshot Representations",[11,2184,2185],{},"Any text-based snapshot representation can be derived from the single-source-of-truth DOM. Which representation is best highly depends on the given web browsing task. A high-level distinction is between (A) navigation and (B) information retrieval tasks. Navigation tasks do not require extensive text, but  actionable elements – if a snapshot omits a relevant button, the LLM is not able to use the encoded UI. Information retrieval tasks, on the other hand, require text, as the text contains topical information, whilst actionable elements do not matter.",[11,2187,2188],{},"Compare the following snapshots representations for the same UI:",[11,2190,2191],{},[14,2192,2193],{},"DOM Snapshot:",[1145,2195,2197],{"className":1147,"code":2196,"language":1149,"meta":522,"style":522},"\u003Csection class=\"container\" required=\"true\" type=\"example\">\n  \u003Cdiv class=\"mx-auto\" required=\"false\">\n    \u003Ch1>Our Pizza\u003C/h1>\n    \u003Cdiv>\n      \u003Ch2>Margherita\u003C/h2>\n      \u003Cp>\n        A simple classic: mozzarella, tomatoes and basil.\n        An everyday choice!\n      \u003C/p>\n      \u003Ca href=\"add?p=margherita\" data-uid=\"1\">Add\u003C/a>\n      \u003Ch2>Capricciosa\u003C/h2>\n      \u003Cp>\n        A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n        A true favourite!\n      \u003C/p>\n      \u003Ca href=\"add?p=capricciosa\" data-uid=\"2\">Add\u003C/a>\n    \u003C/div>\n  \u003C/div>\n\u003C/section>\n",[970,2198,2199,2225,2245,2257,2265,2277,2285,2290,2295,2303,2330,2342,2350,2355,2360,2368,2394,2402,2410],{"__ignoreMap":522},[1153,2200,2201,2203,2205,2207,2209,2211,2213,2215,2217,2219,2221,2223],{"class":1155,"line":1156},[1153,2202,1160],{"class":1159},[1153,2204,1316],{"class":1163},[1153,2206,1319],{"class":1167},[1153,2208,1171],{"class":1159},[1153,2210,1575],{"class":1174},[1153,2212,1586],{"class":1167},[1153,2214,1171],{"class":1159},[1153,2216,1591],{"class":1174},[1153,2218,1168],{"class":1167},[1153,2220,1171],{"class":1159},[1153,2222,1598],{"class":1174},[1153,2224,1183],{"class":1159},[1153,2226,2227,2229,2231,2233,2235,2237,2239,2241,2243],{"class":1155,"line":809},[1153,2228,1331],{"class":1159},[1153,2230,1607],{"class":1163},[1153,2232,1319],{"class":1167},[1153,2234,1171],{"class":1159},[1153,2236,1614],{"class":1174},[1153,2238,1586],{"class":1167},[1153,2240,1171],{"class":1159},[1153,2242,1629],{"class":1174},[1153,2244,1183],{"class":1159},[1153,2246,2247,2249,2251,2253,2255],{"class":1155,"line":818},[1153,2248,1636],{"class":1159},[1153,2250,1639],{"class":1163},[1153,2252,1642],{"class":1159},[1153,2254,1639],{"class":1163},[1153,2256,1183],{"class":1159},[1153,2258,2259,2261,2263],{"class":1155,"line":1275},[1153,2260,1636],{"class":1159},[1153,2262,1607],{"class":1163},[1153,2264,1183],{"class":1159},[1153,2266,2267,2269,2271,2273,2275],{"class":1155,"line":1289},[1153,2268,1659],{"class":1159},[1153,2270,174],{"class":1163},[1153,2272,1680],{"class":1159},[1153,2274,174],{"class":1163},[1153,2276,1183],{"class":1159},[1153,2278,2279,2281,2283],{"class":1155,"line":1376},[1153,2280,1659],{"class":1159},[1153,2282,11],{"class":1163},[1153,2284,1183],{"class":1159},[1153,2286,2287],{"class":1155,"line":1391},[1153,2288,2289],{"class":1159},"        A simple classic: mozzarella, tomatoes and basil.\n",[1153,2291,2292],{"class":1155,"line":1404},[1153,2293,2294],{"class":1159},"        An everyday choice!\n",[1153,2296,2297,2299,2301],{"class":1155,"line":1417},[1153,2298,1739],{"class":1159},[1153,2300,11],{"class":1163},[1153,2302,1183],{"class":1159},[1153,2304,2305,2307,2309,2311,2313,2316,2319,2321,2324,2326,2328],{"class":1155,"line":1705},[1153,2306,1659],{"class":1159},[1153,2308,185],{"class":1163},[1153,2310,1205],{"class":1167},[1153,2312,1171],{"class":1159},[1153,2314,2315],{"class":1174},"\"add?p=margherita\"",[1153,2317,2318],{"class":1167}," data-uid",[1153,2320,1171],{"class":1159},[1153,2322,2323],{"class":1174},"\"1\"",[1153,2325,1729],{"class":1159},[1153,2327,185],{"class":1163},[1153,2329,1183],{"class":1159},[1153,2331,2332,2334,2336,2338,2340],{"class":1155,"line":1715},[1153,2333,1659],{"class":1159},[1153,2335,174],{"class":1163},[1153,2337,1768],{"class":1159},[1153,2339,174],{"class":1163},[1153,2341,1183],{"class":1159},[1153,2343,2344,2346,2348],{"class":1155,"line":1736},[1153,2345,1659],{"class":1159},[1153,2347,11],{"class":1163},[1153,2349,1183],{"class":1159},[1153,2351,2352],{"class":1155,"line":1746},[1153,2353,2354],{"class":1159},"        A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[1153,2356,2357],{"class":1155,"line":1761},[1153,2358,2359],{"class":1159},"        A true favourite!\n",[1153,2361,2362,2364,2366],{"class":1155,"line":1775},[1153,2363,1739],{"class":1159},[1153,2365,11],{"class":1163},[1153,2367,1183],{"class":1159},[1153,2369,2370,2372,2374,2376,2378,2381,2383,2385,2388,2390,2392],{"class":1155,"line":1784},[1153,2371,1659],{"class":1159},[1153,2373,185],{"class":1163},[1153,2375,1205],{"class":1167},[1153,2377,1171],{"class":1159},[1153,2379,2380],{"class":1174},"\"add?p=capricciosa\"",[1153,2382,2318],{"class":1167},[1153,2384,1171],{"class":1159},[1153,2386,2387],{"class":1174},"\"2\"",[1153,2389,1729],{"class":1159},[1153,2391,185],{"class":1163},[1153,2393,1183],{"class":1159},[1153,2395,2396,2398,2400],{"class":1155,"line":1790},[1153,2397,1836],{"class":1159},[1153,2399,1607],{"class":1163},[1153,2401,1183],{"class":1159},[1153,2403,2404,2406,2408],{"class":1155,"line":1796},[1153,2405,1846],{"class":1159},[1153,2407,1607],{"class":1163},[1153,2409,1183],{"class":1159},[1153,2411,2412,2414,2416],{"class":1155,"line":1805},[1153,2413,1369],{"class":1159},[1153,2415,1316],{"class":1163},[1153,2417,1183],{"class":1159},[11,2419,2420],{},[14,2421,2422],{},"Accessibility Tree Snapshot:",[1145,2424,2428],{"className":2425,"code":2426,"language":2427,"meta":522,"style":522},"language-yaml shiki shiki-themes github-dark github-dark","- document:\n  children:\n    - section:\n      children:\n        - heading:\n          name: \"Our Pizza\"\n          level: 1\n        - heading:\n          name: \"Margherita\"\n          level: 2\n        - paragraph:\n          children:\n            - text: \"A simple classic: mozzarella, tomatoes and basil. An everyday choice!\"\n        - link:\n          name: \"Add\"\n          url: \"add?p=margherita\"\n          data-uid: 1\n        - heading:\n          name: \"Capricciosa\"\n          level: 2\n        - paragraph:\n          children:\n            - text: \"A rich taste: mozzarella, ham, mushrooms, artichokes and olives. A true favourite!\"\n        - link:\n          name: \"Add\"\n          url: \"add?p=capricciosa\"\n          data-uid: 2\n","yaml",[970,2429,2430,2441,2448,2457,2464,2474,2485,2496,2504,2513,2522,2531,2538,2551,2560,2569,2579,2588,2596,2605,2613,2621,2627,2638,2647,2656,2666],{"__ignoreMap":522},[1153,2431,2432,2435,2438],{"class":1155,"line":1156},[1153,2433,2434],{"class":1159},"- ",[1153,2436,2437],{"class":1163},"document",[1153,2439,2440],{"class":1159},":\n",[1153,2442,2443,2446],{"class":1155,"line":809},[1153,2444,2445],{"class":1163},"  children",[1153,2447,2440],{"class":1159},[1153,2449,2450,2453,2455],{"class":1155,"line":818},[1153,2451,2452],{"class":1159},"    - ",[1153,2454,1316],{"class":1163},[1153,2456,2440],{"class":1159},[1153,2458,2459,2462],{"class":1155,"line":1275},[1153,2460,2461],{"class":1163},"      children",[1153,2463,2440],{"class":1159},[1153,2465,2466,2469,2472],{"class":1155,"line":1289},[1153,2467,2468],{"class":1159},"        - ",[1153,2470,2471],{"class":1163},"heading",[1153,2473,2440],{"class":1159},[1153,2475,2476,2479,2482],{"class":1155,"line":1376},[1153,2477,2478],{"class":1163},"          name",[1153,2480,2481],{"class":1159},": ",[1153,2483,2484],{"class":1174},"\"Our Pizza\"\n",[1153,2486,2487,2490,2492],{"class":1155,"line":1391},[1153,2488,2489],{"class":1163},"          level",[1153,2491,2481],{"class":1159},[1153,2493,2495],{"class":2494},"s8ozJ","1\n",[1153,2497,2498,2500,2502],{"class":1155,"line":1404},[1153,2499,2468],{"class":1159},[1153,2501,2471],{"class":1163},[1153,2503,2440],{"class":1159},[1153,2505,2506,2508,2510],{"class":1155,"line":1417},[1153,2507,2478],{"class":1163},[1153,2509,2481],{"class":1159},[1153,2511,2512],{"class":1174},"\"Margherita\"\n",[1153,2514,2515,2517,2519],{"class":1155,"line":1705},[1153,2516,2489],{"class":1163},[1153,2518,2481],{"class":1159},[1153,2520,2521],{"class":2494},"2\n",[1153,2523,2524,2526,2529],{"class":1155,"line":1715},[1153,2525,2468],{"class":1159},[1153,2527,2528],{"class":1163},"paragraph",[1153,2530,2440],{"class":1159},[1153,2532,2533,2536],{"class":1155,"line":1736},[1153,2534,2535],{"class":1163},"          children",[1153,2537,2440],{"class":1159},[1153,2539,2540,2543,2546,2548],{"class":1155,"line":1746},[1153,2541,2542],{"class":1159},"            - ",[1153,2544,2545],{"class":1163},"text",[1153,2547,2481],{"class":1159},[1153,2549,2550],{"class":1174},"\"A simple classic: mozzarella, tomatoes and basil. An everyday choice!\"\n",[1153,2552,2553,2555,2558],{"class":1155,"line":1761},[1153,2554,2468],{"class":1159},[1153,2556,2557],{"class":1163},"link",[1153,2559,2440],{"class":1159},[1153,2561,2562,2564,2566],{"class":1155,"line":1775},[1153,2563,2478],{"class":1163},[1153,2565,2481],{"class":1159},[1153,2567,2568],{"class":1174},"\"Add\"\n",[1153,2570,2571,2574,2576],{"class":1155,"line":1784},[1153,2572,2573],{"class":1163},"          url",[1153,2575,2481],{"class":1159},[1153,2577,2578],{"class":1174},"\"add?p=margherita\"\n",[1153,2580,2581,2584,2586],{"class":1155,"line":1790},[1153,2582,2583],{"class":1163},"          data-uid",[1153,2585,2481],{"class":1159},[1153,2587,2495],{"class":2494},[1153,2589,2590,2592,2594],{"class":1155,"line":1796},[1153,2591,2468],{"class":1159},[1153,2593,2471],{"class":1163},[1153,2595,2440],{"class":1159},[1153,2597,2598,2600,2602],{"class":1155,"line":1805},[1153,2599,2478],{"class":1163},[1153,2601,2481],{"class":1159},[1153,2603,2604],{"class":1174},"\"Capricciosa\"\n",[1153,2606,2607,2609,2611],{"class":1155,"line":1824},[1153,2608,2489],{"class":1163},[1153,2610,2481],{"class":1159},[1153,2612,2521],{"class":2494},[1153,2614,2615,2617,2619],{"class":1155,"line":1833},[1153,2616,2468],{"class":1159},[1153,2618,2528],{"class":1163},[1153,2620,2440],{"class":1159},[1153,2622,2623,2625],{"class":1155,"line":1843},[1153,2624,2535],{"class":1163},[1153,2626,2440],{"class":1159},[1153,2628,2629,2631,2633,2635],{"class":1155,"line":1853},[1153,2630,2542],{"class":1159},[1153,2632,2545],{"class":1163},[1153,2634,2481],{"class":1159},[1153,2636,2637],{"class":1174},"\"A rich taste: mozzarella, ham, mushrooms, artichokes and olives. A true favourite!\"\n",[1153,2639,2641,2643,2645],{"class":1155,"line":2640},24,[1153,2642,2468],{"class":1159},[1153,2644,2557],{"class":1163},[1153,2646,2440],{"class":1159},[1153,2648,2650,2652,2654],{"class":1155,"line":2649},25,[1153,2651,2478],{"class":1163},[1153,2653,2481],{"class":1159},[1153,2655,2568],{"class":1174},[1153,2657,2659,2661,2663],{"class":1155,"line":2658},26,[1153,2660,2573],{"class":1163},[1153,2662,2481],{"class":1159},[1153,2664,2665],{"class":1174},"\"add?p=capricciosa\"\n",[1153,2667,2669,2671,2673],{"class":1155,"line":2668},27,[1153,2670,2583],{"class":1163},[1153,2672,2481],{"class":1159},[1153,2674,2521],{"class":2494},[11,2676,2677],{},[14,2678,2679],{},"Markdown Snapshot:",[1145,2681,2684],{"className":2682,"code":2683,"language":829,"meta":522,"style":522},"language-md shiki shiki-themes github-dark github-dark","# Our Pizza\n\n## Margherita\n\nA simple classic: mozzarella, tomatoes and basil.\nAn everyday choice!\n\n[Add](add?p=margherita)\n\n## Capricciosa\n\nA rich taste: mozzarella, ham, mushrooms, artichokes and olives.\nA true favourite!\n\n[Add](add?p=capricciosa)\n",[970,2685,2686,2691,2696,2701,2705,2710,2715,2719,2724,2728,2733,2737,2742,2747,2751],{"__ignoreMap":522},[1153,2687,2688],{"class":1155,"line":1156},[1153,2689,2690],{},"# Our Pizza\n",[1153,2692,2693],{"class":1155,"line":809},[1153,2694,2695],{"emptyLinePlaceholder":865},"\n",[1153,2697,2698],{"class":1155,"line":818},[1153,2699,2700],{},"## Margherita\n",[1153,2702,2703],{"class":1155,"line":1275},[1153,2704,2695],{"emptyLinePlaceholder":865},[1153,2706,2707],{"class":1155,"line":1289},[1153,2708,2709],{},"A simple classic: mozzarella, tomatoes and basil.\n",[1153,2711,2712],{"class":1155,"line":1376},[1153,2713,2714],{},"An everyday choice!\n",[1153,2716,2717],{"class":1155,"line":1391},[1153,2718,2695],{"emptyLinePlaceholder":865},[1153,2720,2721],{"class":1155,"line":1404},[1153,2722,2723],{},"[Add](add?p=margherita)\n",[1153,2725,2726],{"class":1155,"line":1417},[1153,2727,2695],{"emptyLinePlaceholder":865},[1153,2729,2730],{"class":1155,"line":1705},[1153,2731,2732],{},"## Capricciosa\n",[1153,2734,2735],{"class":1155,"line":1715},[1153,2736,2695],{"emptyLinePlaceholder":865},[1153,2738,2739],{"class":1155,"line":1736},[1153,2740,2741],{},"A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[1153,2743,2744],{"class":1155,"line":1746},[1153,2745,2746],{},"A true favourite!\n",[1153,2748,2749],{"class":1155,"line":1761},[1153,2750,2695],{"emptyLinePlaceholder":865},[1153,2752,2753],{"class":1155,"line":1775},[1153,2754,2755],{},"[Add](add?p=capricciosa)\n",[11,2757,2758],{},[14,2759,2760],{},"Custom Format Snapshot:",[1145,2762,2766],{"className":2763,"code":2765,"language":2545},[2764],"language-text","[0] H1 \"Our Pizza\"\n[1] H2 \"Margherita\"\n[2] P \"A simple classic: mozzarella, tomatoes and basil.\\nAn everyday choice!\"\n[3] A \"Add\" \"add?p=margherita\"\n[4] H2 \"Capricciosa\"\n[5] P \"A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\\nA true...\"\n[6] A \"Add\" \"add?p=capricciosa\"\n",[970,2767,2765],{"__ignoreMap":522},[11,2769,2770,2771,305],{},"Browser Use agents implement hybrid snapshots: the grounded GUI snapshot is a supplement to a custom-format indexed-list DOM snapshot, similar to the above example. Evaluations show that the indexed-list snapshot representation generally conveys most of the salient UI features for the web agent LLM backend",[1013,2772,2773],{},[185,2774,1552],{"href":1549,"ariaDescribedBy":2775,"dataFootnoteRef":522,"id":2776},[1019],"user-content-fnref-4-3",[174,2778,2780],{"id":2779},"non-idiomatic-application-designs","Non-Idiomatic Application Designs",[11,2782,2783],{},"An actionable element might lack affordance – not communicating its actionability through its bare appearance. That is bad UI/UX design practice. The concept extends to the DOM through HTML semantics, which communicate the general purpose of an element to web browsers and developers rather than users. Generalising to web agent LLM backends, the concept of affordance aligns between the GUI and the DOM – the model is a UI representation-agnostic user.",[11,2785,2786],{},"Missing affordances should be rectified in a snapshot wherever they can be discovered. Discoverability is another circular problem: what is an indirect affordance for actionability? Ideally, the underlying actionability is tracked, such as by checking for attached action event listeners. However, there is no browser API for this practice.",[11,2788,2789],{},"As an example for bad affordance in the GUI, imagine an overly subtle 'Shop' button. Bounding boxes in grounded GUI snapshots are a workaround – provided the element is discoverable as actionable in the DOM in the first place.",[290,2791],{"src":2792,"alt":2793,"loading":331,":width":2794},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/9.png","Example of a fictional UI/UX bad practice (overly subtle button)","480",[11,2796,2797,2798,2803],{},"As an example for bad affordance in the DOM, look at this datepicker from ",[185,2799,2802],{"href":2800,"rel":2801},"https://www.webmd.com",[189],"webmd.com",":",[290,2805],{"src":2806,"alt":2807,"loading":331,":width":2794},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/10.png","Example of an HTML bad practice (ignoring tag name semantics)",[1145,2809,2811],{"className":1147,"code":2810,"language":1149,"meta":522,"style":522},"\u003Cdiv>\n  \u003Cspan class=\"cell day-header\">Sun\u003C/span>\n  \u003Cspan class=\"cell day-header\">Mon\u003C/span>\n  \u003Cspan class=\"cell day-header\">Tue\u003C/span>\n  \u003Cspan class=\"cell day-header\">Wed\u003C/span>\n  \u003Cspan class=\"cell day-header\">Thu\u003C/span>\n  \u003Cspan class=\"cell day-header\">Fri\u003C/span>\n  \u003Cspan class=\"cell day-header\">Sat\u003C/span>\n  \u003C!-- [...] -->\n  \u003Cspan class=\"cell day blank\">\u003C/span>\n  \u003Cspan class=\"cell day blank\">\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>1\u003C/span>\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>2\u003C/span>\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>3\u003C/span>\u003C/span>\n  \u003C!-- [...] -->\n  \u003Cspan class=\"cell day\">\u003Cspan>29\u003C/span>\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>30\u003C/span>\u003C/span>\n\u003C/div>\n",[970,2812,2813,2821,2841,2860,2879,2898,2917,2936,2955,2961,2981,2999,3028,3055,3082,3086,3113,3140],{"__ignoreMap":522},[1153,2814,2815,2817,2819],{"class":1155,"line":1156},[1153,2816,1160],{"class":1159},[1153,2818,1607],{"class":1163},[1153,2820,1183],{"class":1159},[1153,2822,2823,2825,2827,2829,2831,2834,2837,2839],{"class":1155,"line":809},[1153,2824,1331],{"class":1159},[1153,2826,1153],{"class":1163},[1153,2828,1319],{"class":1167},[1153,2830,1171],{"class":1159},[1153,2832,2833],{"class":1174},"\"cell day-header\"",[1153,2835,2836],{"class":1159},">Sun\u003C/",[1153,2838,1153],{"class":1163},[1153,2840,1183],{"class":1159},[1153,2842,2843,2845,2847,2849,2851,2853,2856,2858],{"class":1155,"line":818},[1153,2844,1331],{"class":1159},[1153,2846,1153],{"class":1163},[1153,2848,1319],{"class":1167},[1153,2850,1171],{"class":1159},[1153,2852,2833],{"class":1174},[1153,2854,2855],{"class":1159},">Mon\u003C/",[1153,2857,1153],{"class":1163},[1153,2859,1183],{"class":1159},[1153,2861,2862,2864,2866,2868,2870,2872,2875,2877],{"class":1155,"line":1275},[1153,2863,1331],{"class":1159},[1153,2865,1153],{"class":1163},[1153,2867,1319],{"class":1167},[1153,2869,1171],{"class":1159},[1153,2871,2833],{"class":1174},[1153,2873,2874],{"class":1159},">Tue\u003C/",[1153,2876,1153],{"class":1163},[1153,2878,1183],{"class":1159},[1153,2880,2881,2883,2885,2887,2889,2891,2894,2896],{"class":1155,"line":1289},[1153,2882,1331],{"class":1159},[1153,2884,1153],{"class":1163},[1153,2886,1319],{"class":1167},[1153,2888,1171],{"class":1159},[1153,2890,2833],{"class":1174},[1153,2892,2893],{"class":1159},">Wed\u003C/",[1153,2895,1153],{"class":1163},[1153,2897,1183],{"class":1159},[1153,2899,2900,2902,2904,2906,2908,2910,2913,2915],{"class":1155,"line":1376},[1153,2901,1331],{"class":1159},[1153,2903,1153],{"class":1163},[1153,2905,1319],{"class":1167},[1153,2907,1171],{"class":1159},[1153,2909,2833],{"class":1174},[1153,2911,2912],{"class":1159},">Thu\u003C/",[1153,2914,1153],{"class":1163},[1153,2916,1183],{"class":1159},[1153,2918,2919,2921,2923,2925,2927,2929,2932,2934],{"class":1155,"line":1391},[1153,2920,1331],{"class":1159},[1153,2922,1153],{"class":1163},[1153,2924,1319],{"class":1167},[1153,2926,1171],{"class":1159},[1153,2928,2833],{"class":1174},[1153,2930,2931],{"class":1159},">Fri\u003C/",[1153,2933,1153],{"class":1163},[1153,2935,1183],{"class":1159},[1153,2937,2938,2940,2942,2944,2946,2948,2951,2953],{"class":1155,"line":1404},[1153,2939,1331],{"class":1159},[1153,2941,1153],{"class":1163},[1153,2943,1319],{"class":1167},[1153,2945,1171],{"class":1159},[1153,2947,2833],{"class":1174},[1153,2949,2950],{"class":1159},">Sat\u003C/",[1153,2952,1153],{"class":1163},[1153,2954,1183],{"class":1159},[1153,2956,2957],{"class":1155,"line":1417},[1153,2958,2960],{"class":2959},"sJ8bj","  \u003C!-- [...] -->\n",[1153,2962,2963,2965,2967,2969,2971,2974,2977,2979],{"class":1155,"line":1705},[1153,2964,1331],{"class":1159},[1153,2966,1153],{"class":1163},[1153,2968,1319],{"class":1167},[1153,2970,1171],{"class":1159},[1153,2972,2973],{"class":1174},"\"cell day blank\"",[1153,2975,2976],{"class":1159},">\u003C/",[1153,2978,1153],{"class":1163},[1153,2980,1183],{"class":1159},[1153,2982,2983,2985,2987,2989,2991,2993,2995,2997],{"class":1155,"line":1715},[1153,2984,1331],{"class":1159},[1153,2986,1153],{"class":1163},[1153,2988,1319],{"class":1167},[1153,2990,1171],{"class":1159},[1153,2992,2973],{"class":1174},[1153,2994,2976],{"class":1159},[1153,2996,1153],{"class":1163},[1153,2998,1183],{"class":1159},[1153,3000,3001,3003,3005,3007,3009,3012,3015,3017,3020,3022,3024,3026],{"class":1155,"line":1736},[1153,3002,1331],{"class":1159},[1153,3004,1153],{"class":1163},[1153,3006,1319],{"class":1167},[1153,3008,1171],{"class":1159},[1153,3010,3011],{"class":1174},"\"cell day\"",[1153,3013,3014],{"class":1159},">\u003C",[1153,3016,1153],{"class":1163},[1153,3018,3019],{"class":1159},">1\u003C/",[1153,3021,1153],{"class":1163},[1153,3023,2976],{"class":1159},[1153,3025,1153],{"class":1163},[1153,3027,1183],{"class":1159},[1153,3029,3030,3032,3034,3036,3038,3040,3042,3044,3047,3049,3051,3053],{"class":1155,"line":1746},[1153,3031,1331],{"class":1159},[1153,3033,1153],{"class":1163},[1153,3035,1319],{"class":1167},[1153,3037,1171],{"class":1159},[1153,3039,3011],{"class":1174},[1153,3041,3014],{"class":1159},[1153,3043,1153],{"class":1163},[1153,3045,3046],{"class":1159},">2\u003C/",[1153,3048,1153],{"class":1163},[1153,3050,2976],{"class":1159},[1153,3052,1153],{"class":1163},[1153,3054,1183],{"class":1159},[1153,3056,3057,3059,3061,3063,3065,3067,3069,3071,3074,3076,3078,3080],{"class":1155,"line":1761},[1153,3058,1331],{"class":1159},[1153,3060,1153],{"class":1163},[1153,3062,1319],{"class":1167},[1153,3064,1171],{"class":1159},[1153,3066,3011],{"class":1174},[1153,3068,3014],{"class":1159},[1153,3070,1153],{"class":1163},[1153,3072,3073],{"class":1159},">3\u003C/",[1153,3075,1153],{"class":1163},[1153,3077,2976],{"class":1159},[1153,3079,1153],{"class":1163},[1153,3081,1183],{"class":1159},[1153,3083,3084],{"class":1155,"line":1775},[1153,3085,2960],{"class":2959},[1153,3087,3088,3090,3092,3094,3096,3098,3100,3102,3105,3107,3109,3111],{"class":1155,"line":1784},[1153,3089,1331],{"class":1159},[1153,3091,1153],{"class":1163},[1153,3093,1319],{"class":1167},[1153,3095,1171],{"class":1159},[1153,3097,3011],{"class":1174},[1153,3099,3014],{"class":1159},[1153,3101,1153],{"class":1163},[1153,3103,3104],{"class":1159},">29\u003C/",[1153,3106,1153],{"class":1163},[1153,3108,2976],{"class":1159},[1153,3110,1153],{"class":1163},[1153,3112,1183],{"class":1159},[1153,3114,3115,3117,3119,3121,3123,3125,3127,3129,3132,3134,3136,3138],{"class":1155,"line":1790},[1153,3116,1331],{"class":1159},[1153,3118,1153],{"class":1163},[1153,3120,1319],{"class":1167},[1153,3122,1171],{"class":1159},[1153,3124,3011],{"class":1174},[1153,3126,3014],{"class":1159},[1153,3128,1153],{"class":1163},[1153,3130,3131],{"class":1159},">30\u003C/",[1153,3133,1153],{"class":1163},[1153,3135,2976],{"class":1159},[1153,3137,1153],{"class":1163},[1153,3139,1183],{"class":1159},[1153,3141,3142,3144,3146],{"class":1155,"line":1796},[1153,3143,1369],{"class":1159},[1153,3145,1607],{"class":1163},[1153,3147,1183],{"class":1159},[11,3149,3150,3151,3153,3154,3157,3158,3160,3161,3164,3165,3167],{},"The days wrapped in ",[970,3152,1153],{}," (e.g., ",[970,3155,3156],{},"16",") are clickable, but the HTML communicates no such actionability. The idiomatic DOM equivalent would instead use ",[970,3159,1164],{}," elements throughout, or at least assign suitable ",[970,3162,3163],{},"role"," attributes like ",[970,3166,1164],{},". A DOM snapshot pipeline leaves space for solving this problem: rewrite the virtual DOM with idiomatic elements.",[1145,3169,3171],{"className":1147,"code":3170,"language":1149,"meta":522,"style":522},"  \u003Cbutton class=\"cell day\">29\u003C/button>\n  \u003C!-- OR -->\n  \u003Cspan role=\"button\" class=\"cell day\">30\u003C/button>\n",[970,3172,3173,3191,3196],{"__ignoreMap":522},[1153,3174,3175,3177,3179,3181,3183,3185,3187,3189],{"class":1155,"line":1156},[1153,3176,1331],{"class":1159},[1153,3178,1164],{"class":1163},[1153,3180,1319],{"class":1167},[1153,3182,1171],{"class":1159},[1153,3184,3011],{"class":1174},[1153,3186,3104],{"class":1159},[1153,3188,1164],{"class":1163},[1153,3190,1183],{"class":1159},[1153,3192,3193],{"class":1155,"line":809},[1153,3194,3195],{"class":2959},"  \u003C!-- OR -->\n",[1153,3197,3198,3200,3202,3205,3207,3209,3211,3213,3215,3217,3219],{"class":1155,"line":818},[1153,3199,1331],{"class":1159},[1153,3201,1153],{"class":1163},[1153,3203,3204],{"class":1167}," role",[1153,3206,1171],{"class":1159},[1153,3208,1726],{"class":1174},[1153,3210,1319],{"class":1167},[1153,3212,1171],{"class":1159},[1153,3214,3011],{"class":1174},[1153,3216,3131],{"class":1159},[1153,3218,1164],{"class":1163},[1153,3220,1183],{"class":1159},[174,3222,3224],{"id":3223},"choosing-between-snapshot-representations","Choosing Between Snapshot Representations",[11,3226,3227],{},"No snapshot representation fits all purposes; the two base representations alone – GUI and DOM (HTML) – come with markedly different advantages and limitations. A proper snapshot tool should therefore route and cascade:",[341,3229,3230,3242,3248,3254],{},[143,3231,3232,3235,3236,3238,3239,3241],{},[14,3233,3234],{},"Route Modality."," Fall back to GUI snapshots if the UI is based on ",[970,3237,1070],{}," or cross-origin ",[970,3240,1066],{}," elements; use DOM snapshots by default.",[143,3243,3244,3247],{},[14,3245,3246],{},"Route Representation."," Categorise task and translate DOM snapshot to the most suitable representation. Categorising the task can be done with increasing levels of machine learning – simple keyword matching, or deploying a language model.",[143,3249,3250,3253],{},[14,3251,3252],{},"Scope Escalation."," If elicited action suggestions are unusable, widen the snapshot scope and representation richness.",[143,3255,3256,3259],{},[14,3257,3258],{},"Cascade to Hybrid."," If elicited action suggestions are still unusable, augment the rich DOM snapshot with a GUI snapshot for maximum grounding.",[19,3261,3262,3281],{},[22,3263,3264],{},[25,3265,3266,3269,3272,3275,3278],{},[28,3267],{"align":3268},"left",[28,3270,3271],{},"GUI",[28,3273,3274],{},"grounded",[28,3276,3277],{},"DOM",[28,3279,3280],{},"transformed",[37,3282,3283,3305,3326,3349,3368,3389],{},[25,3284,3285,3288,3293,3297,3300],{},[42,3286,3287],{"align":3268},"LLM Input Tokens",[42,3289,3290],{},[14,3291,3292],{},"Low",[42,3294,3295],{},[14,3296,3292],{},[42,3298,3299],{},"High",[42,3301,3302,3304],{},[14,3303,3292],{}," to Moderate",[25,3306,3307,3310,3313,3318,3322],{},[42,3308,3309],{"align":3268},"Element Targeting",[42,3311,3312],{},"Absolute",[42,3314,3315],{},[14,3316,3317],{},"Relative",[42,3319,3320],{},[14,3321,3317],{},[42,3323,3324],{},[14,3325,3317],{},[25,3327,3328,3331,3337,3341,3345],{},[42,3329,3330],{"align":3268},"Snapshot Scope",[42,3332,3333,3336],{},[14,3334,3335],{},"Free","/Viewport",[42,3338,3339,3336],{},[14,3340,3335],{},[42,3342,3343],{},[14,3344,3335],{},[42,3346,3347],{},[14,3348,3335],{},[25,3350,3351,3354,3357,3359,3364],{},[42,3352,3353],{"align":3268},"Snapshot Context",[42,3355,3356],{},"UI",[42,3358,3356],{},[42,3360,3361],{},[14,3362,3363],{},"Full",[42,3365,3366],{},[14,3367,3363],{},[25,3369,3370,3375,3380,3384,3387],{},[42,3371,3372,3374],{"align":3268},[970,3373,1066],{}," Context",[42,3376,3377],{},[14,3378,3379],{},"Viewport",[42,3381,3382],{},[14,3383,3379],{},[42,3385,3386],{},"Same-origin only",[42,3388,3386],{},[25,3390,3391,3395,3400,3404,3407],{},[42,3392,3393,3374],{"align":3268},[970,3394,1070],{},[42,3396,3397],{},[14,3398,3399],{},"Yes",[42,3401,3402],{},[14,3403,3399],{},[42,3405,3406],{},"No",[42,3408,3406],{},[174,3410,3412],{"id":3411},"using-snapshots-in-the-model-context","Using Snapshots in the Model Context",[11,3414,3415],{},"Creating a snapshot is one thing. Using it as context to a model prompt is the other. Usually, conversational AI maintains a full trace of history as model context. Since a snapshot is time-sensitive, however, only the latest snapshot can be targeted with actions by an LLM.",[11,3417,3418],{},"Whether to keep a trace of previous actions and model reasoning is a question in its own right. If web browsing is modelled as a history-independent task, the snapshot-to-actions agent loop could even be separated from the persistent conversational loop as transient, one-shot model prompts.",[174,3420,3422],{"id":3421},"timing-snapshots-in-the-web-client","Timing Snapshots in the Web Client",[11,3424,3425,3426,305],{},"LLM-based web agents are much slower than humans in taking and processing snapshots. Not least for that reason, snapshot rounds should be kept to a minimum and timed so that a snapshot is taken as soon as the UI has reached a stable state. That implies a non-trivial challenge: defining what qualifies as a UI change, and at what intensity such a change counts as a state shift – beyond hard page transitions, e.g., in a single-page application",[1013,3427,3428],{},[185,3429,3433],{"href":3430,"ariaDescribedBy":3431,"dataFootnoteRef":522,"id":3432},"#user-content-fn-5",[1019],"user-content-fnref-5","5",[3435,3436],"hr",{},[11,3438,3439,3440,305],{},"The snapshot is the central artefact in a web agent harness. Its representation is among the most significant factors for agent success",[1013,3441,3442],{},[185,3443,3447],{"href":3444,"ariaDescribedBy":3445,"dataFootnoteRef":522,"id":3446},"#user-content-fn-6",[1019],"user-content-fnref-6","6",[11,3449,3450,3451,3453],{},"At Webfuse, we develop a platform for agentic browsing. A focus of our research is on efficient snapshot representations. The Automation API can be used to create next-level snapshots: timed, rectified, across ",[970,3452,1066],{}," boundaries, etc.",[1316,3455,3458,3463],{"className":3456,"dataFootnotes":522},[3457],"footnotes",[174,3459,3462],{"className":3460,"id":1019},[3461],"sr-only","Footnotes",[341,3464,3465,3479,3492,3504,3530,3542],{},[143,3466,3468,233,3472],{"id":3467},"user-content-fn-1",[185,3469,3470],{"href":3470,"rel":3471},"https://arxiv.org/abs/2310.11441",[189],[185,3473,3478],{"href":3474,"ariaLabel":3475,"className":3476,"dataFootnoteBackref":522},"#user-content-fnref-1","Back to reference 1",[3477],"data-footnote-backref","↩",[143,3480,3482,3483,233,3487],{"id":3481},"user-content-fn-2","Our reimplementation of Browser Use's grounded GUI snapshots: ",[185,3484,3485],{"href":3485,"rel":3486},"https://github.com/webfuse-com/D2Snap/blob/main/snapshots/_bu.js",[189],[185,3488,3478],{"href":3489,"ariaLabel":3490,"className":3491,"dataFootnoteBackref":522},"#user-content-fnref-2","Back to reference 2",[3477],[143,3493,3495,233,3499],{"id":3494},"user-content-fn-3",[185,3496,3497],{"href":3497,"rel":3498},"https://arxiv.org/abs/2210.03945",[189],[185,3500,3478],{"href":3501,"ariaLabel":3502,"className":3503,"dataFootnoteBackref":522},"#user-content-fnref-3","Back to reference 3",[3477],[143,3505,3507,233,3511,233,3516,233,3523],{"id":3506},"user-content-fn-4",[185,3508,3509],{"href":3509,"rel":3510},"https://arxiv.org/abs/2508.04412",[189],[185,3512,3478],{"href":3513,"ariaLabel":3514,"className":3515,"dataFootnoteBackref":522},"#user-content-fnref-4","Back to reference 4",[3477],[185,3517,3478,3521],{"href":3518,"ariaLabel":3519,"className":3520,"dataFootnoteBackref":522},"#user-content-fnref-4-2","Back to reference 4-2",[3477],[1013,3522,1044],{},[185,3524,3478,3528],{"href":3525,"ariaLabel":3526,"className":3527,"dataFootnoteBackref":522},"#user-content-fnref-4-3","Back to reference 4-3",[3477],[1013,3529,1059],{},[143,3531,3533,233,3537],{"id":3532},"user-content-fn-5",[185,3534,3535],{"href":3535,"rel":3536},"https://github.com/webfuse-com/DOMQuake",[189],[185,3538,3478],{"href":3539,"ariaLabel":3540,"className":3541,"dataFootnoteBackref":522},"#user-content-fnref-5","Back to reference 5",[3477],[143,3543,3545,233,3549],{"id":3544},"user-content-fn-6",[185,3546,3547],{"href":3547,"rel":3548},"https://arxiv.org/abs/2504.01382",[189],[185,3550,3478],{"href":3551,"ariaLabel":3552,"className":3553,"dataFootnoteBackref":522},"#user-content-fnref-6","Back to reference 6",[3477],[3555,3556,3557],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .suv1-, html code.shiki .suv1-{--shiki-default:#E1E4E8;--shiki-dark:#E1E4E8}html pre.shiki code .sxg3X, html code.shiki .sxg3X{--shiki-default:#85E89D;--shiki-dark:#85E89D}html pre.shiki code .sFR8T, html code.shiki .sFR8T{--shiki-default:#B392F0;--shiki-dark:#B392F0}html pre.shiki code .s4wv1, html code.shiki .s4wv1{--shiki-default:#9ECBFF;--shiki-dark:#9ECBFF}html pre.shiki code .s8ozJ, html code.shiki .s8ozJ{--shiki-default:#79B8FF;--shiki-dark:#79B8FF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}",{"title":522,"searchDepth":809,"depth":809,"links":3559},[3560,3564,3570,3571,3572,3573,3574],{"id":922,"depth":809,"text":923,"children":3561},[3562,3563],{"id":945,"depth":818,"text":946},{"id":985,"depth":818,"text":986},{"id":1047,"depth":809,"text":1048,"children":3565},[3566,3567,3568,3569],{"id":1077,"depth":818,"text":1078},{"id":1102,"depth":818,"text":1103},{"id":2030,"depth":818,"text":2031},{"id":2054,"depth":818,"text":2055},{"id":2779,"depth":809,"text":2780},{"id":3223,"depth":809,"text":3224},{"id":3411,"depth":809,"text":3412},{"id":3421,"depth":809,"text":3422},{"id":1019,"depth":809,"text":3462},"2025-08-28","Understand how web-based user interfaces can be serialised for use with LLM-based web agents.",{"homepage":865,"relatedLinks":3578},[3579,3582],{"text":3580,"href":3581},"DOM Downsampling for LLM-Based Web Agents","/rnd/dom-downsampling-for-llm-based-web-agents",{"text":3583,"href":3584,"external":865},"Webfuse Automation API","https://dev.webfuse.com/automation-api/","/blog/serialising-web-ui-state-for-llms-the-complete-guide",{"title":880,"description":3576},{"loc":3585},"blog/1013.serialising-web-ui-state-for-llms-the-complete-guide",[875,3590,3591,3592,3593],"web-agents","web-automation","llms","snapshots","X7R93b2XjyDfLllTDaNgFoo_c8AvF9oBDGzv4yiISFA",{"id":3596,"title":3597,"authorId":881,"body":3598,"category":875,"created":4074,"description":4075,"extension":829,"faqs":852,"featurePriority":852,"head":852,"hideCta":865,"landingPath":852,"meta":4076,"navigation":865,"ogImage":852,"path":4082,"robots":852,"schemaOrg":852,"seo":4083,"sitemap":4084,"stem":4085,"tags":4086,"__hash__":4088},"blog/blog/1011.a-gentle-introduction-to-ai-agents-for-the-web.md","A Gentle Introduction to AI Agents for the Web",{"type":8,"value":3599,"toc":4055},[3600,3618,3622,3630,3635,3638,3642,3645,3665,3669,3678,3682,3698,3703,3706,3710,3721,3725,3728,3732,3739,3743,3766,3770,3774,3777,3780,3784,3805,3808,3812,3816,3825,3836,3841,3845,3854,3857,3861,3864,3867,3871,3884,3887,3891,3900,3903,3907,3911,3914,3918,3922,3925,3999,4001,4004,4018],[11,3601,3602,3607,3608,3607,3612,3617],{},[185,3603,3606],{"href":3604,"rel":3605},"https://openai.com/index/introducing-operator/",[189],"Operator",", ",[185,3609,1035],{"href":3610,"rel":3611},"https://browser-use.com/",[189],[185,3613,3616],{"href":3614,"rel":3615},"https://www.skyvern.com/",[189],"Skyvern"," – we are currently witnessing an evolution of autonomous web browsing agents. The concept of agents is, in general, ubiquitous in the era of AI.",[174,3619,3621],{"id":3620},"what-is-an-agent","What is an Agent?",[11,3623,3624,3625,305],{},"An agent is an entity that acts on behalf of another entity – its principal. Robotic agents are modelled after humans: they perceive the world and act in it to achieve a specified goal. Whilst humans perceive through their senses, agents perceive through sensors; whilst humans act through motor systems (e.g., fingers), agents act through actuators. This terminology has been transferred from hard- to software agents, particularly those premised on artificial intelligence (AI). An agent's purpose, that is, its goal, is clearly defined; the principal interfaces with that goal by assigning the agent tasks",[1013,3626,3627],{},[185,3628,1021],{"href":1017,"ariaDescribedBy":3629,"dataFootnoteRef":522,"id":1020},[1019],[290,3631],{"alt":3632,"loading":331,"src":3633,"provider":332,":width":3634},"Figurative comparison of component equivalents between a human and an AI agent","/blog/a-gentle-introduction-to-ai-agents-for-the-web/1.svg","430",[11,3636,3637],{},"The analogy is more than decorative: an agent takes over the tool-use expertise, leaving the principal with only the domain expertise. Traditional software applications like Photoshop demand both.",[525,3639,3641],{"id":3640},"what-makes-an-agent-intelligent","What Makes an Agent Intelligent?",[11,3643,3644],{},"For acting on behalf of a human, developing an agent could be as simple as making it follow a hardcoded sequence of actions. This describes process automation, but not an artificially intelligent agent. Whilst definitions for AI differ, agent intelligence comes from making informed decisions that are not hardcoded. Here are common – not necessarily mutually exclusive – high-level approaches to induce intelligence:",[140,3646,3647,3653,3659],{},[143,3648,3649,3652],{},[14,3650,3651],{},"Search"," – the agent has a complete or heuristic model of the world and the action space to anticipate the effects of actions and search for a goal state.",[143,3654,3655,3658],{},[14,3656,3657],{},"Reinforcement learning"," – the agent has an initial action space and iteratively discovers the world and effects of actions, learning an optimal policy to act in order to reach a goal state.",[143,3660,3661,3664],{},[14,3662,3663],{},"Neural networks"," – the agent has no model, only examples: a network is fitted to task–solution instances and generalises to unseen, yet related tasks.",[525,3666,3668],{"id":3667},"a-quick-note-on-multi-agent-systems","A Quick Note on Multi-Agent Systems",[11,3670,3671,3672,3677],{},"Agent design gets an additional edge when other entities act in the same world as the agent. Such entities can be considered agents on their own. Their beliefs and goals might affect the agent: cooperative agents share the designed agent's goal, whilst adversarial agents have an opposite goal. In a zero-sum game, an adversarial agent maximises the exact inverse of the designed agent's utility. Knowing this, the designed agent needs to anticipate intelligent decisions by the adversary (e.g., by applying the ",[185,3673,3676],{"href":3674,"rel":3675},"https://cs.stanford.edu/people/eroberts/courses/soco/projects/2003-04/intelligent-search/minimax.html",[189],"Minimax algorithm","), as opposed to random world changes. That said, web browsing – modelled after humans – is not a typical multi-agent scenario.",[525,3679,3681],{"id":3680},"the-agent-loop","The Agent Loop",[11,3683,3684,3689,3690,3697],{},[185,3685,3688],{"href":3686,"rel":3687},"https://www.nngroup.com/people/don-norman/",[189],"Don Norman","'s ",[185,3691,3694],{"href":3692,"rel":3693},"https://jnd.org/books/the-design-of-everyday-things-revised-and-expanded-edition/",[189],[906,3695,3696],{},"Seven Stages of Action"," is a model of the human cognition cycle that generalises to an agent's lifecycle – the agent loop: with a (1) goal, (2) intent to act, (3) plan actions, (4) execute the action plan – this alters the world – then (5) perceive the new world state, (6) interpret the perception, and finally (7) evaluate the interpretation – report task success or an irrevocable failure; continue the cycle if the world state not yet reflects a goal state.",[290,3699],{"alt":3700,"loading":331,"src":3701,"provider":332,":width":3702},"Don Norman's Seven Stages of Action – a model of human cognition","/blog/a-gentle-introduction-to-ai-agents-for-the-web/2.svg","500",[11,3704,3705],{},"Humans can run many of these cycles hierarchically, with increasing frequency the smaller the task. Take driving a car as an example: choosing a route runs on the order of minutes, choosing when to change lanes on seconds, and holding the car in its lane on fractions of a second. A cycle for holding the lane (1) is: intent to actively hold the lane (2), plan to keep the steering wheel straight, then watch the road (3), execute plan (4), observe the road (5), check whether the car is in the lane (6), if not, cycle again with an adjusted action plan (e.g., slightly rotate the steering wheel to the right) (7).",[525,3707,3709],{"id":3708},"errors-and-recovery","Errors and Recovery",[11,3711,3712,3713,3716,3717,3720],{},"Precisely speaking, \"to err is human\" means that errors are inevitable when heuristically navigating the world. Consequently, AI agents regularly run into errors, too. Norman's Seven Stages of Action make two error spaces apparent: when executing an action plan; the ",[906,3714,3715],{},"Gulf of Action (Execution)"," – how to act in the world? – and when perceiving world state; the ",[906,3718,3719],{},"Gulf of Perception (Evaluation)"," – how to understand the world?",[290,3722],{"alt":3723,"loading":331,"src":3724,"provider":332,":width":3702},"The Gulf of Action (Execution) and  Perception (Evaluation) in Don Norman's Seven Stages of Action","/blog/a-gentle-introduction-to-ai-agents-for-the-web/3.svg",[11,3726,3727],{},"There are two notable kinds of errors within the Gulf of Action: mistakes and slips. A mistake occurs when the action plan is wrong, although executing this action plan might go right in itself. A slip occurs when the action plan is right, but the execution goes wrong. Here is an example: stopping a manual-shift car without pressing the clutch. This error describes a mistake if the driver did not know about the need to press the clutch, e.g., because they only knew how to operate automatic-drive cars. It is, in turn, a slip if they simply forgot to press the clutch because of fatigue. Since cognition cycles are hierarchic, most errors can be recovered within a cycle. Hearing a choking engine sound, the driver might adjust the action plan and press the clutch late. Errors that cannot be recovered – picture the car's engine dying – require adjusting the goal, which can be seen as moving on to a new cycle.",[174,3729,3731],{"id":3730},"web-browsing-agents","Web Browsing Agents",[11,3733,3734,3735,3738],{},"The advent of large language models (LLMs) has recently opened a new perspective on developing web agents – falling back on the neural networks approach to intelligence. An LLM can serve as a plug-in reasoning backend: given a task and somehow encoded UI state, it can suggest input actions – type and target, e.g., click at ",[970,3736,3737],{},"(200, 375)"," – that progress the given task. Moreover, it can decide whether a goal state has been reached. Providing a state-and-action history within context conveys to the model prior outcomes and contributes to in-context learning.",[525,3740,3742],{"id":3741},"web-browsing-is-all-about-the-journey","Web Browsing is All About the Journey",[11,3744,3745,3746,3749,3750,3755,3756,3761,3762,3765],{},"A web browsing task is any task that can be solved through a web-based user interface (UI). For instance: ",[906,3747,3748],{},"\"Book the cheapest flight from Amsterdam to New York tomorrow night\""," on ",[185,3751,3754],{"href":3752,"rel":3753},"https://www.klm.com",[189],"klm.com",". Application programming interfaces (APIs; e.g., ",[185,3757,3760],{"href":3758,"rel":3759},"https://www.ibm.com/think/topics/rest-apis",[189],"REST APIs",") are often restricted, incomplete, or unavailable. For that reason, web browsing agents today have to rely on human-facing web application UIs. Many agent use cases, such as for customer experience, even require a human in the loop. In this context, web browsing agents come at two levels: outcome-driven agents care only about reaching a goal state, which favours shortcuts like direct URL navigation or expert navigation like ",[970,3763,3764],{},"Ctrl + F",", whilst journey-driven agents are supposed to browse a web application in human fashion, via the application-native UI.",[290,3767],{"alt":3768,"loading":331,"src":3769},"A depiction of a web journey over three web application UI states","/blog/a-gentle-introduction-to-ai-agents-for-the-web/4.png",[525,3771,3773],{"id":3772},"a-formal-model-for-web-browsing","A Formal Model for Web Browsing",[11,3775,3776],{},"The world of web application UIs is hard to formalise; a complete model spans the union of every reachable UI state in every web application on the web. For that reason, conventional AI has failed to produce capable generalist web agents. From a web agent design perspective, however, an abstract model of web browsing is sufficient.",[11,3778,3779],{},"Web browsing might afford a model that is history-independent. A human could certainly pick up a partially completed browsing task. For example, completing the payment process on an airline website for an already selected flight – assuming prior actions were correct with respect to the specified task (e.g., the selected destination).",[290,3781],{"alt":3782,"loading":331,"src":3783},"Screenshot of the payment page on klm.com that reflects outcomes of prior actions and affords next actions","/blog/a-gentle-introduction-to-ai-agents-for-the-web/5.png",[11,3785,3786,3787,3792,3793,3798,3799,3804],{},"Yet, many web applications lack ",[185,3788,3791],{"href":3789,"rel":3790},"https://www.nngroup.com/articles/ten-usability-heuristics/#toc-1-visibility-of-system-status-1",[189],"visibility of system status"," – against ",[185,3794,3797],{"href":3795,"rel":3796},"https://www.nngroup.com/people/jakob-nielsen/",[189],"Jakob Nielsen","'s first heuristic for UI usability. Visibility of system status comprises reflecting both outcomes of prior actions and affordances for next actions in the UI. With this concept, web browsing is best described as a partially observable Markov decision process (POMDP)",[1013,3800,3801],{},[185,3802,1044],{"href":1041,"ariaDescribedBy":3803,"dataFootnoteRef":522,"id":1043},[1019]," with, adapting to discovered UI states, where states can be tuples of UI state trajectories to contain history.",[11,3806,3807],{},"Reflex agents take actions that solely depend on the current world state. Planning agents, in turn, plan ahead. For most web browsing tasks, future UI states are roughly predictable, given that web journeys are similar across different web applications that help with similar tasks. In POMDP terms, the expected next UI state has a high probability, whilst unexpected states remain with a non-negligible probability. The further a web journey is progressed, the higher the probability of the next expected UI state gets: the flight selection step is likely followed by the passenger details step, which is even more likely followed by the payment step.",[290,3809],{"alt":3810,"loading":331,"src":3811},"Graph-based web journey representation of an unknown, adaptive, partially observable Markov decision process (POMDP)","/blog/a-gentle-introduction-to-ai-agents-for-the-web/6.png",[525,3813,3815],{"id":3814},"llm-sensors-actuators-agent","LLM + Sensors + Actuators = Agent",[11,3817,3818,3819,3824],{},"Conversationally satisfying simple information requests does not reasonably qualify an LLM as an agent – no more than it would a TF-IDF search system",[1013,3820,3821],{},[185,3822,1059],{"href":1056,"ariaDescribedBy":3823,"dataFootnoteRef":522,"id":1058},[1019],", or a colleague telling you the time. The web agent LLM backend is metaphorically the brain of the agent. It abstracts the world model and state transition function, and can decide whether a goal state was reached.",[11,3826,3827,3828,3607,3830,3607,3832,3835],{},"Turning an LLM into a web agent means, at the core, connecting it with sensors and actuators, for example, the ability to take DOM snapshots and dispatch clicks via CSS selectors. This is a specialised architecture of LLM tool calling. The set of available input actions is known from human computer use: ",[906,3829,908],{},[906,3831,911],{},[906,3833,3834],{},"scrolling",", etc. The agent loop is then a continuous cycle of serialising UI state for the model to elicit action suggestions under a goal derived from the user-specified task.",[290,3837],{"alt":3838,"loading":331,"src":3839,"provider":332,":width":3840},"Figurative depiction of components in the agent loop","/blog/a-gentle-introduction-to-ai-agents-for-the-web/7.svg","460",[525,3842,3844],{"id":3843},"snapshots-serialised-ui-state","Snapshots – Serialised UI State",[11,3846,3847,3848,3853],{},"There are plenty of ways to serialise the state of a web-based UI – referred to as a snapshot. A screenshot resembles what humans see when viewing a web page's graphical UI (GUI) at a given point in time. That said, a UI can be represented in versatile ways. LLMs have demonstrated the ability to interpret HTML and even navigate a UI encoded in this way",[1013,3849,3850],{},[185,3851,1552],{"href":1549,"ariaDescribedBy":3852,"dataFootnoteRef":522,"id":1551},[1019],". The document object model (DOM) is a web browser's runtime model of a web page. Serialised as HTML, to that effect, DOM snapshots represent a compelling alternative to GUI snapshots, offering certain advantages, such as processability (e.g., filtering). Filtering the DOM by pruning hidden, disabled, or redundant subtrees saves LLM communication costs and works towards more stable and predictable journeys.",[11,3855,3856],{},"Outcome-driven web agents have used UI-invasive snapshot measures, such as grounding LLMs with bounding boxes around actionable elements.",[290,3858],{"alt":3859,"loading":331,"src":3860},"Example of a screenshot (GUI snapshot) with coloured bounding boxes that visually ground an LLM","/blog/a-gentle-introduction-to-ai-agents-for-the-web/8.png",[11,3862,3863],{},"The ideal snapshot representation depends on the task. To get an idea, classify between navigation and information retrieval tasks: navigation tasks require snapshots to preserve actionable elements, whilst retrieval tasks require them to preserve text – both can be sourced from DOM snapshots. Accessibility tree snapshots are another DOM-sourced representation. Web UIs rarely afford merely a single task, so many of their components are noise in any given snapshot – which is not just a UI serialisation, but one scoped to the task at hand. GUI snapshots become a necessity when the target application is graphics-based, such as a canvas-based application like Google Maps.",[11,3865,3866],{},"A snapshot impacts the width of Norman's Gulf of Perception: if a relevant element is omitted or diffused, the model is hindered in identifying correct actions to progress a web browsing task. The Gulf of Action mostly grows when the snapshot-encoded UI state and the live UI state have irrecoverably drifted apart. The agent loop should be fast to prevent such drift. The Gulf of Perception is much wider by the nature of the problem: what snapshot representation can an LLM interpret, and which representation is best?",[525,3868,3870],{"id":3869},"targeting-action-suggestions","Targeting – Action Suggestions",[11,3872,3873,3874,3607,3876,3879,3880,3883],{},"An action is a user event, such as ",[906,3875,963],{},[906,3877,3878],{},"type",", or ",[906,3881,3882],{},"scroll",", targeting a specific element in the UI. The set of available actions thereby is defined by the accepted user input types which are generally derived and standardised over the mouse and keyboard devices.",[11,3885,3886],{},"The type of snapshot representation provided to an LLM dictates how an element can be targeted for an action. Vision-based, geometric targeting is indirect and absolute, i.e., by coordinates. DOM-based targeting is direct and relative, e.g., by CSS selectors or object references. Relative targeting remains valid if the UI shifts between snapshot and action time. A shift can happen due to many reasons, for instance, a lazy-loaded image.",[525,3888,3890],{"id":3889},"the-agent-harness","The Agent Harness",[11,3892,3893,3894,3899],{},"Any custom logic that sits between the LLM backend and the web page UI (problem world) is considered an agent harness – often linguistically equated with the agent in general. Common aspects of a harness include task decomposition, model routing, snapshot creation, history management, error recovery, and timeouts. ",[185,3895,3898],{"href":3896,"rel":3897},"https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents",[189],"System prompts"," are also part of the harness, instructing a general-purpose LLM to model a web agent for a specific domain – right in context.",[11,3901,3902],{},"Designing a harness is where the POMDP and Seven Stages of Action models come into play. The POMDP model helps with context engineering, for instance, how much history to provide to the LLM. The Seven Stages of Action set out problem components and artefacts, for example, the LLM or snapshots, respectively.",[290,3904],{"alt":3905,"loading":331,"src":3906,"provider":332},"Don Norman's Seven Stages of Action mapped to the agentic web browsing problem","/blog/a-gentle-introduction-to-ai-agents-for-the-web/9.svg",[525,3908,3910],{"id":3909},"the-model-context-protocol","The Model Context Protocol",[11,3912,3913],{},"LLM providers commonly offer a structured output interface, which allows eliciting well-formed action suggestions from an LLM. A client can subsequently parse and drive these suggested actions in the live user interface. Through the model context protocol (MCP) – another web-based RPC standard – structured output can be bridged with tools that LLMs can call directly, without an extra client trip.",[290,3915],{"alt":3916,"loading":331,"src":3917,"provider":332,":width":3702},"Figurative depiction of components in the agent loop using the model context protocol (MCP)","/blog/a-gentle-introduction-to-ai-agents-for-the-web/10.svg",[525,3919,3921],{"id":3920},"web-browser-environments","Web Browser Environments",[11,3923,3924],{},"When connected to web perception and actuation tools via MCP, an ordinary AI chat UI already qualifies as an agent UI; the MCP integration decides where the actual web browser runs. There are several types of browser runtime environment:",[140,3926,3927,3942,3955,3967],{},[143,3928,3929,3932,3933,3936,3937],{},[14,3930,3931],{},"Agentic browser"," – the agent's UI embeds a browser.",[3934,3935],"br",{},"\nExample: ",[185,3938,3941],{"href":3939,"rel":3940},"https://www.perplexity.ai/comet",[189],"Perplexity Comet",[143,3943,3944,3947,3948,3936,3950],{},[14,3945,3946],{},"Cloud browser"," – the agent spins up a browser in the cloud.",[3934,3949],{},[185,3951,3954],{"href":3952,"rel":3953},"https://www.browserbase.com",[189],"Browserbase",[143,3956,3957,3960,3961,3936,3963],{},[14,3958,3959],{},"Local browser (3rd-party browser)"," – the agent spins up an isolated browser locally.",[3934,3962],{},[185,3964,3966],{"href":895,"rel":3965},[189],"Chromium + Playwright",[143,3968,3969,3972,3973],{},[14,3970,3971],{},"Local browser (1st-party browser)"," – the agent is embedded within a web application running in a user's personal browser, e.g., in the form of a chatbot.\n",[140,3974,3975,3988],{},[143,3976,3977,3980,3981,3936,3983],{},[14,3978,3979],{},"Implemented (1st-party app)"," – served directly through the website.",[3934,3982],{},[185,3984,3987],{"href":3985,"rel":3986},"https://www.cognigy.com/platform/ai-agent-studio",[189],"Cognigy Studio",[143,3989,3990,3993,3994,3936,3996],{},[14,3991,3992],{},"Augmented (3rd-party app)"," – served indirectly through a website proxy.",[3934,3995],{},[185,3997,903],{"href":901,"rel":3998},[189],[3435,4000],{},[11,4002,4003],{},"Building AI agents has been approached from different angles. LLMs have opened up new perspectives on the design and development of AI agents, particularly web agents, which work in a problem domain that is hard to grasp with a formal model. Whilst higher-level agent methodology can borrow much from established research, LLMs have introduced new challenges, such as how to take UI state snapshots for eliciting input action suggestions from such models.",[11,4005,4006,4007,4012,4013,305],{},"At Webfuse, we research agentic web browsing and develop a platform for agentic browsing. Our web application reverse proxy allows LLMs to perceive and act on the live web. Use cases include connecting a chat UI like ",[185,4008,4011],{"href":4009,"rel":4010},"https://claude.com/",[189],"Claude",", or shipping ",[185,4014,4017],{"href":4015,"rel":4016},"https://github.com/webfuse-com/extension-elevenlabs-mcp",[189],"perceive-and-act-ready chatbots",[1316,4019,4021,4024],{"className":4020,"dataFootnotes":522},[3457],[174,4022,3462],{"className":4023,"id":1019},[3461],[341,4025,4026,4035,4041,4047],{},[143,4027,4028,233,4032],{"id":3467},[185,4029,4030],{"href":4030,"rel":4031},"https://inst.eecs.berkeley.edu/~cs188/textbook/textbook_full.pdf",[189],[185,4033,3478],{"href":3474,"ariaLabel":3475,"className":4034,"dataFootnoteBackref":522},[3477],[143,4036,4037,4038],{"id":3481},"A POMDP is a decision process in which the agent cannot see the exact world state, only partial observations of it, and so must choose actions against a probability distribution over the states it might be in. ",[185,4039,3478],{"href":3489,"ariaLabel":3490,"className":4040,"dataFootnoteBackref":522},[3477],[143,4042,4043,4044],{"id":3494},"TF-IDF (term frequency-inverse document frequency) is an information retrieval method that ranks documents by how often a query term occurs in each one, weighted down by how common that term is across the whole collection – so terms that are frequent here but rare elsewhere carry the most weight. ",[185,4045,3478],{"href":3501,"ariaLabel":3502,"className":4046,"dataFootnoteBackref":522},[3477],[143,4048,4049,233,4052],{"id":3506},[185,4050,3497],{"href":3497,"rel":4051},[189],[185,4053,3478],{"href":3513,"ariaLabel":3514,"className":4054,"dataFootnoteBackref":522},[3477],{"title":522,"searchDepth":809,"depth":809,"links":4056},[4057,4063,4073],{"id":3620,"depth":809,"text":3621,"children":4058},[4059,4060,4061,4062],{"id":3640,"depth":818,"text":3641},{"id":3667,"depth":818,"text":3668},{"id":3680,"depth":818,"text":3681},{"id":3708,"depth":818,"text":3709},{"id":3730,"depth":809,"text":3731,"children":4064},[4065,4066,4067,4068,4069,4070,4071,4072],{"id":3741,"depth":818,"text":3742},{"id":3772,"depth":818,"text":3773},{"id":3814,"depth":818,"text":3815},{"id":3843,"depth":818,"text":3844},{"id":3869,"depth":818,"text":3870},{"id":3889,"depth":818,"text":3890},{"id":3909,"depth":818,"text":3910},{"id":3920,"depth":818,"text":3921},{"id":1019,"depth":809,"text":3462},"2025-06-15","Get an idea of agents and how to build AI agents that browse the web: key concepts, terminology, and challenges.",{"homepage":865,"relatedLinks":4077},[4078,4080,4081],{"text":4079,"href":3585},"Serialising Web UI State for LLMs",{"text":3580,"href":3581},{"text":3583,"href":3584,"external":865},"/blog/a-gentle-introduction-to-ai-agents-for-the-web",{"title":3597,"description":4075},{"loc":4082},"blog/1011.a-gentle-introduction-to-ai-agents-for-the-web",[875,4087,3590,3591,3592],"browser-agents","65hAoQNete8P3AgO-H42P0eBhVfIkl5vIfZZAkvL-9o",1791897078478]