[{"data":1,"prerenderedAt":4315},["ShallowReactive",2],{"/blog/forward-deployed-engineering-fde-guide":3,"related-/blog/forward-deployed-engineering-fde-guide":1103},{"id":4,"title":5,"authorId":6,"body":7,"category":1054,"created":1055,"description":1056,"extension":1057,"faqs":1058,"featurePriority":1074,"head":1075,"hideCta":1076,"landingPath":1075,"meta":1077,"navigation":1092,"ogImage":1075,"path":1093,"robots":1075,"schemaOrg":1075,"seo":1094,"sitemap":1095,"stem":1096,"tags":1097,"__hash__":1102},"blog/blog/1055.forward-deployed-engineering-fde-guide.md","Forward Deployed Engineering (FDE): The Complete 2026 Guide","salome-koshadze",{"type":8,"value":9,"toc":1020},"minimark",[10,22,31,39,45,50,69,74,88,108,117,122,125,249,252,261,267,276,280,283,292,295,316,319,325,329,332,341,347,353,359,363,366,396,399,402,407,411,414,464,470,473,487,491,494,513,516,536,539,542,546,549,555,559,566,570,573,577,580,600,604,610,616,619,623,626,629,667,673,677,685,688,692,695,701,704,741,745,768,777,781,788,792,801,821,824,828,836,856,859,862,866,869,889,892,898,902,905,976,982,986,992,998,1004,1010,1014],[11,12,13,14,21],"p",{},"On Palantir's second-quarter 2026 earnings call, CTO Shyam Sankar gave investors a one-liner: \"Remember, only Palantir has FDEs. Everyone else has sparkling sales engineers.\" He said it weeks after ",[15,16,20],"a",{"href":17,"rel":18},"https://siliconangle.com/2026/06/30/aws-launches-forward-deployed-engineering-team-speed-enterprise-agentic-ai-adoption/",[19],"nofollow","AWS"," and Microsoft committed billions to their own forward deployed units, and after OpenAI and Anthropic set up private-equity-backed firms to put engineers inside customer businesses.",[23,24],"nuxt-picture",{":height":25,":width":26,"alt":27,"loading":28,"provider":29,"src":30},"423","826","Palantir CTO Shyam Sankar's Q2 2026 earnings call quote that only Palantir has FDEs, above four cards for the competing moves. AWS has a $1 billion FDE organization, Microsoft has a $2.5 billion unit with 6,000 embedded experts, and OpenAI and Anthropic each have private-equity-backed deployment firms.","lazy","none","/blog/forward-deployed-engineering-fde-guide/big-techs-fde-pivot.svg",[11,32,33,34,38],{},"All four bets rest on the same premise: an AI agent succeeds or fails inside the customer's systems, with the customer's permissions, data, and approval chains. The model is the cheap part. ",[35,36,37],"strong",{},"Forward deployed engineering"," is the name the industry settled on for the work around it, and this guide covers how that work runs at a technical level, what it costs, and how to judge an engagement.",[11,40,41,44],{},[35,42,43],{},"Short answer:"," Forward deployed engineering (FDE) is a delivery model in which a vendor's software engineers work inside a customer's organization to get the vendor's product running in production on the customer's real data and systems. The engineer writes production code, owns a business outcome, and feeds what they learn back into the vendor's product. Palantir popularized the model, and AI companies now use it to deploy agents.",[11,46,47],{},[35,48,49],{},"Key takeaways:",[51,52,53,57,60,63,66],"ul",{},[54,55,56],"li",{},"An FDE ships production code on the customer's live systems, owns a business result, and sends findings back to the product team. Without that loop, the job is consulting.",[54,58,59],{},"The work follows an eight-step loop. The two steps teams skip most, building an eval set and generalizing what was built, decide whether the agent provably works and whether the vendor's product improves.",[54,61,62],{},"AI agents revived the model. Their value depends on permissions, data access, evals, and process changes that a vendor cannot ship in a box.",[54,64,65],{},"A real engagement leaves the customer able to run the system alone: infrastructure as code, evals, MCP servers, runbooks, and a trained owner.",[54,67,68],{},"The 2026 hiring and funding announcements are large, but job-posting growth starts from a near-zero 2025 base.",[70,71,73],"h2",{"id":72},"what-a-forward-deployed-engineer-actually-does","What a forward deployed engineer actually does",[11,75,76,77,82,83,87],{},"The title comes from U.S. military jargon for forward-based units. ",[15,78,81],{"href":79,"rel":80},"https://en.wikipedia.org/wiki/Forward_deployed_engineer",[19],"Wikipedia",", citing a 2009 article in ",[84,85,86],"em",{},"Science",", records Palantir using it by that year. Three traits separate the role from adjacent jobs:",[51,89,90,96,102],{},[54,91,92,95],{},[35,93,94],{},"Production code."," The FDE ships software that runs against the customer's live systems.",[54,97,98,101],{},[35,99,100],{},"Outcome ownership."," The engagement is measured by a business result, such as tickets resolved or hours removed from a process.",[54,103,104,107],{},[35,105,106],{},"A loop back to the product."," What the FDE learns in the field changes the vendor's roadmap.",[11,109,110,111,116],{},"Palantir's own engineering blog describes the split in a ",[15,112,115],{"href":113,"rel":114},"https://blog.palantir.com/dev-versus-delta-demystifying-engineering-roles-at-palantir-ad44c2a6e87",[19],"short formula",": a product engineer (\"Dev\") works on \"one capability, many customers,\" and a forward deployed engineer (\"Delta\") works on \"one customer, many capabilities.\"",[118,119,121],"h3",{"id":120},"fde-vs-adjacent-roles","FDE vs adjacent roles",[11,123,124],{},"The title overlaps with several older jobs, and some companies use it as a relabel. The table below shows where the lines usually fall.",[126,127,128,150],"table",{},[129,130,131],"thead",{},[132,133,134,138,141,144,147],"tr",{},[135,136,137],"th",{},"Role",[135,139,140],{},"When they engage",[135,142,143],{},"Owns production code?",[135,145,146],{},"Owns an outcome?",[135,148,149],{},"Feeds the product roadmap?",[151,152,153,172,192,212,229],"tbody",{},[132,154,155,161,164,167,169],{},[156,157,158],"td",{},[35,159,160],{},"Forward deployed engineer",[156,162,163],{},"After the contract, for weeks to months",[156,165,166],{},"Yes",[156,168,166],{},[156,170,171],{},"Yes, by design",[132,173,174,180,183,186,189],{},[156,175,176],{},[15,177,179],{"href":178},"/compare/fde-vs-sales-engineer","Solutions or sales engineer",[156,181,182],{},"Before the sale",[156,184,185],{},"Rarely, demos and proofs of concept",[156,187,188],{},"No, owns the technical win",[156,190,191],{},"Informally",[132,193,194,200,203,206,209],{},[156,195,196],{},[15,197,199],{"href":198},"/compare/fde-vs-solutions-architect","Solutions architect",[156,201,202],{},"Before and after the sale",[156,204,205],{},"Rarely, designs more than codes",[156,207,208],{},"Partly",[156,210,211],{},"Sometimes",[132,213,214,217,220,223,226],{},[156,215,216],{},"Customer success or support engineer",[156,218,219],{},"After the sale, ticket-driven",[156,221,222],{},"Fixes and scripts",[156,224,225],{},"Retention",[156,227,228],{},"Through bug reports",[132,230,231,237,240,243,246],{},[156,232,233],{},[15,234,236],{"href":235},"/blog/forward-deployed-engineer-vs-technical-consultant","Professional services or SI consultant",[156,238,239],{},"Project-based",[156,241,242],{},"Yes, often custom",[156,244,245],{},"Project delivery",[156,247,248],{},"Usually not",[11,250,251],{},"The two roles most often confused with an FDE each have a full comparison:",[253,254],"article-cheatsheet-card",{"cta":255,"description":256,"href":198,"image":257,"imageAlt":258,"label":259,"title":260},"Open the comparison","Production code or architecture breadth compared: pay, pre-sale work, outcome ownership, and when to hire each.","/misc/fde-vs-solutions-architect-poster.png","FDE vs Solutions Architect comparison preview","","FDE vs Solutions Architect",[253,262],{"cta":255,"description":263,"href":178,"image":264,"imageAlt":265,"label":259,"title":266},"Pre-sale demos or post-sale production code compared: pay, travel, the hand-off on one deal, and when to hire each.","/misc/fde-vs-sales-engineer-poster.png","FDE vs Sales Engineer comparison preview","FDE vs Sales Engineer",[11,268,269,270,275],{},"The Pragmatic Engineer's ",[15,271,274],{"href":272,"rel":273},"https://newsletter.pragmaticengineer.com/p/forward-deployed-engineers",[19],"2025 reporting on OpenAI"," shows the line in practice: OpenAI solutions architects tend to build MVPs offline with anonymized data, and FDEs write code directly on customer infrastructure. That habit of working on the real system goes back to the role's origins at Palantir.",[70,277,279],{"id":278},"where-the-model-came-from","Where the model came from",[11,281,282],{},"Palantir built the model for customers whose requirements could not be written down. Intelligence and defense work involved undocumented schemas, classified context, and workflows that headquarters engineers would never see. Embedding engineers on site was the only practical way to discover what \"working software\" meant.",[11,284,285,286,291],{},"Sankar tells the origin story himself. According to ",[15,287,290],{"href":288,"rel":289},"https://www.theregister.com/channel/2026/10/03/palantirs-fondness-for-french-food-cooked-up-techs-latest-fad-forward-deployed-engineers/5300360",[19],"The Register"," (October 3, 2026), he says he coined the term in 2007 after CEO Alex Karp asked why French restaurants are so good. Karp's answer was that the wait staff are part of the kitchen and know its techniques like the cooks do. Sankar says Karp asked him to build that for engineering. Treat this as founder lore: the only source is Sankar's own account.",[11,293,294],{},"Two structural details from Palantir carried into the current wave:",[296,297,298,310],"ol",{},[54,299,300,303,304,309],{},[35,301,302],{},"The paired unit."," Palantir pairs the Delta with an \"Echo,\" a deployment strategist who is often not a software engineer. Practitioner accounts describe Echoes as domain operators, such as former officers or clinicians, who handle stakeholders, adoption, and account shape. Palantir's own ",[15,305,308],{"href":306,"rel":307},"https://blog.palantir.com/a-day-in-the-life-of-a-palantir-deployment-strategist-951cb59a5a96",[19],"blog on the deployment strategist role"," describes the job, and the Delta and Echo labels still appear as team names on Palantir's own job board.",[54,311,312,315],{},[35,313,314],{},"Field-heavy staffing."," The Pragmatic Engineer reports that until about 2016, Palantir had more FDEs than regular software engineers. After Foundry launched that year, many moved into product engineering as the platform absorbed patterns they had built by hand.",[11,317,318],{},"That second point is the model in miniature: field work finds patterns, the product absorbs them, and the next deployment needs less custom code. AI agents added a third piece, shown on the right of the diagram: the code got cheap, and the context around it did not.",[23,320],{":height":321,":width":322,"alt":323,"loading":28,"provider":29,"src":324},"368","856","The forward deployed engineer model in three panels. A paired team unit of a Delta software engineer and an Echo domain expert who embed on site for unwritten context. A field flywheel of field discovery, pattern extraction, and product absorption. An AI deployment gap, where code is cheap and the hard parts are permissions, tools, and human guardrails.","/blog/forward-deployed-engineering-fde-guide/the-forward-deployed-engineer-model.svg",[70,326,328],{"id":327},"why-ai-agents-brought-the-model-back","Why AI agents brought the model back",[11,330,331],{},"Enterprise software has always had an implementation gap. Agents widen it. A chatbot needs a model and a knowledge base. An agent that changes records needs credentials, scoped permissions, tool definitions, a test suite of real tasks, rollback paths, and a process that knows when to stop and ask a person.",[11,333,334,335,340],{},"OpenAI FDE Luis Velasco described this at the Shift conference in September 2026, as ",[15,336,339],{"href":337,"rel":338},"https://shiftmag.dev/openai-fde-code-is-very-cheap-to-produce-12341/",[19],"reported by ShiftMag"," on October 2:",[342,343,344],"blockquote",{},[11,345,346],{},"\"Code is very cheap to produce. It's all about building the system around it.\"",[11,348,349,350],{},"He described that system as \"the system of intent, the guardrails, the constraints for the agent to work reliably,\" and called access to a company's data, procedures, and tools ",[35,351,352],{},"\"the deployment gap.\"",[11,354,355,356],{},"He also noted that agents now handle far longer tasks, from work a human would finish in about six minutes to work of around 16 hours. He was citing METR's published time-horizon benchmark, and the six-minute and 16-hour figures are his summary of it. ",[35,357,358],{},"Longer tasks touch more systems, so the deployment gap grows with them.",[118,360,362],{"id":361},"the-production-gap-in-numbers","The production gap in numbers",[11,364,365],{},"Several figures circulate about agent failure. Each measures something different, so read them one at a time:",[51,367,368,380,386],{},[54,369,370,373,374,379],{},[35,371,372],{},"Gartner forecast, June 25, 2025."," In a ",[15,375,378],{"href":376,"rel":377},"https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027",[19],"press release",", Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, or inadequate risk controls. The 40% is a prediction about 2027. A January 2025 Gartner poll of 3,412 webinar attendees in the same release found 19% had made sizeable agentic investments, 42% conservative ones, 8% none, and 31% were waiting or unsure.",[54,381,382,385],{},[35,383,384],{},"\"Agent washing.\""," In the same research, Gartner estimated that only about 130 of the thousands of vendors marketing agentic AI offer genuine agentic capability. The number is an analyst estimate.",[54,387,388,391,392,395],{},[35,389,390],{},"MIT NANDA, 2025."," The ",[84,393,394],{},"GenAI Divide"," report found that about 95% of organizations saw no measurable return from generative AI pilots. Its method combined a review of 300+ publicly disclosed initiatives, interviews with 52 organizations, and a survey of 153 leaders. The result rests on disclosures and a survey, and it covers all generative AI pilots, agents among them.",[11,397,398],{},"Figures such as \"88% of agent pilots never reach production\" circulate in blog posts without a traceable primary report; treat them as secondary claims.",[11,400,401],{},"Forward deployed engineering is the vendors' answer to that gap. The right side of the image below condenses the FDE's work into four stages; the next section breaks them into the full eight-step loop.",[23,403],{":height":404,":width":26,"alt":405,"loading":28,"provider":29,"src":406},"450","Crossing the agent production gap. On the left, three figures about the gap, 95% of pilots without ROI, 40% canceled by 2027, and about 130 genuine agent vendors out of thousands marketing agentic capabilities. On the right, a four-step FDE loop of scoping the workflow, building evals first, building tools and identity, and handoff and scale, with evals first and handoff marked critical.","/blog/forward-deployed-engineering-fde-guide/crossing-the-agent-production-gap.svg",[70,408,410],{"id":409},"the-fde-operating-loop","The FDE operating loop",[11,412,413],{},"The loop below reflects how OpenAI, Anthropic, and Palantir describe their deployments, plus common practitioner writeups. Real engagements compress or repeat steps, but the order holds.",[296,415,416,422,428,434,440,446,452,458],{},[54,417,418,421],{},[35,419,420],{},"Embed and scope."," Spend the first days on site with the people who do the work as well as the sponsor. The Pragmatic Engineer describes OpenAI FDEs starting with a short on-site scoping phase.",[54,423,424,427],{},[35,425,426],{},"Map the real workflow."," The documented process is almost always shorter than the real one. Capture exceptions, side spreadsheets, approvals that live in someone's head, and the systems people copy data between.",[54,429,430,433],{},[35,431,432],{},"Sort every step."," Each step lands in one of four buckets: delete it, automate it with plain rules, give it to an agent, or keep it human. Many steps should never reach a model.",[54,435,436,439],{},[35,437,438],{},"Build the eval set before the agent."," Collect real historical cases with known correct outcomes. This becomes the definition of \"working\" that both sides sign off on.",[54,441,442,445],{},[35,443,444],{},"Build inside the customer's tools."," Agents call the customer's existing systems through defined tools, and those systems stay in place. This is where MCP servers and tool schemas get written.",[54,447,448,451],{},[35,449,450],{},"Productionize identity and permissions."," Give the agent its own identity, least-privilege scopes, secrets management, audit logs, and tracing from day one.",[54,453,454,457],{},[35,455,456],{},"Hand off."," Transfer code, infrastructure, evals, and runbooks to a named internal owner, and train that team to change the system without the vendor.",[54,459,460,463],{},[35,461,462],{},"Generalize."," Report which parts were custom and which were repeatable. OpenAI's Deployment Company page calls this cycle \"build, prove, generalize,\" with recurring patterns moving into products such as its Agent SDK, authoring systems, and reliability tools.",[23,465],{":height":466,":width":467,"alt":468,"loading":28,"provider":29,"src":469},"568","876","The FDE operating loop as eight connected steps in two rows. Embed and scope, map workflow, sort every step, and build eval set across the top, then customer tools, productionize, hand off, and generalize back along the bottom. Build eval set and generalize are marked often skipped, and a product feedback arrow runs from generalize back to the start.","/blog/forward-deployed-engineering-fde-guide/the-fde-operating-loop.svg",[11,471,472],{},"Steps 4 and 8 are the ones most often skipped:",[51,474,475,481],{},[54,476,477,480],{},[35,478,479],{},"Skipping step 4"," means nobody can prove the agent works.",[54,482,483,486],{},[35,484,485],{},"Skipping step 8"," means the vendor is running a services business with a product attached.",[118,488,490],{"id":489},"what-a-typical-week-looks-like","What a typical week looks like",[11,492,493],{},"Engagement shapes vary widely:",[51,495,496,502,507],{},[54,497,498,501],{},[35,499,500],{},"Palantir"," historically kept engineers on site for months.",[54,503,504,506],{},[35,505,20],{}," describes 45-day sprint cycles, and coverage of the launch reported pods of about five or six engineers per client.",[54,508,509,512],{},[35,510,511],{},"Anthropic's"," Applied AI FDE listing calls for 25-50% travel to customer sites, so many lab FDEs split time between customer sites and remote work.",[11,514,515],{},"Inside a sprint, a common rhythm looks like this:",[51,517,518,524,530],{},[54,519,520,523],{},[35,521,522],{},"Early in the week:"," sessions with operators and a review of last week's eval failures.",[54,525,526,529],{},[35,527,528],{},"Midweek:"," build time, including permission requests that need the customer's security team.",[54,531,532,535],{},[35,533,534],{},"Late in the week:"," a sponsor review against the agreed metric and a written note to the vendor's product team on what the platform could not do.",[11,537,538],{},"Those notes are the raw material for step 8.",[11,540,541],{},"The build time in the middle of the week goes into a production stack, and that stack is what the customer inherits.",[70,543,545],{"id":544},"the-technical-stack-an-fde-builds","The technical stack an FDE builds",[11,547,548],{},"Practitioner writeups in 2026 describe layered production stacks, and the exact layers vary by vendor. The four-layer version below is a synthesis of those accounts, read from the bottom up.",[23,550],{":height":551,":width":552,"alt":553,"loading":28,"provider":29,"src":554},"490","926","The FDE technical stack. Four production layers stacked from the bottom, environment provisioning as code in the customer's cloud tenant, multi-step simulation with mocked APIs, governed deployment with a dedicated agent identity, scoped permissions and full call audit, and the eval suite with model routing. Beside them, a handoff package of MCP servers and schemas, sub-agents and skills, infrastructure as code and eval test suites, a semantic knowledge layer, and no-API web drivers.","/blog/forward-deployed-engineering-fde-guide/the-fde-technical-stack.svg",[118,556,558],{"id":557},"environment-provisioning","Environment provisioning",[11,560,561,562,565],{},"Everything runs in the customer's cloud account or tenant, ",[35,563,564],{},"defined as code",". Terraform or Helm templates create the runtime, connect to SSO, wire up a secrets manager, and isolate each business unit or customer. An environment that exists only as an FDE's console clicks is one the customer cannot rebuild.",[118,567,569],{"id":568},"simulation","Simulation",[11,571,572],{},"Before an agent touches production, it runs multi-step tasks against mocked tools that mirror the real API shapes. Simulation catches loops, bad retries, and destructive sequences without side effects.",[118,574,576],{"id":575},"governed-deployment","Governed deployment",[11,578,579],{},"Agents need identities the way service accounts do. In practice this means:",[51,581,582,588,594],{},[54,583,584,587],{},[35,585,586],{},"A dedicated agent identity"," in the customer's identity provider, separate from every human account.",[54,589,590,593],{},[35,591,592],{},"Scoped permissions"," per tool, with read and write separated and high-risk actions behind a human approval step.",[54,595,596,599],{},[35,597,598],{},"Observability from day one:"," traces of every model call and tool call, linked to the business record it touched.",[118,601,603],{"id":602},"the-eval-suite","The eval suite",[11,605,606,609],{},[35,607,608],{},"The eval set is the contract in executable form."," A useful suite holds a few hundred real cases per workflow, scores both the final outcome and the intermediate tool calls, and runs on every prompt, model, or tool change. It tracks cost and latency per task alongside accuracy, which makes model routing a measured decision: use the cheapest model that clears the bar for each step.",[11,611,612,613],{},"Together with simulation, it covers both sides of readiness: ",[35,614,615],{},"simulation measures safety, and the eval suite measures quality.",[11,617,618],{},"The right side of the diagram, the handoff package, is what the customer keeps once the stack is running.",[70,620,622],{"id":621},"artifacts-what-should-exist-after-the-fde-leaves","Artifacts: what should exist after the FDE leaves",[11,624,625],{},"The clearest change between Palantir-era and agent-era forward deployment is the deliverable list. Anthropic's Applied AI FDE listing asks engineers to \"deliver technical artifacts for customers like MCP servers, sub-agents, and agent skills that will be used in production workflows,\" and to \"identify and codify repeatable deployment patterns.\"",[11,627,628],{},"A complete handoff package usually contains:",[51,630,631,637,643,649,655,661],{},[54,632,633,636],{},[35,634,635],{},"MCP servers and tool schemas"," that wrap the customer's internal systems with typed inputs, clear descriptions, and permission checks enforced in the server code, where a prompt cannot override them.",[54,638,639,642],{},[35,640,641],{},"Sub-agents and skills"," for repeatable tasks, versioned like code.",[54,644,645,648],{},[35,646,647],{},"Eval sets and the runner"," that scores them.",[54,650,651,654],{},[35,652,653],{},"Infrastructure-as-code"," in the customer's account.",[54,656,657,660],{},[35,658,659],{},"Runbooks"," for incidents, model upgrades, and permission changes.",[54,662,663,666],{},[35,664,665],{},"A semantic layer"," where needed. AWS says its teams leave behind a knowledge graph of the customer's business entities and processes, which AWS's Francessca Vasquez describes as \"a customer's ontology of their information.\" SiliconANGLE called this a \"system of intelligence\" for agents.",[11,668,669,672],{},[35,670,671],{},"MCP matters here for a practical reason."," A tool defined once as an MCP server can be reused by any compliant agent client, so the customer keeps using it after switching models or vendors. Tool schemas written as functions inside one vendor's framework stay tied to that framework.",[118,674,676],{"id":675},"when-there-is-no-api","When there is no API",[11,678,679,680,684],{},"Some workflows live in web applications the customer uses but does not own: a supplier portal, a government filing site, a legacy SaaS tool with no usable API. FDEs can wait for an integration that may never come, or have the agent operate the web interface directly. ",[15,681,683],{"href":682},"/blog/how-do-i-build-an-ai-agent-that-works-on-a-website-i-dont-own","Webfuse is one option for the second path",": a live-session MCP that lets an agent work inside a live web session of a site the vendor does not control, with no change to that site's code. Browser-automation frameworks and cloud browsers are the other common choices, with different trade-offs on authentication and human handoff.",[11,686,687],{},"That is the work. The next question is who is paying for it, and at what scale.",[70,689,691],{"id":690},"who-is-building-fde-teams-in-2026","Who is building FDE teams in 2026",[11,693,694],{},"Hiring and capital figures in this section come from company announcements and press coverage; none are audited.",[23,696],{":height":697,":width":698,"alt":699,"loading":28,"provider":29,"src":700},"557","935","Who is building FDE teams in 2026. Six cards, OpenAI with $4 billion plus and 150 plus FDEs from the Tomoro acquisition, Microsoft with $2.5 billion and 6,000 embedded experts, Anthropic with a $1.5 billion enterprise unit backed by Blackstone, Hellman & Friedman and Goldman, AWS with a $1 billion FDE organization on 45-day cycles, Google Cloud with a $750 million partner fund, and Salesforce with a 1,000 FDE target. A bottom strip shows FDE job postings up 729% year over year, Palantir's $4.48 billion FY25 revenue up 56%, and a U.S. median total compensation of $216K for FDEs on Levels.fyi.","/blog/forward-deployed-engineering-fde-guide/who-is-building-fde-teams-in-2026.svg",[11,702,703],{},"Where each card comes from, with the details that did not fit on it:",[51,705,706,712,718,723,729,735],{},[54,707,708,711],{},[35,709,710],{},"OpenAI"," announced the OpenAI Deployment Company on May 11, 2026. OpenAI holds a majority stake, TPG led the initial investment, and the Tomoro acquisition brings about 150 FDEs and deployment specialists.",[54,713,714,717],{},[35,715,716],{},"Anthropic"," announced its enterprise AI services firm with Blackstone, Hellman & Friedman, and Goldman Sachs on May 4, 2026. The $1.5 billion figure comes from CNBC's reporting.",[54,719,720,722],{},[35,721,20],{}," launched its Forward Deployed Engineering organization on June 30, 2026, per SiliconANGLE. Named customers include the Allen Institute for AI, Cox Automotive, the NBA, the NFL, and Ricoh.",[54,724,725,728],{},[35,726,727],{},"Microsoft"," announced Microsoft Frontier Company on July 2, 2026, working with partners such as Accenture and PwC. Land O'Lakes and Unilever are among the early customers.",[54,730,731,734],{},[35,732,733],{},"Google Cloud"," is hiring \"hundreds\" of FDEs inside its go-to-market AI organization, according to 2026 press reports, alongside the partner-ecosystem commitment.",[54,736,737,740],{},[35,738,739],{},"Salesforce"," \"has committed to building a team of 1,000 FDEs\" for Agentforce, according to its blog in November 2025 (APAC edition dated March 27, 2026).",[118,742,744],{"id":743},"how-the-labs-fund-it","How the labs fund it",[11,746,747,748,753,754,757,758,761,762,767],{},"The OpenAI deal terms show how a lab pays for people-heavy work. ",[15,749,752],{"href":750,"rel":751},"https://www.axios.com/2026/05/11/openai-deployco-private-equity",[19],"Axios"," reported a ",[35,755,756],{},"$10 billion pre-money valuation"," and a ",[35,759,760],{},"guaranteed minimum 17.5% return"," to investors, with profits capped. ",[15,763,766],{"href":764,"rel":765},"https://siliconangle.com/2026/05/11/openai-launches-professional-services-business-4b-investment/",[19],"SiliconANGLE"," named Tomoro's prior clients as including Tesco and Virgin Atlantic. A guaranteed-return structure looks more like infrastructure finance than venture capital, and placing services in a separate vehicle keeps lower-margin work off the lab's own books.",[11,769,770,771,776],{},"Investors had already made the case for accepting that margin. a16z partner Joe Schmidt's ",[15,772,775],{"href":773,"rel":774},"https://a16z.com/services-led-growth/",[19],"\"Trading Margin for Moat\""," (June 4, 2025) called the FDE \"the hottest job in startups.\" His argument: AI applications should become the systems of work that generate and store company data, and young companies should accept less than 80% gross margin when that buys them ownership of the workflow. He cited ServiceNow's 63.2% and Workday's 54.1% gross margins at IPO as precedent.",[118,778,780],{"id":779},"job-postings","Job postings",[11,782,783,784,787],{},"The hiring follows the money, from a tiny base. Business Insider, citing job-board data, reported that FDE postings in April 2026 sat about 5,230% above January 2025 levels, against 543% above in April 2025, which works out to the ",[35,785,786],{},"roughly 729% year-over-year increase"," on the card above. Other trackers report growth above 1,000% over different windows. When a January 2025 baseline is near zero, percentages inflate quickly; the installed base of FDEs remains small next to ordinary software engineering.",[118,789,791],{"id":790},"palantir-as-the-reference-point","Palantir as the reference point",[11,793,794,795,800],{},"Palantir's results are the strongest public evidence that the model can scale financially. Its ",[15,796,799],{"href":797,"rel":798},"https://www.sec.gov/Archives/edgar/data/1321655/000132165526000004/a2025q4ex991earningsrelease.htm",[19],"FY2025 earnings release"," and later results show the growth:",[51,802,803,809,815],{},[54,804,805,808],{},[35,806,807],{},"FY2025 revenue:"," $4.475 billion, up 56% from $2.865 billion in 2024.",[54,810,811,814],{},[35,812,813],{},"Q2 2026 revenue:"," $1.935 billion, up 93%.",[54,816,817,820],{},[35,818,819],{},"Q2 2026 U.S. commercial revenue:"," $764 million, up 149%.",[11,822,823],{},"Its 2024 10-K lists short AIP bootcamps on customer data as a sales approach and warns that converting those pilots into long-term contracts is not guaranteed. Bootcamp conversion rates quoted online are company-reported or secondary.",[118,825,827],{"id":826},"what-fdes-earn","What FDEs earn",[11,829,830,835],{},[15,831,834],{"href":832,"rel":833},"https://www.levels.fyi/t/software-engineer/title/forward-deployed-engineer/locations/united-states",[19],"Levels.fyi"," and job postings give the current picture:",[51,837,838,844,850],{},[54,839,840,843],{},[35,841,842],{},"U.S. median total compensation:"," $216,250 for the FDE title on Levels.fyi in early October 2026, with the 25th to 75th percentile at about $179,000 to $290,000.",[54,845,846,849],{},[35,847,848],{},"AI lab base pay:"," Anthropic's Applied AI FDE listing posted a base salary range of $200,000 to $300,000 when we checked in October 2026 (it may since have closed).",[54,851,852,855],{},[35,853,854],{},"Sales quota:"," most postings carry none.",[11,857,858],{},"Treat all of these as self-reported or posted figures.",[11,860,861],{},"At those salaries, each FDE is an expensive hire, which makes the next decision a real one for any company: build the function or buy it.",[70,863,865],{"id":864},"how-to-build-or-buy-an-fde-function","How to build or buy an FDE function",[11,867,868],{},"Hiring is the hard part. The profile combines three things:",[51,870,871,877,883],{},[54,872,873,876],{},[35,874,875],{},"A generalist production engineer:"," backend, APIs, cloud, data pipelines, SSO and RBAC.",[54,878,879,882],{},[35,880,881],{},"Shipped agent systems:"," tool use, MCP, evals, tracing.",[54,884,885,888],{},[35,886,887],{},"Process sense:"," as comfortable changing an operator's process as changing a prompt.",[11,890,891],{},"Sankar's description of the ideal Palantir candidate, quoted by The Register, is a person \"crazy enough to get on a last-minute plane to Iraq,\" \"smart enough to ship quality, same-day code,\" and with EQ \"high enough to talk to users.\"",[11,893,894,897],{},[35,895,896],{},"The judgment calls matter more than any single skill:"," what to leave manual, what to productize versus leave custom, and when to kill a pilot. Pair each FDE with a domain or deployment lead, Palantir's Echo pattern, so a second person owns stakeholders and adoption.",[118,899,901],{"id":900},"real-engagement-or-rebranded-services-contract","Real engagement or rebranded services contract?",[11,903,904],{},"Buyers face the mirror-image question: is the vendor's FDE offer the real thing? A few questions sort the two quickly.",[126,906,907,920],{},[129,908,909],{},[132,910,911,914,917],{},[135,912,913],{},"Question",[135,915,916],{},"Real FDE engagement",[135,918,919],{},"Rebranded services",[151,921,922,933,944,955,966],{},[132,923,924,927,930],{},[156,925,926],{},"Where does the code run?",[156,928,929],{},"In your account, defined as code",[156,931,932],{},"In the vendor's environment or a consultant's laptop",[132,934,935,938,941],{},[156,936,937],{},"How is success measured?",[156,939,940],{},"An eval set and a business metric agreed up front",[156,942,943],{},"Hours, milestones, or a demo",[132,945,946,949,952],{},[156,947,948],{},"What is left behind?",[156,950,951],{},"Evals, IaC, MCP servers or tool schemas, runbooks, a trained owner",[156,953,954],{},"Documentation and a renewal quote",[132,956,957,960,963],{},[156,958,959],{},"Does the vendor product change?",[156,961,962],{},"Yes, field findings reach the roadmap",[156,964,965],{},"No visible link",[132,967,968,971,973],{},[156,969,970],{},"Can you run it without the vendor?",[156,972,171],{},[156,974,975],{},"Not without a new contract",[11,977,978,981],{},[35,979,980],{},"The practical test is the last row."," If the honest answer is no, the engagement has built a dependency, whatever the job titles say. Even a real engagement carries risks, on both sides of the contract.",[70,983,985],{"id":984},"risks-and-failure-modes","Risks and failure modes",[11,987,988,991],{},[35,989,990],{},"Rebrand risk."," Wikipedia notes the role overlaps with solutions architects, sales engineers, professional services, and IT consulting, and cites a management professor who argued the unconventional title projects organizational identity more than it reflects a new occupation. The Register reported that cloud vendor DoIT has used embedded engineers since 2011 and adopted the FDE label only in Q4 2025.",[11,993,994,997],{},[35,995,996],{},"Margin and scale."," Headcount grows with every new customer. Palantir carried this cost for years before its platform absorbed enough patterns. Analyst Jay McBain, quoted by The Register, forecast that FDE investments will \"fall back into the channel by 2031 when this matures.\"",[11,999,1000,1003],{},[35,1001,1002],{},"Talent brittleness."," Travel, ambiguity, short deadlines, and customer politics make the role unattractive to many engineers, and security-clearance requirements shrink the defense talent pool further. \"Hottest job\" rankings also overstate how many experienced FDEs exist today.",[11,1005,1006,1009],{},[35,1007,1008],{},"Lock-in by accident."," An FDE who moves fast without IaC, evals, or a named owner leaves a system only the vendor can operate. The usual cause is deadline pressure. Contracts should list the handoff artifacts as deliverables.",[70,1011,1013],{"id":1012},"what-to-take-from-this","What to take from this",[11,1015,1016,1017],{},"Every lab and cloud vendor sells access to the same class of models. What separates a working agent from a canceled pilot is the work inside each customer: permissions, tools, evals, and process changes. Vendors that send engineers to do that work, write it down as reusable artifacts, and feed it into their products will compound their lead. Buyers can check which kind of vendor they have with one question: ",[35,1018,1019],{},"what will we own when the team leaves?",{"title":259,"searchDepth":1021,"depth":1021,"links":1022},2,[1023,1027,1028,1031,1034,1040,1043,1049,1052,1053],{"id":72,"depth":1021,"text":73,"children":1024},[1025],{"id":120,"depth":1026,"text":121},3,{"id":278,"depth":1021,"text":279},{"id":327,"depth":1021,"text":328,"children":1029},[1030],{"id":361,"depth":1026,"text":362},{"id":409,"depth":1021,"text":410,"children":1032},[1033],{"id":489,"depth":1026,"text":490},{"id":544,"depth":1021,"text":545,"children":1035},[1036,1037,1038,1039],{"id":557,"depth":1026,"text":558},{"id":568,"depth":1026,"text":569},{"id":575,"depth":1026,"text":576},{"id":602,"depth":1026,"text":603},{"id":621,"depth":1021,"text":622,"children":1041},[1042],{"id":675,"depth":1026,"text":676},{"id":690,"depth":1021,"text":691,"children":1044},[1045,1046,1047,1048],{"id":743,"depth":1026,"text":744},{"id":779,"depth":1026,"text":780},{"id":790,"depth":1026,"text":791},{"id":826,"depth":1026,"text":827},{"id":864,"depth":1021,"text":865,"children":1050},[1051],{"id":900,"depth":1026,"text":901},{"id":984,"depth":1021,"text":985},{"id":1012,"depth":1021,"text":1013},"enterprise-ai","2026-10-05","How forward deployed engineering works in 2026: the operating loop, the artifacts an FDE should leave behind, evals, agent identity, MCP tooling, economics, and how to tell a real engagement from a rebranded services contract.","md",[1059,1062,1065,1068,1071],{"question":1060,"answer":1061},"What is forward deployed engineering?","Forward deployed engineering is a delivery model in which a vendor's software engineers embed with a customer, write production code against the customer's real data and systems, and feed what they learn back into the vendor's product. Palantir popularized the role. In 2026 OpenAI, Anthropic, AWS, Microsoft and Google Cloud all announced FDE organizations or embedded-engineer units focused on AI agents.",{"question":1063,"answer":1064},"How is a forward deployed engineer different from a solutions engineer?","A solutions or sales engineer usually works before the sale, builds demos and proofs of concept, and does not own production code. An FDE works after the contract is signed, ships code that runs in the customer's environment, owns a business outcome, and reports product gaps back to the vendor's engineering team.",{"question":1066,"answer":1067},"Why did forward deployed engineering come back with AI agents?","An agent's value depends on things a model vendor cannot ship in a box: access to the customer's data and tools, scoped permissions, evals against real tasks, and a redesigned process with human checkpoints. Gartner forecast in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027. Embedded engineers are the vendors' answer to that production gap.",{"question":1069,"answer":1070},"How much do forward deployed engineers earn?","Levels.fyi listed a U.S. median total compensation of $216,250 for the title in early October 2026, with the 25th to 75th percentile at about $179,000 to $290,000. Pay at AI labs and large cloud vendors sits higher. These are self-reported and posted figures, not audited payroll data.",{"question":1072,"answer":1073},"How can a buyer tell a real FDE engagement from a rebranded consulting contract?","Ask what the customer can run without the vendor once the engagement ends. A real engagement leaves infrastructure-as-code in the customer's account, eval sets, tool schemas or MCP servers, runbooks, and a named internal owner. On the vendor side, it produces product changes. A rebranded contract leaves a dependency on the vendor's people and a roadmap that never changes.",0,null,false,{"shortTitle":1078,"relatedLinks":1079},"Forward Deployed Engineering Guide",[1080,1084,1088],{"text":1081,"href":1082,"description":1083},"Forward Deployed Engineer vs Software Engineer (2026)","/blog/forward-deployed-engineer-vs-software-engineer","Pay, daily rhythm, customer time, and code ownership compared for FDEs and product engineers.",{"text":1085,"href":1086,"description":1087},"Human Approval Checkpoints for AI Agents (2026)","/blog/human-approval-checkpoints-for-ai-agents","Where to pause an agent before a high-stakes action and hand the decision to a person.",{"text":1089,"href":1090,"description":1091},"How to Connect an AI Agent to the Live Web: 5 Architectures Compared (2026)","/blog/how-to-connect-an-ai-agent-to-the-live-web","The five execution layers for giving an agent access to the web, and how to pick between them.",true,"/blog/forward-deployed-engineering-fde-guide",{"title":5,"description":1056},{"loc":1093},"blog/1055.forward-deployed-engineering-fde-guide",[1098,1099,1100,1054,1101],"forward-deployed-engineering","fde","ai-agents","mcp","FVo5unoRfpHvjtZ8Izs37LvsfjQryil63TcbaPevN6Y",[1104,3821],{"id":1105,"title":1106,"authorId":1107,"body":1108,"category":1100,"created":3801,"description":3802,"extension":1057,"faqs":1075,"featurePriority":1075,"head":1075,"hideCta":1092,"landingPath":1075,"meta":3803,"navigation":1092,"ogImage":1075,"path":3811,"robots":1075,"schemaOrg":1075,"seo":3812,"sitemap":3813,"stem":3814,"tags":3815,"__hash__":3820},"blog/blog/1013.serialising-web-ui-state-for-llms-the-complete-guide.md","Serialising Web UI State for LLMs: The Complete Guide","thassilo-schiepanski",{"type":8,"value":1109,"toc":3784},[1110,1117,1139,1143,1146,1150,1153,1156,1165,1169,1173,1180,1199,1203,1206,1209,1213,1220,1224,1229,1232,1249,1253,1271,1275,1287,1301,1305,1319,1323,1326,1330,1348,1351,1355,1365,1370,1444,1447,1450,1455,1526,1531,1650,1655,1764,1768,1779,1784,2086,2091,2245,2254,2258,2269,2274,2278,2282,2285,2288,2291,2398,2402,2405,2409,2412,2415,2420,2644,2649,2901,2906,2982,2987,2994,3003,3007,3010,3013,3016,3021,3030,3034,3374,3394,3447,3451,3454,3486,3634,3638,3641,3644,3648,3659,3662,3673,3679,3780],[11,1111,1112,1113,1116],{},"LLMs have become a core component of AI web browsing agents. Previously, automating a web journey required formal models of both web browsing and web-based user interfaces (UIs). An LLM can serve as a plug-in reasoning backend: given serialised web UI state, it is expected to suggest input actions that progress a specified web browsing task. Serialised UI state is, more ergonomically, referred to as a ",[35,1114,1115],{},"snapshot",".",[11,1118,1119,1120,1125,1126,1131,1132,1125,1135,1138],{},"Snapshots and action suggestions are the two key artefacts along the agent loop. Action suggestions are elicited from the web agent LLM backend. Suggestions can be acted out via automated web browser environments, such as ",[15,1121,1124],{"href":1122,"rel":1123},"https://playwright.dev",[19],"Playwright"," or ",[15,1127,1130],{"href":1128,"rel":1129},"https://www.webfuse.com",[19],"Webfuse",", for example, ",[84,1133,1134],{},"clicking",[84,1136,1137],{},"typing",". The snapshot, on the other hand, is an artefact created by the agent harness, that is, the individual application component between the LLM and the web browser.",[23,1140],{"src":1141,":alt":1142},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/1.svg","{\"Figurative depiction of two key artefacts along the agent loop\":\"the snapshot towards the LLM and the input action towards the browser\",\"provider\":\"none\",\"loading\":\"lazy\",\"width\":460}",[11,1144,1145],{},"Creating snapshots for LLMs is a central challenge of developing efficient web agents. It is a trade-off between representation utility – how well models can interpret the such encoded UI – and LLM token costs. Snapshots are the major cost factor of an agentic web journey, accumulated over multiple rounds of the loop.",[70,1147,1149],{"id":1148},"gui-snapshots","GUI Snapshots",[11,1151,1152],{},"Screenshots, i.e., GUI snapshots, resemble what humans see when viewing a web page at a given point in time. Early LLM-based web agents were primarily premised on GUI snapshots because of this evident equivalence. By design, the GUI surfaces only the UI context designated to be visible to the user.",[11,1154,1155],{},"The document object model (DOM) is a web browser's model of a web application UI's runtime state. With few exceptions, the DOM is the single source of truth regarding the UI. Most snapshot representations are thus, technically speaking, derivatives of a raw DOM snapshot. For the cases not reflected in the DOM, GUI snapshots are a first-class base representation.",[11,1157,1158,1159,1164],{},"Rendering a GUI snapshot from the DOM is, moreover, non-trivial; no client-side library manages to create GUI snapshots in reasonable quality or time – ",[15,1160,1163],{"href":1161,"rel":1162},"https://html2canvas.hertzen.com",[19],"HTML2Canvas"," demonstrates the limits.",[23,1166],{"src":1167,"alt":1168,"loading":28},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/2.png","The same web-based user interface serialised via browser-native screenshot API and the client-side library HTML2Canvas showing significant geometric differences",[118,1170,1172],{"id":1171},"vision-based-targeting","Vision-Based Targeting",[11,1174,1175,1176,1179],{},"The snapshot representation dictates how elements can be targeted in the live UI. Unambiguous targeting based on a visual representation must be either geometric (absolute), e.g., via point coordinates, or semantic (relative), e.g., via cues, such as ",[84,1177,1178],{},"\"Button with Text 'Shop'\"",". On its own, semantic targeting introduces a circular problem: which element has this text?",[11,1181,1182,1183,1186,1187,1190,1191,1194,1195,1116],{},"Consider the web browsing task: ",[84,1184,1185],{},"\"Navigate to the Apple online shop\"",". In combination with an action, in this case ",[84,1188,1189],{},"click",", the agent could target the ",[84,1192,1193],{},"Shop"," button via a point coordinate, e.g., ",[1196,1197,1198],"code",{},"(725, 750)",[23,1200],{"src":1201,"alt":1202,"loading":28},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/3.png","A GUI snapshot (screenshot) visualising point coordinate targeting",[11,1204,1205],{},"When the UI shifts between snapshot and action time, geometric targeting suggestions might have become obsolete.",[11,1207,1208],{},"Depending on the web browser environment, screenshots might be constrained to the GUI viewport. Scrolling is a secondary task that is imperative with viewport-contrained GUI snapshots.",[118,1210,1212],{"id":1211},"the-problem-with-gui-snapshots","The Problem with GUI Snapshots",[11,1214,1215,1216,1219],{},"Image data is downsampled to make it economically viable as input to LLMs. It reduces LLM input tokens to a few thousand for real-world-representative full-page screenshots. A common downsampling approach is averaging ",[84,1217,1218],{},"n×n","-tiles of pixels, as known from image downscaling. Side effects are: pixel-precise targeting does not map back to the live UI, and small geometric features are not preserved (e.g., small labels). Downsampling effects can be accounted for under certain circumstances, e.g., by providing a snapshot format that downsamples to its identity. However, GUI aspect ratio and detail might be beyond the agent's control.",[23,1221],{"src":1222,"alt":1223,"loading":28},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/4.png","A GUI snapshot (screenshot) compared to its figuratively downsampled counterpart",[1225,1226,1228],"h4",{"id":1227},"grounded-gui-snapshots","Grounded GUI Snapshots",[11,1230,1231],{},"Geometric targeting remains unreliable with certain models and responsive UIs. To rely on semantic targeting instead, the agent must link relevant elements in the live UI to semantics in the screenshot. Relevance usually equates with actionability, but ultimately depends on the snapshot's purpose.",[11,1233,1234,1237,1238,1248],{},[84,1235,1236],{},"Set-of-Mark"," prompting is the go-to technique to create grounded GUI snapshots",[1239,1240,1241],"sup",{},[15,1242,1247],{"href":1243,"ariaDescribedBy":1244,"dataFootnoteRef":259,"id":1246},"#user-content-fn-1",[1245],"footnote-label","user-content-fnref-1","1",". In this context, relevant elements are parsed from the live DOM and indexed with a unique numeric identifier. Each indexed element in the live UI is then enhanced with a coloured bounding box associated with the respective identifier.",[23,1250],{"src":1251,"alt":1252,"loading":28},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/5.png","A GUI snapshot (screenshot) visualising grounded cue targeting",[11,1254,1255,1262,1263,1116],{},[15,1256,1259],{"href":1257,"rel":1258},"https://browser-use.com",[19],[84,1260,1261],{},"Browser Use"," agents implement Set-of-Mark-grounded GUI snapshots",[1239,1264,1265],{},[15,1266,1270],{"href":1267,"ariaDescribedBy":1268,"dataFootnoteRef":259,"id":1269},"#user-content-fn-2",[1245],"user-content-fnref-2","2",[70,1272,1274],{"id":1273},"dom-snapshots","DOM Snapshots",[11,1276,1277,1278,1286],{},"The DOM is parsed from HTML. In its latest state it can be serialised back to HTML – including all runtime mutations. Research has supported LLMs' capabilities to interpret HTML, and even to navigate such encoded UIs",[1239,1279,1280],{},[15,1281,1285],{"href":1282,"ariaDescribedBy":1283,"dataFootnoteRef":259,"id":1284},"#user-content-fn-3",[1245],"user-content-fnref-3","3",". For that reason, DOM snapshots are a compelling alternative to GUI snapshots. LLMs excel at code generation and review, so these capabilities are no surprise.",[11,1288,1289,1290,1293,1294,1297,1298,1300],{},"Cross-origin ",[1196,1291,1292],{},"iframe"," elements and ",[1196,1295,1296],{},"canvas"," elements are exceptions to the DOM-is-source-of-truth assumption: ",[1196,1299,1296],{},"-drawn GUI is not reflected in the DOM, nor are external, self-contained DOM subtrees if no explicit policy allows it. Structurally, both are leaves in the serialisable DOM tree.",[118,1302,1304],{"id":1303},"dom-based-targeting","DOM-Based Targeting",[11,1306,1307,1308,1311,1312,1315,1316,1116],{},"Programmatically, DOM elements can be targeted by different relative means. CSS selectors abstract targeting with a standalone language. Whilst CSS selectors can resolve to more than one element in the DOM – compare semantics of the HTML ",[1196,1309,1310],{},"class"," and ",[1196,1313,1314],{},"id"," attributes – every element has at least one unique CSS selector, constructed from nth-child node paths, e.g., ",[1196,1317,1318],{},"main .section:nth-child(2) > .row:nth-of-type(5) > form button",[23,1320],{"src":1321,"alt":1322,"loading":28},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/6.png","A DOM snapshot (HTML) visualising CSS selector targeting",[11,1324,1325],{},"A raw DOM snapshot is complete, beyond the GUI that renders in the spatially limited browser viewport. For a human-like visual browsing experience, scrolling can be abstracted by an indirect scroll-target-into-view mechanism.",[118,1327,1329],{"id":1328},"the-problem-with-dom-snapshots","The Problem with DOM Snapshots",[11,1331,1332,1333,1338,1339,1347],{},"DOM snapshots of real-world web pages may come at a size of several megabytes, which converts to millions of LLM input tokens – we estimate a raw snapshot of ",[15,1334,1337],{"href":1335,"rel":1336},"https://edition.cnn.com",[19],"cnn.com"," at around 1.5M tokens. At this size, DOM snapshots over-utilise the context window of many LLMs, or even exceed it (such as of ",[15,1340,1343,1346],{"href":1341,"rel":1342},"https://openai.com/index/gpt-4o-system-card/",[19],[84,1344,1345],{},"GPT-4o"," (OpenAI)","). Ultimately, raw DOM snapshots out-cost GUI snapshots.",[11,1349,1350],{},"Using DOM snapshots with LLM-based web agents requires size-reducing transformations, similar to image downsampling implemented behind LLM APIs.",[1225,1352,1354],{"id":1353},"extracted-dom-snapshots","Extracted DOM Snapshots",[11,1356,1357,1358,1361,1362,1364],{},"The naïve approach to use information from the DOM as a snapshot is element extraction: relevant elements are copied from the live DOM to an initially empty virtual DOM. For small top-",[84,1359,1360],{},"k"," extractions – the ",[84,1363,1360],{}," highest-scoring elements – a snapshot of the virtual DOM comes at a fraction of the original DOM's size.",[11,1366,1367],{},[35,1368,1369],{},"Top-3 DOM Snapshot:",[1371,1372,1376],"pre",{"className":1373,"code":1374,"language":1375,"meta":259,"style":259},"language-html shiki shiki-themes github-dark github-dark","\u003Cbutton type=\"submit\">Submit\u003C/button>\n\u003Cbutton>More\u003C/button>\n\u003Ca href=\"/more\">Find out more\u003C/a>\n","html",[1196,1377,1378,1410,1423],{"__ignoreMap":259},[1379,1380,1383,1387,1391,1395,1398,1402,1405,1407],"span",{"class":1381,"line":1382},"line",1,[1379,1384,1386],{"class":1385},"suv1-","\u003C",[1379,1388,1390],{"class":1389},"sxg3X","button",[1379,1392,1394],{"class":1393},"sFR8T"," type",[1379,1396,1397],{"class":1385},"=",[1379,1399,1401],{"class":1400},"s4wv1","\"submit\"",[1379,1403,1404],{"class":1385},">Submit\u003C/",[1379,1406,1390],{"class":1389},[1379,1408,1409],{"class":1385},">\n",[1379,1411,1412,1414,1416,1419,1421],{"class":1381,"line":1021},[1379,1413,1386],{"class":1385},[1379,1415,1390],{"class":1389},[1379,1417,1418],{"class":1385},">More\u003C/",[1379,1420,1390],{"class":1389},[1379,1422,1409],{"class":1385},[1379,1424,1425,1427,1429,1432,1434,1437,1440,1442],{"class":1381,"line":1026},[1379,1426,1386],{"class":1385},[1379,1428,15],{"class":1389},[1379,1430,1431],{"class":1393}," href",[1379,1433,1397],{"class":1385},[1379,1435,1436],{"class":1400},"\"/more\"",[1379,1438,1439],{"class":1385},">Find out more\u003C/",[1379,1441,15],{"class":1389},[1379,1443,1409],{"class":1385},[11,1445,1446],{},"If done manually, element extraction presupposes correct assumptions about which elements and element properties count as relevant. Alternatively, element relevance scoring can be outsourced to an LLM, at the cost of another inference step.",[11,1448,1449],{},"Extraction disposes of deep DOM hierarchy, which might be salient to the web agent LLM backend. From the below extracted DOM snapshot alone, the original content relationship is lost:",[11,1451,1452],{},[35,1453,1454],{},"Extracted DOM Snapshot:",[1371,1456,1458],{"className":1373,"code":1457,"language":1375,"meta":259,"style":259},"\u003Cstrong>MacBook Air\u003C/strong>\n\u003Cbutton>Add to cart\u003C/button>\n\u003Cp>Out of stock\u003C/p>\n\u003Ch2>MacBook Pro\u003C/h2>\n\u003Cbutton>Add to cart\u003C/button>\n",[1196,1459,1460,1473,1486,1499,1513],{"__ignoreMap":259},[1379,1461,1462,1464,1466,1469,1471],{"class":1381,"line":1382},[1379,1463,1386],{"class":1385},[1379,1465,35],{"class":1389},[1379,1467,1468],{"class":1385},">MacBook Air\u003C/",[1379,1470,35],{"class":1389},[1379,1472,1409],{"class":1385},[1379,1474,1475,1477,1479,1482,1484],{"class":1381,"line":1021},[1379,1476,1386],{"class":1385},[1379,1478,1390],{"class":1389},[1379,1480,1481],{"class":1385},">Add to cart\u003C/",[1379,1483,1390],{"class":1389},[1379,1485,1409],{"class":1385},[1379,1487,1488,1490,1492,1495,1497],{"class":1381,"line":1026},[1379,1489,1386],{"class":1385},[1379,1491,11],{"class":1389},[1379,1493,1494],{"class":1385},">Out of stock\u003C/",[1379,1496,11],{"class":1389},[1379,1498,1409],{"class":1385},[1379,1500,1502,1504,1506,1509,1511],{"class":1381,"line":1501},4,[1379,1503,1386],{"class":1385},[1379,1505,70],{"class":1389},[1379,1507,1508],{"class":1385},">MacBook Pro\u003C/",[1379,1510,70],{"class":1389},[1379,1512,1409],{"class":1385},[1379,1514,1516,1518,1520,1522,1524],{"class":1381,"line":1515},5,[1379,1517,1386],{"class":1385},[1379,1519,1390],{"class":1389},[1379,1521,1481],{"class":1385},[1379,1523,1390],{"class":1389},[1379,1525,1409],{"class":1385},[11,1527,1528],{},[35,1529,1530],{},"Original DOM Snapshot – Candidate A:",[1371,1532,1534],{"className":1373,"code":1533,"language":1375,"meta":259,"style":259},"\u003Csection class=\"product\">\n  \u003Cstrong>MacBook Air\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n  \u003Cp>Out of stock\u003C/p>  👈\n\u003C/section>\n\u003Csection class=\"product\">\n  \u003Cstrong>MacBook Pro\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n\u003C/section>\n",[1196,1535,1536,1553,1566,1578,1591,1600,1615,1628,1641],{"__ignoreMap":259},[1379,1537,1538,1540,1543,1546,1548,1551],{"class":1381,"line":1382},[1379,1539,1386],{"class":1385},[1379,1541,1542],{"class":1389},"section",[1379,1544,1545],{"class":1393}," class",[1379,1547,1397],{"class":1385},[1379,1549,1550],{"class":1400},"\"product\"",[1379,1552,1409],{"class":1385},[1379,1554,1555,1558,1560,1562,1564],{"class":1381,"line":1021},[1379,1556,1557],{"class":1385},"  \u003C",[1379,1559,35],{"class":1389},[1379,1561,1468],{"class":1385},[1379,1563,35],{"class":1389},[1379,1565,1409],{"class":1385},[1379,1567,1568,1570,1572,1574,1576],{"class":1381,"line":1026},[1379,1569,1557],{"class":1385},[1379,1571,1390],{"class":1389},[1379,1573,1481],{"class":1385},[1379,1575,1390],{"class":1389},[1379,1577,1409],{"class":1385},[1379,1579,1580,1582,1584,1586,1588],{"class":1381,"line":1501},[1379,1581,1557],{"class":1385},[1379,1583,11],{"class":1389},[1379,1585,1494],{"class":1385},[1379,1587,11],{"class":1389},[1379,1589,1590],{"class":1385},">  👈\n",[1379,1592,1593,1596,1598],{"class":1381,"line":1515},[1379,1594,1595],{"class":1385},"\u003C/",[1379,1597,1542],{"class":1389},[1379,1599,1409],{"class":1385},[1379,1601,1603,1605,1607,1609,1611,1613],{"class":1381,"line":1602},6,[1379,1604,1386],{"class":1385},[1379,1606,1542],{"class":1389},[1379,1608,1545],{"class":1393},[1379,1610,1397],{"class":1385},[1379,1612,1550],{"class":1400},[1379,1614,1409],{"class":1385},[1379,1616,1618,1620,1622,1624,1626],{"class":1381,"line":1617},7,[1379,1619,1557],{"class":1385},[1379,1621,35],{"class":1389},[1379,1623,1508],{"class":1385},[1379,1625,35],{"class":1389},[1379,1627,1409],{"class":1385},[1379,1629,1631,1633,1635,1637,1639],{"class":1381,"line":1630},8,[1379,1632,1557],{"class":1385},[1379,1634,1390],{"class":1389},[1379,1636,1481],{"class":1385},[1379,1638,1390],{"class":1389},[1379,1640,1409],{"class":1385},[1379,1642,1644,1646,1648],{"class":1381,"line":1643},9,[1379,1645,1595],{"class":1385},[1379,1647,1542],{"class":1389},[1379,1649,1409],{"class":1385},[11,1651,1652],{},[35,1653,1654],{},"Original DOM Snapshot – Candidate B:",[1371,1656,1658],{"className":1373,"code":1657,"language":1375,"meta":259,"style":259},"\u003Csection class=\"product\">\n  \u003Cstrong>MacBook Air\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n\u003C/section>\n\u003Csection class=\"product\">\n  \u003Cp>Out of stock\u003C/p>  👈\n  \u003Cstrong>MacBook Pro\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n\u003C/section>\n",[1196,1659,1660,1674,1686,1698,1706,1720,1732,1744,1756],{"__ignoreMap":259},[1379,1661,1662,1664,1666,1668,1670,1672],{"class":1381,"line":1382},[1379,1663,1386],{"class":1385},[1379,1665,1542],{"class":1389},[1379,1667,1545],{"class":1393},[1379,1669,1397],{"class":1385},[1379,1671,1550],{"class":1400},[1379,1673,1409],{"class":1385},[1379,1675,1676,1678,1680,1682,1684],{"class":1381,"line":1021},[1379,1677,1557],{"class":1385},[1379,1679,35],{"class":1389},[1379,1681,1468],{"class":1385},[1379,1683,35],{"class":1389},[1379,1685,1409],{"class":1385},[1379,1687,1688,1690,1692,1694,1696],{"class":1381,"line":1026},[1379,1689,1557],{"class":1385},[1379,1691,1390],{"class":1389},[1379,1693,1481],{"class":1385},[1379,1695,1390],{"class":1389},[1379,1697,1409],{"class":1385},[1379,1699,1700,1702,1704],{"class":1381,"line":1501},[1379,1701,1595],{"class":1385},[1379,1703,1542],{"class":1389},[1379,1705,1409],{"class":1385},[1379,1707,1708,1710,1712,1714,1716,1718],{"class":1381,"line":1515},[1379,1709,1386],{"class":1385},[1379,1711,1542],{"class":1389},[1379,1713,1545],{"class":1393},[1379,1715,1397],{"class":1385},[1379,1717,1550],{"class":1400},[1379,1719,1409],{"class":1385},[1379,1721,1722,1724,1726,1728,1730],{"class":1381,"line":1602},[1379,1723,1557],{"class":1385},[1379,1725,11],{"class":1389},[1379,1727,1494],{"class":1385},[1379,1729,11],{"class":1389},[1379,1731,1590],{"class":1385},[1379,1733,1734,1736,1738,1740,1742],{"class":1381,"line":1617},[1379,1735,1557],{"class":1385},[1379,1737,35],{"class":1389},[1379,1739,1508],{"class":1385},[1379,1741,35],{"class":1389},[1379,1743,1409],{"class":1385},[1379,1745,1746,1748,1750,1752,1754],{"class":1381,"line":1630},[1379,1747,1557],{"class":1385},[1379,1749,1390],{"class":1389},[1379,1751,1481],{"class":1385},[1379,1753,1390],{"class":1389},[1379,1755,1409],{"class":1385},[1379,1757,1758,1760,1762],{"class":1381,"line":1643},[1379,1759,1595],{"class":1385},[1379,1761,1542],{"class":1389},[1379,1763,1409],{"class":1385},[1225,1765,1767],{"id":1766},"downsampled-dom-snapshots","Downsampled DOM Snapshots",[11,1769,1770,1771,1116],{},"Downsampling means reducing data point count whilst preserving the overall data shape and most of the relevant features. Picture an image being downsized: the depicted object remains recognisable to a large degree. The concept of image downsampling can be transferred to the DOM by locally 'averaging' DOM subtrees – trading fidelity for size",[1239,1772,1773],{},[15,1774,1778],{"href":1775,"ariaDescribedBy":1776,"dataFootnoteRef":259,"id":1777},"#user-content-fn-4",[1245],"user-content-fnref-4","4",[11,1780,1781],{},[35,1782,1783],{},"Original DOM Snapshot:",[1371,1785,1787],{"className":1373,"code":1786,"language":1375,"meta":259,"style":259},"\u003Csection class=\"container\" tabindex=\"3\" required=\"true\" type=\"example\">\n  \u003Cdiv class=\"mx-auto\" data-topic=\"products\" required=\"false\">\n    \u003Ch1>Our Pizza\u003C/h1>\n    \u003Cdiv>\n      \u003Cdiv class=\"shadow-lg\">\n        \u003Ch2>Margherita\u003C/h2>\n        \u003Cp>\n         A simple classic: mozzarella, tomatoes and basil.\n         An everyday choice!\n        \u003C/p>\n        \u003Cbutton type=\"button\">Add\u003C/button>\n      \u003C/div>\n      \u003Cdiv class=\"shadow-lg\">\n        \u003Ch2>Capricciosa\u003C/h2>\n        \u003Cp>\n          A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n          A true favourite!\n        \u003C/p>\n        \u003Cbutton type=\"button\">Add\u003C/button>\n      \u003C/div>\n    \u003C/div>\n  \u003C/div>\n\u003C/section>\n",[1196,1788,1789,1827,1858,1873,1881,1897,1911,1919,1924,1929,1939,1960,1970,1985,1999,2008,2014,2020,2029,2048,2057,2067,2077],{"__ignoreMap":259},[1379,1790,1791,1793,1795,1797,1799,1802,1805,1807,1810,1813,1815,1818,1820,1822,1825],{"class":1381,"line":1382},[1379,1792,1386],{"class":1385},[1379,1794,1542],{"class":1389},[1379,1796,1545],{"class":1393},[1379,1798,1397],{"class":1385},[1379,1800,1801],{"class":1400},"\"container\"",[1379,1803,1804],{"class":1393}," tabindex",[1379,1806,1397],{"class":1385},[1379,1808,1809],{"class":1400},"\"3\"",[1379,1811,1812],{"class":1393}," required",[1379,1814,1397],{"class":1385},[1379,1816,1817],{"class":1400},"\"true\"",[1379,1819,1394],{"class":1393},[1379,1821,1397],{"class":1385},[1379,1823,1824],{"class":1400},"\"example\"",[1379,1826,1409],{"class":1385},[1379,1828,1829,1831,1834,1836,1838,1841,1844,1846,1849,1851,1853,1856],{"class":1381,"line":1021},[1379,1830,1557],{"class":1385},[1379,1832,1833],{"class":1389},"div",[1379,1835,1545],{"class":1393},[1379,1837,1397],{"class":1385},[1379,1839,1840],{"class":1400},"\"mx-auto\"",[1379,1842,1843],{"class":1393}," data-topic",[1379,1845,1397],{"class":1385},[1379,1847,1848],{"class":1400},"\"products\"",[1379,1850,1812],{"class":1393},[1379,1852,1397],{"class":1385},[1379,1854,1855],{"class":1400},"\"false\"",[1379,1857,1409],{"class":1385},[1379,1859,1860,1863,1866,1869,1871],{"class":1381,"line":1026},[1379,1861,1862],{"class":1385},"    \u003C",[1379,1864,1865],{"class":1389},"h1",[1379,1867,1868],{"class":1385},">Our Pizza\u003C/",[1379,1870,1865],{"class":1389},[1379,1872,1409],{"class":1385},[1379,1874,1875,1877,1879],{"class":1381,"line":1501},[1379,1876,1862],{"class":1385},[1379,1878,1833],{"class":1389},[1379,1880,1409],{"class":1385},[1379,1882,1883,1886,1888,1890,1892,1895],{"class":1381,"line":1515},[1379,1884,1885],{"class":1385},"      \u003C",[1379,1887,1833],{"class":1389},[1379,1889,1545],{"class":1393},[1379,1891,1397],{"class":1385},[1379,1893,1894],{"class":1400},"\"shadow-lg\"",[1379,1896,1409],{"class":1385},[1379,1898,1899,1902,1904,1907,1909],{"class":1381,"line":1602},[1379,1900,1901],{"class":1385},"        \u003C",[1379,1903,70],{"class":1389},[1379,1905,1906],{"class":1385},">Margherita\u003C/",[1379,1908,70],{"class":1389},[1379,1910,1409],{"class":1385},[1379,1912,1913,1915,1917],{"class":1381,"line":1617},[1379,1914,1901],{"class":1385},[1379,1916,11],{"class":1389},[1379,1918,1409],{"class":1385},[1379,1920,1921],{"class":1381,"line":1630},[1379,1922,1923],{"class":1385},"         A simple classic: mozzarella, tomatoes and basil.\n",[1379,1925,1926],{"class":1381,"line":1643},[1379,1927,1928],{"class":1385},"         An everyday choice!\n",[1379,1930,1932,1935,1937],{"class":1381,"line":1931},10,[1379,1933,1934],{"class":1385},"        \u003C/",[1379,1936,11],{"class":1389},[1379,1938,1409],{"class":1385},[1379,1940,1942,1944,1946,1948,1950,1953,1956,1958],{"class":1381,"line":1941},11,[1379,1943,1901],{"class":1385},[1379,1945,1390],{"class":1389},[1379,1947,1394],{"class":1393},[1379,1949,1397],{"class":1385},[1379,1951,1952],{"class":1400},"\"button\"",[1379,1954,1955],{"class":1385},">Add\u003C/",[1379,1957,1390],{"class":1389},[1379,1959,1409],{"class":1385},[1379,1961,1963,1966,1968],{"class":1381,"line":1962},12,[1379,1964,1965],{"class":1385},"      \u003C/",[1379,1967,1833],{"class":1389},[1379,1969,1409],{"class":1385},[1379,1971,1973,1975,1977,1979,1981,1983],{"class":1381,"line":1972},13,[1379,1974,1885],{"class":1385},[1379,1976,1833],{"class":1389},[1379,1978,1545],{"class":1393},[1379,1980,1397],{"class":1385},[1379,1982,1894],{"class":1400},[1379,1984,1409],{"class":1385},[1379,1986,1988,1990,1992,1995,1997],{"class":1381,"line":1987},14,[1379,1989,1901],{"class":1385},[1379,1991,70],{"class":1389},[1379,1993,1994],{"class":1385},">Capricciosa\u003C/",[1379,1996,70],{"class":1389},[1379,1998,1409],{"class":1385},[1379,2000,2002,2004,2006],{"class":1381,"line":2001},15,[1379,2003,1901],{"class":1385},[1379,2005,11],{"class":1389},[1379,2007,1409],{"class":1385},[1379,2009,2011],{"class":1381,"line":2010},16,[1379,2012,2013],{"class":1385},"          A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[1379,2015,2017],{"class":1381,"line":2016},17,[1379,2018,2019],{"class":1385},"          A true favourite!\n",[1379,2021,2023,2025,2027],{"class":1381,"line":2022},18,[1379,2024,1934],{"class":1385},[1379,2026,11],{"class":1389},[1379,2028,1409],{"class":1385},[1379,2030,2032,2034,2036,2038,2040,2042,2044,2046],{"class":1381,"line":2031},19,[1379,2033,1901],{"class":1385},[1379,2035,1390],{"class":1389},[1379,2037,1394],{"class":1393},[1379,2039,1397],{"class":1385},[1379,2041,1952],{"class":1400},[1379,2043,1955],{"class":1385},[1379,2045,1390],{"class":1389},[1379,2047,1409],{"class":1385},[1379,2049,2051,2053,2055],{"class":1381,"line":2050},20,[1379,2052,1965],{"class":1385},[1379,2054,1833],{"class":1389},[1379,2056,1409],{"class":1385},[1379,2058,2060,2063,2065],{"class":1381,"line":2059},21,[1379,2061,2062],{"class":1385},"    \u003C/",[1379,2064,1833],{"class":1389},[1379,2066,1409],{"class":1385},[1379,2068,2070,2073,2075],{"class":1381,"line":2069},22,[1379,2071,2072],{"class":1385},"  \u003C/",[1379,2074,1833],{"class":1389},[1379,2076,1409],{"class":1385},[1379,2078,2080,2082,2084],{"class":1381,"line":2079},23,[1379,2081,1595],{"class":1385},[1379,2083,1542],{"class":1389},[1379,2085,1409],{"class":1385},[11,2087,2088],{},[35,2089,2090],{},"Downsampled DOM Snapshot:",[1371,2092,2094],{"className":1373,"code":2093,"language":1375,"meta":259,"style":259},"\u003Csection class=\"container\" required=\"true\" type=\"example\">\n  \u003Cdiv class=\"mx-auto\" required=\"false\">\n    Our Pizza\n    \u003Cdiv>\n      Margherita\n      A simple classic: mozzarella, tomatoes and basil.\n      An everyday choice!\n      \u003Cbutton type=\"button\">Add\u003C/button>\n      Capricciosa\n      A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n      A true favourite!\n      \u003Cbutton type=\"button\">Add\u003C/button>\n    \u003C/div>\n  \u003C/div>\n\u003C/section>\n",[1196,2095,2096,2122,2142,2147,2155,2160,2165,2170,2188,2193,2198,2203,2221,2229,2237],{"__ignoreMap":259},[1379,2097,2098,2100,2102,2104,2106,2108,2110,2112,2114,2116,2118,2120],{"class":1381,"line":1382},[1379,2099,1386],{"class":1385},[1379,2101,1542],{"class":1389},[1379,2103,1545],{"class":1393},[1379,2105,1397],{"class":1385},[1379,2107,1801],{"class":1400},[1379,2109,1812],{"class":1393},[1379,2111,1397],{"class":1385},[1379,2113,1817],{"class":1400},[1379,2115,1394],{"class":1393},[1379,2117,1397],{"class":1385},[1379,2119,1824],{"class":1400},[1379,2121,1409],{"class":1385},[1379,2123,2124,2126,2128,2130,2132,2134,2136,2138,2140],{"class":1381,"line":1021},[1379,2125,1557],{"class":1385},[1379,2127,1833],{"class":1389},[1379,2129,1545],{"class":1393},[1379,2131,1397],{"class":1385},[1379,2133,1840],{"class":1400},[1379,2135,1812],{"class":1393},[1379,2137,1397],{"class":1385},[1379,2139,1855],{"class":1400},[1379,2141,1409],{"class":1385},[1379,2143,2144],{"class":1381,"line":1026},[1379,2145,2146],{"class":1385},"    Our Pizza\n",[1379,2148,2149,2151,2153],{"class":1381,"line":1501},[1379,2150,1862],{"class":1385},[1379,2152,1833],{"class":1389},[1379,2154,1409],{"class":1385},[1379,2156,2157],{"class":1381,"line":1515},[1379,2158,2159],{"class":1385},"      Margherita\n",[1379,2161,2162],{"class":1381,"line":1602},[1379,2163,2164],{"class":1385},"      A simple classic: mozzarella, tomatoes and basil.\n",[1379,2166,2167],{"class":1381,"line":1617},[1379,2168,2169],{"class":1385},"      An everyday choice!\n",[1379,2171,2172,2174,2176,2178,2180,2182,2184,2186],{"class":1381,"line":1630},[1379,2173,1885],{"class":1385},[1379,2175,1390],{"class":1389},[1379,2177,1394],{"class":1393},[1379,2179,1397],{"class":1385},[1379,2181,1952],{"class":1400},[1379,2183,1955],{"class":1385},[1379,2185,1390],{"class":1389},[1379,2187,1409],{"class":1385},[1379,2189,2190],{"class":1381,"line":1643},[1379,2191,2192],{"class":1385},"      Capricciosa\n",[1379,2194,2195],{"class":1381,"line":1931},[1379,2196,2197],{"class":1385},"      A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[1379,2199,2200],{"class":1381,"line":1941},[1379,2201,2202],{"class":1385},"      A true favourite!\n",[1379,2204,2205,2207,2209,2211,2213,2215,2217,2219],{"class":1381,"line":1962},[1379,2206,1885],{"class":1385},[1379,2208,1390],{"class":1389},[1379,2210,1394],{"class":1393},[1379,2212,1397],{"class":1385},[1379,2214,1952],{"class":1400},[1379,2216,1955],{"class":1385},[1379,2218,1390],{"class":1389},[1379,2220,1409],{"class":1385},[1379,2222,2223,2225,2227],{"class":1381,"line":1972},[1379,2224,2062],{"class":1385},[1379,2226,1833],{"class":1389},[1379,2228,1409],{"class":1385},[1379,2230,2231,2233,2235],{"class":1381,"line":1987},[1379,2232,2072],{"class":1385},[1379,2234,1833],{"class":1389},[1379,2236,1409],{"class":1385},[1379,2238,2239,2241,2243],{"class":1381,"line":2001},[1379,2240,1595],{"class":1385},[1379,2242,1542],{"class":1389},[1379,2244,1409],{"class":1385},[11,2246,2247,2248,1116],{},"When downsampling is defined over different types of nodes – for instance, elements, attributes, and text – it becomes a powerful technique to control DOM size in a snapshot pipeline. Adaptive implementations can moreover produce snapshots that fall below a hard token limit. Our evaluation revealed that moderate downsampling ratios comfortably fit the mean DOM snapshot of real-world-representative web pages within GPT-4o's context window – leaving headroom for an agent system prompt and a trace of reasoning and action history",[1239,2249,2250],{},[15,2251,1778],{"href":1775,"ariaDescribedBy":2252,"dataFootnoteRef":259,"id":2253},[1245],"user-content-fnref-4-2",[118,2255,2257],{"id":2256},"the-problem-with-dom-based-targeting","The Problem with DOM-Based Targeting",[11,2259,2260,2261,2264,2265,2268],{},"When DOM snapshots are transformation artefacts, relative CSS selectors like nth-child paths might not map back to the live DOM. There are two straightforward ways to account for breaking CSS selectors: (1) assigning unique IDs to relevant elements in the live UI that persist transformations (e.g., via a unique attribute ",[1196,2262,2263],{},"UNIQUE-ID=\"27\"",", which can be selected via ",[1196,2266,2267],{},"[UNIQUE-ID=\"27\"]","), and (2) maintaining a source map between elements in the live DOM and the transformed DOM. The latter approach does favourably not affect the snapshot directly.",[11,2270,2271],{},[35,2272,2273],{},"DOM Snapshot with Inlined IDs:",[23,2275],{"src":2276,"alt":2277,"loading":28},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/7.png","A DOM snapshot (HTML) visualising unique ID (attribute) targeting",[118,2279,2281],{"id":2280},"the-dom-based-snapshot-pipeline","The DOM-Based Snapshot Pipeline",[11,2283,2284],{},"LLMs are able to reason over HTML. Yet, HTML carries noise, such as obfuscated scripts or hidden subtrees. DOM snapshots should for that reason be seen as a base representation, rather than a target representation.",[11,2286,2287],{},"The concept of DOM downsampling can be utilised as a universal size control in a snapshot pipeline – ideally exposed as quality, the inverse property of downsampling. Moreover, the DOM can be pre-transformed by deterministic techniques, such as pruning hidden subtrees to surface only the tangible UI – a characteristic that is natural with GUI snapshots.",[11,2289,2290],{},"The pipeline approach traceably goes from the raw DOM to an abstract snapshot representation. For example:",[296,2292,2293,2316,2322,2380,2386,2392],{},[54,2294,2295,2298,2299,2304,2305,2308,2309,1311,2312,2315],{},[35,2296,2297],{},"Scope Subtree."," Scope in to a task-relevant subtree. An information retrieval task on ",[15,2300,2303],{"href":2301,"rel":2302},"https://www.wikipedia.org",[19],"wikipedia.org"," can scope into the ",[1196,2306,2307],{},"main"," element – ",[1196,2310,2311],{},"header",[1196,2313,2314],{},"footer"," are irrelevant.",[54,2317,2318,2321],{},[35,2319,2320],{},"Clone."," Clone the live DOM.",[54,2323,2324,2327,2328],{},[35,2325,2326],{},"Filter."," Apply arbitrary filters to the cloned DOM (in memory). For example:\n",[51,2329,2330,2368,2374],{},[54,2331,2332,2335,2336],{},[35,2333,2334],{},"Prune Subtrees"," E.g.:\n",[51,2337,2338,2344,2350],{},[54,2339,2340,2343],{},[35,2341,2342],{},"Non-interactive Nodes",", e.g., hidden elements.",[54,2345,2346,2349],{},[35,2347,2348],{},"Irrelevant Nodes",", e.g., comment nodes.",[54,2351,2352,2355,2356,2358,2359,2361,2362,2365,2366,1116],{},[35,2353,2354],{},"Redundant Nodes",", e.g., ",[1196,2357,15],{}," in the ",[1196,2360,2314],{}," that has a same-",[1196,2363,2364],{},"href"," equivalent in the ",[1196,2367,2311],{},[54,2369,2370,2373],{},[35,2371,2372],{},"Sanitise."," E.g., sensitive information from content-editables.",[54,2375,2376,2379],{},[35,2377,2378],{},"Clean."," E.g., remove stop words from text nodes.",[54,2381,2382,2385],{},[35,2383,2384],{},"Serialise."," Serialise the cloned DOM to HTML.",[54,2387,2388,2391],{},[35,2389,2390],{},"Downsample."," Downsample the HTML to control size.",[54,2393,2394,2397],{},[35,2395,2396],{},"Translate."," Translate the downsampled HTML to any DOM-based representation, e.g., the accessibility tree.",[23,2399],{"src":2400,"alt":2401,"loading":28,"provider":29},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/8.svg","Example of a DOM snapshot pipeline",[11,2403,2404],{},"A filtered DOM drastically increases stability of repeatedly run web journeys: if only a single button routes to a target page, the agent perceives an unambiguous action target element. The rule of thumb is: the less actionable options a snapshot transcribes, the better – given all relevant options are covered.",[1225,2406,2408],{"id":2407},"arbitrary-snapshot-representations","Arbitrary Snapshot Representations",[11,2410,2411],{},"Any text-based snapshot representation can be derived from the single-source-of-truth DOM. Which representation is best highly depends on the given web browsing task. A high-level distinction is between (A) navigation and (B) information retrieval tasks. Navigation tasks do not require extensive text, but  actionable elements – if a snapshot omits a relevant button, the LLM is not able to use the encoded UI. Information retrieval tasks, on the other hand, require text, as the text contains topical information, whilst actionable elements do not matter.",[11,2413,2414],{},"Compare the following snapshots representations for the same UI:",[11,2416,2417],{},[35,2418,2419],{},"DOM Snapshot:",[1371,2421,2423],{"className":1373,"code":2422,"language":1375,"meta":259,"style":259},"\u003Csection class=\"container\" required=\"true\" type=\"example\">\n  \u003Cdiv class=\"mx-auto\" required=\"false\">\n    \u003Ch1>Our Pizza\u003C/h1>\n    \u003Cdiv>\n      \u003Ch2>Margherita\u003C/h2>\n      \u003Cp>\n        A simple classic: mozzarella, tomatoes and basil.\n        An everyday choice!\n      \u003C/p>\n      \u003Ca href=\"add?p=margherita\" data-uid=\"1\">Add\u003C/a>\n      \u003Ch2>Capricciosa\u003C/h2>\n      \u003Cp>\n        A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n        A true favourite!\n      \u003C/p>\n      \u003Ca href=\"add?p=capricciosa\" data-uid=\"2\">Add\u003C/a>\n    \u003C/div>\n  \u003C/div>\n\u003C/section>\n",[1196,2424,2425,2451,2471,2483,2491,2503,2511,2516,2521,2529,2556,2568,2576,2581,2586,2594,2620,2628,2636],{"__ignoreMap":259},[1379,2426,2427,2429,2431,2433,2435,2437,2439,2441,2443,2445,2447,2449],{"class":1381,"line":1382},[1379,2428,1386],{"class":1385},[1379,2430,1542],{"class":1389},[1379,2432,1545],{"class":1393},[1379,2434,1397],{"class":1385},[1379,2436,1801],{"class":1400},[1379,2438,1812],{"class":1393},[1379,2440,1397],{"class":1385},[1379,2442,1817],{"class":1400},[1379,2444,1394],{"class":1393},[1379,2446,1397],{"class":1385},[1379,2448,1824],{"class":1400},[1379,2450,1409],{"class":1385},[1379,2452,2453,2455,2457,2459,2461,2463,2465,2467,2469],{"class":1381,"line":1021},[1379,2454,1557],{"class":1385},[1379,2456,1833],{"class":1389},[1379,2458,1545],{"class":1393},[1379,2460,1397],{"class":1385},[1379,2462,1840],{"class":1400},[1379,2464,1812],{"class":1393},[1379,2466,1397],{"class":1385},[1379,2468,1855],{"class":1400},[1379,2470,1409],{"class":1385},[1379,2472,2473,2475,2477,2479,2481],{"class":1381,"line":1026},[1379,2474,1862],{"class":1385},[1379,2476,1865],{"class":1389},[1379,2478,1868],{"class":1385},[1379,2480,1865],{"class":1389},[1379,2482,1409],{"class":1385},[1379,2484,2485,2487,2489],{"class":1381,"line":1501},[1379,2486,1862],{"class":1385},[1379,2488,1833],{"class":1389},[1379,2490,1409],{"class":1385},[1379,2492,2493,2495,2497,2499,2501],{"class":1381,"line":1515},[1379,2494,1885],{"class":1385},[1379,2496,70],{"class":1389},[1379,2498,1906],{"class":1385},[1379,2500,70],{"class":1389},[1379,2502,1409],{"class":1385},[1379,2504,2505,2507,2509],{"class":1381,"line":1602},[1379,2506,1885],{"class":1385},[1379,2508,11],{"class":1389},[1379,2510,1409],{"class":1385},[1379,2512,2513],{"class":1381,"line":1617},[1379,2514,2515],{"class":1385},"        A simple classic: mozzarella, tomatoes and basil.\n",[1379,2517,2518],{"class":1381,"line":1630},[1379,2519,2520],{"class":1385},"        An everyday choice!\n",[1379,2522,2523,2525,2527],{"class":1381,"line":1643},[1379,2524,1965],{"class":1385},[1379,2526,11],{"class":1389},[1379,2528,1409],{"class":1385},[1379,2530,2531,2533,2535,2537,2539,2542,2545,2547,2550,2552,2554],{"class":1381,"line":1931},[1379,2532,1885],{"class":1385},[1379,2534,15],{"class":1389},[1379,2536,1431],{"class":1393},[1379,2538,1397],{"class":1385},[1379,2540,2541],{"class":1400},"\"add?p=margherita\"",[1379,2543,2544],{"class":1393}," data-uid",[1379,2546,1397],{"class":1385},[1379,2548,2549],{"class":1400},"\"1\"",[1379,2551,1955],{"class":1385},[1379,2553,15],{"class":1389},[1379,2555,1409],{"class":1385},[1379,2557,2558,2560,2562,2564,2566],{"class":1381,"line":1941},[1379,2559,1885],{"class":1385},[1379,2561,70],{"class":1389},[1379,2563,1994],{"class":1385},[1379,2565,70],{"class":1389},[1379,2567,1409],{"class":1385},[1379,2569,2570,2572,2574],{"class":1381,"line":1962},[1379,2571,1885],{"class":1385},[1379,2573,11],{"class":1389},[1379,2575,1409],{"class":1385},[1379,2577,2578],{"class":1381,"line":1972},[1379,2579,2580],{"class":1385},"        A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[1379,2582,2583],{"class":1381,"line":1987},[1379,2584,2585],{"class":1385},"        A true favourite!\n",[1379,2587,2588,2590,2592],{"class":1381,"line":2001},[1379,2589,1965],{"class":1385},[1379,2591,11],{"class":1389},[1379,2593,1409],{"class":1385},[1379,2595,2596,2598,2600,2602,2604,2607,2609,2611,2614,2616,2618],{"class":1381,"line":2010},[1379,2597,1885],{"class":1385},[1379,2599,15],{"class":1389},[1379,2601,1431],{"class":1393},[1379,2603,1397],{"class":1385},[1379,2605,2606],{"class":1400},"\"add?p=capricciosa\"",[1379,2608,2544],{"class":1393},[1379,2610,1397],{"class":1385},[1379,2612,2613],{"class":1400},"\"2\"",[1379,2615,1955],{"class":1385},[1379,2617,15],{"class":1389},[1379,2619,1409],{"class":1385},[1379,2621,2622,2624,2626],{"class":1381,"line":2016},[1379,2623,2062],{"class":1385},[1379,2625,1833],{"class":1389},[1379,2627,1409],{"class":1385},[1379,2629,2630,2632,2634],{"class":1381,"line":2022},[1379,2631,2072],{"class":1385},[1379,2633,1833],{"class":1389},[1379,2635,1409],{"class":1385},[1379,2637,2638,2640,2642],{"class":1381,"line":2031},[1379,2639,1595],{"class":1385},[1379,2641,1542],{"class":1389},[1379,2643,1409],{"class":1385},[11,2645,2646],{},[35,2647,2648],{},"Accessibility Tree Snapshot:",[1371,2650,2654],{"className":2651,"code":2652,"language":2653,"meta":259,"style":259},"language-yaml shiki shiki-themes github-dark github-dark","- document:\n  children:\n    - section:\n      children:\n        - heading:\n          name: \"Our Pizza\"\n          level: 1\n        - heading:\n          name: \"Margherita\"\n          level: 2\n        - paragraph:\n          children:\n            - text: \"A simple classic: mozzarella, tomatoes and basil. An everyday choice!\"\n        - link:\n          name: \"Add\"\n          url: \"add?p=margherita\"\n          data-uid: 1\n        - heading:\n          name: \"Capricciosa\"\n          level: 2\n        - paragraph:\n          children:\n            - text: \"A rich taste: mozzarella, ham, mushrooms, artichokes and olives. A true favourite!\"\n        - link:\n          name: \"Add\"\n          url: \"add?p=capricciosa\"\n          data-uid: 2\n","yaml",[1196,2655,2656,2667,2674,2683,2690,2700,2711,2722,2730,2739,2748,2757,2764,2777,2786,2795,2805,2814,2822,2831,2839,2847,2853,2864,2873,2882,2892],{"__ignoreMap":259},[1379,2657,2658,2661,2664],{"class":1381,"line":1382},[1379,2659,2660],{"class":1385},"- ",[1379,2662,2663],{"class":1389},"document",[1379,2665,2666],{"class":1385},":\n",[1379,2668,2669,2672],{"class":1381,"line":1021},[1379,2670,2671],{"class":1389},"  children",[1379,2673,2666],{"class":1385},[1379,2675,2676,2679,2681],{"class":1381,"line":1026},[1379,2677,2678],{"class":1385},"    - ",[1379,2680,1542],{"class":1389},[1379,2682,2666],{"class":1385},[1379,2684,2685,2688],{"class":1381,"line":1501},[1379,2686,2687],{"class":1389},"      children",[1379,2689,2666],{"class":1385},[1379,2691,2692,2695,2698],{"class":1381,"line":1515},[1379,2693,2694],{"class":1385},"        - ",[1379,2696,2697],{"class":1389},"heading",[1379,2699,2666],{"class":1385},[1379,2701,2702,2705,2708],{"class":1381,"line":1602},[1379,2703,2704],{"class":1389},"          name",[1379,2706,2707],{"class":1385},": ",[1379,2709,2710],{"class":1400},"\"Our Pizza\"\n",[1379,2712,2713,2716,2718],{"class":1381,"line":1617},[1379,2714,2715],{"class":1389},"          level",[1379,2717,2707],{"class":1385},[1379,2719,2721],{"class":2720},"s8ozJ","1\n",[1379,2723,2724,2726,2728],{"class":1381,"line":1630},[1379,2725,2694],{"class":1385},[1379,2727,2697],{"class":1389},[1379,2729,2666],{"class":1385},[1379,2731,2732,2734,2736],{"class":1381,"line":1643},[1379,2733,2704],{"class":1389},[1379,2735,2707],{"class":1385},[1379,2737,2738],{"class":1400},"\"Margherita\"\n",[1379,2740,2741,2743,2745],{"class":1381,"line":1931},[1379,2742,2715],{"class":1389},[1379,2744,2707],{"class":1385},[1379,2746,2747],{"class":2720},"2\n",[1379,2749,2750,2752,2755],{"class":1381,"line":1941},[1379,2751,2694],{"class":1385},[1379,2753,2754],{"class":1389},"paragraph",[1379,2756,2666],{"class":1385},[1379,2758,2759,2762],{"class":1381,"line":1962},[1379,2760,2761],{"class":1389},"          children",[1379,2763,2666],{"class":1385},[1379,2765,2766,2769,2772,2774],{"class":1381,"line":1972},[1379,2767,2768],{"class":1385},"            - ",[1379,2770,2771],{"class":1389},"text",[1379,2773,2707],{"class":1385},[1379,2775,2776],{"class":1400},"\"A simple classic: mozzarella, tomatoes and basil. An everyday choice!\"\n",[1379,2778,2779,2781,2784],{"class":1381,"line":1987},[1379,2780,2694],{"class":1385},[1379,2782,2783],{"class":1389},"link",[1379,2785,2666],{"class":1385},[1379,2787,2788,2790,2792],{"class":1381,"line":2001},[1379,2789,2704],{"class":1389},[1379,2791,2707],{"class":1385},[1379,2793,2794],{"class":1400},"\"Add\"\n",[1379,2796,2797,2800,2802],{"class":1381,"line":2010},[1379,2798,2799],{"class":1389},"          url",[1379,2801,2707],{"class":1385},[1379,2803,2804],{"class":1400},"\"add?p=margherita\"\n",[1379,2806,2807,2810,2812],{"class":1381,"line":2016},[1379,2808,2809],{"class":1389},"          data-uid",[1379,2811,2707],{"class":1385},[1379,2813,2721],{"class":2720},[1379,2815,2816,2818,2820],{"class":1381,"line":2022},[1379,2817,2694],{"class":1385},[1379,2819,2697],{"class":1389},[1379,2821,2666],{"class":1385},[1379,2823,2824,2826,2828],{"class":1381,"line":2031},[1379,2825,2704],{"class":1389},[1379,2827,2707],{"class":1385},[1379,2829,2830],{"class":1400},"\"Capricciosa\"\n",[1379,2832,2833,2835,2837],{"class":1381,"line":2050},[1379,2834,2715],{"class":1389},[1379,2836,2707],{"class":1385},[1379,2838,2747],{"class":2720},[1379,2840,2841,2843,2845],{"class":1381,"line":2059},[1379,2842,2694],{"class":1385},[1379,2844,2754],{"class":1389},[1379,2846,2666],{"class":1385},[1379,2848,2849,2851],{"class":1381,"line":2069},[1379,2850,2761],{"class":1389},[1379,2852,2666],{"class":1385},[1379,2854,2855,2857,2859,2861],{"class":1381,"line":2079},[1379,2856,2768],{"class":1385},[1379,2858,2771],{"class":1389},[1379,2860,2707],{"class":1385},[1379,2862,2863],{"class":1400},"\"A rich taste: mozzarella, ham, mushrooms, artichokes and olives. A true favourite!\"\n",[1379,2865,2867,2869,2871],{"class":1381,"line":2866},24,[1379,2868,2694],{"class":1385},[1379,2870,2783],{"class":1389},[1379,2872,2666],{"class":1385},[1379,2874,2876,2878,2880],{"class":1381,"line":2875},25,[1379,2877,2704],{"class":1389},[1379,2879,2707],{"class":1385},[1379,2881,2794],{"class":1400},[1379,2883,2885,2887,2889],{"class":1381,"line":2884},26,[1379,2886,2799],{"class":1389},[1379,2888,2707],{"class":1385},[1379,2890,2891],{"class":1400},"\"add?p=capricciosa\"\n",[1379,2893,2895,2897,2899],{"class":1381,"line":2894},27,[1379,2896,2809],{"class":1389},[1379,2898,2707],{"class":1385},[1379,2900,2747],{"class":2720},[11,2902,2903],{},[35,2904,2905],{},"Markdown Snapshot:",[1371,2907,2910],{"className":2908,"code":2909,"language":1057,"meta":259,"style":259},"language-md shiki shiki-themes github-dark github-dark","# Our Pizza\n\n## Margherita\n\nA simple classic: mozzarella, tomatoes and basil.\nAn everyday choice!\n\n[Add](add?p=margherita)\n\n## Capricciosa\n\nA rich taste: mozzarella, ham, mushrooms, artichokes and olives.\nA true favourite!\n\n[Add](add?p=capricciosa)\n",[1196,2911,2912,2917,2922,2927,2931,2936,2941,2945,2950,2954,2959,2963,2968,2973,2977],{"__ignoreMap":259},[1379,2913,2914],{"class":1381,"line":1382},[1379,2915,2916],{},"# Our Pizza\n",[1379,2918,2919],{"class":1381,"line":1021},[1379,2920,2921],{"emptyLinePlaceholder":1092},"\n",[1379,2923,2924],{"class":1381,"line":1026},[1379,2925,2926],{},"## Margherita\n",[1379,2928,2929],{"class":1381,"line":1501},[1379,2930,2921],{"emptyLinePlaceholder":1092},[1379,2932,2933],{"class":1381,"line":1515},[1379,2934,2935],{},"A simple classic: mozzarella, tomatoes and basil.\n",[1379,2937,2938],{"class":1381,"line":1602},[1379,2939,2940],{},"An everyday choice!\n",[1379,2942,2943],{"class":1381,"line":1617},[1379,2944,2921],{"emptyLinePlaceholder":1092},[1379,2946,2947],{"class":1381,"line":1630},[1379,2948,2949],{},"[Add](add?p=margherita)\n",[1379,2951,2952],{"class":1381,"line":1643},[1379,2953,2921],{"emptyLinePlaceholder":1092},[1379,2955,2956],{"class":1381,"line":1931},[1379,2957,2958],{},"## Capricciosa\n",[1379,2960,2961],{"class":1381,"line":1941},[1379,2962,2921],{"emptyLinePlaceholder":1092},[1379,2964,2965],{"class":1381,"line":1962},[1379,2966,2967],{},"A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[1379,2969,2970],{"class":1381,"line":1972},[1379,2971,2972],{},"A true favourite!\n",[1379,2974,2975],{"class":1381,"line":1987},[1379,2976,2921],{"emptyLinePlaceholder":1092},[1379,2978,2979],{"class":1381,"line":2001},[1379,2980,2981],{},"[Add](add?p=capricciosa)\n",[11,2983,2984],{},[35,2985,2986],{},"Custom Format Snapshot:",[1371,2988,2992],{"className":2989,"code":2991,"language":2771},[2990],"language-text","[0] H1 \"Our Pizza\"\n[1] H2 \"Margherita\"\n[2] P \"A simple classic: mozzarella, tomatoes and basil.\\nAn everyday choice!\"\n[3] A \"Add\" \"add?p=margherita\"\n[4] H2 \"Capricciosa\"\n[5] P \"A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\\nA true...\"\n[6] A \"Add\" \"add?p=capricciosa\"\n",[1196,2993,2991],{"__ignoreMap":259},[11,2995,2996,2997,1116],{},"Browser Use agents implement hybrid snapshots: the grounded GUI snapshot is a supplement to a custom-format indexed-list DOM snapshot, similar to the above example. Evaluations show that the indexed-list snapshot representation generally conveys most of the salient UI features for the web agent LLM backend",[1239,2998,2999],{},[15,3000,1778],{"href":1775,"ariaDescribedBy":3001,"dataFootnoteRef":259,"id":3002},[1245],"user-content-fnref-4-3",[70,3004,3006],{"id":3005},"non-idiomatic-application-designs","Non-Idiomatic Application Designs",[11,3008,3009],{},"An actionable element might lack affordance – not communicating its actionability through its bare appearance. That is bad UI/UX design practice. The concept extends to the DOM through HTML semantics, which communicate the general purpose of an element to web browsers and developers rather than users. Generalising to web agent LLM backends, the concept of affordance aligns between the GUI and the DOM – the model is a UI representation-agnostic user.",[11,3011,3012],{},"Missing affordances should be rectified in a snapshot wherever they can be discovered. Discoverability is another circular problem: what is an indirect affordance for actionability? Ideally, the underlying actionability is tracked, such as by checking for attached action event listeners. However, there is no browser API for this practice.",[11,3014,3015],{},"As an example for bad affordance in the GUI, imagine an overly subtle 'Shop' button. Bounding boxes in grounded GUI snapshots are a workaround – provided the element is discoverable as actionable in the DOM in the first place.",[23,3017],{"src":3018,"alt":3019,"loading":28,":width":3020},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/9.png","Example of a fictional UI/UX bad practice (overly subtle button)","480",[11,3022,3023,3024,3029],{},"As an example for bad affordance in the DOM, look at this datepicker from ",[15,3025,3028],{"href":3026,"rel":3027},"https://www.webmd.com",[19],"webmd.com",":",[23,3031],{"src":3032,"alt":3033,"loading":28,":width":3020},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/10.png","Example of an HTML bad practice (ignoring tag name semantics)",[1371,3035,3037],{"className":1373,"code":3036,"language":1375,"meta":259,"style":259},"\u003Cdiv>\n  \u003Cspan class=\"cell day-header\">Sun\u003C/span>\n  \u003Cspan class=\"cell day-header\">Mon\u003C/span>\n  \u003Cspan class=\"cell day-header\">Tue\u003C/span>\n  \u003Cspan class=\"cell day-header\">Wed\u003C/span>\n  \u003Cspan class=\"cell day-header\">Thu\u003C/span>\n  \u003Cspan class=\"cell day-header\">Fri\u003C/span>\n  \u003Cspan class=\"cell day-header\">Sat\u003C/span>\n  \u003C!-- [...] -->\n  \u003Cspan class=\"cell day blank\">\u003C/span>\n  \u003Cspan class=\"cell day blank\">\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>1\u003C/span>\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>2\u003C/span>\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>3\u003C/span>\u003C/span>\n  \u003C!-- [...] -->\n  \u003Cspan class=\"cell day\">\u003Cspan>29\u003C/span>\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>30\u003C/span>\u003C/span>\n\u003C/div>\n",[1196,3038,3039,3047,3067,3086,3105,3124,3143,3162,3181,3187,3207,3225,3254,3281,3308,3312,3339,3366],{"__ignoreMap":259},[1379,3040,3041,3043,3045],{"class":1381,"line":1382},[1379,3042,1386],{"class":1385},[1379,3044,1833],{"class":1389},[1379,3046,1409],{"class":1385},[1379,3048,3049,3051,3053,3055,3057,3060,3063,3065],{"class":1381,"line":1021},[1379,3050,1557],{"class":1385},[1379,3052,1379],{"class":1389},[1379,3054,1545],{"class":1393},[1379,3056,1397],{"class":1385},[1379,3058,3059],{"class":1400},"\"cell day-header\"",[1379,3061,3062],{"class":1385},">Sun\u003C/",[1379,3064,1379],{"class":1389},[1379,3066,1409],{"class":1385},[1379,3068,3069,3071,3073,3075,3077,3079,3082,3084],{"class":1381,"line":1026},[1379,3070,1557],{"class":1385},[1379,3072,1379],{"class":1389},[1379,3074,1545],{"class":1393},[1379,3076,1397],{"class":1385},[1379,3078,3059],{"class":1400},[1379,3080,3081],{"class":1385},">Mon\u003C/",[1379,3083,1379],{"class":1389},[1379,3085,1409],{"class":1385},[1379,3087,3088,3090,3092,3094,3096,3098,3101,3103],{"class":1381,"line":1501},[1379,3089,1557],{"class":1385},[1379,3091,1379],{"class":1389},[1379,3093,1545],{"class":1393},[1379,3095,1397],{"class":1385},[1379,3097,3059],{"class":1400},[1379,3099,3100],{"class":1385},">Tue\u003C/",[1379,3102,1379],{"class":1389},[1379,3104,1409],{"class":1385},[1379,3106,3107,3109,3111,3113,3115,3117,3120,3122],{"class":1381,"line":1515},[1379,3108,1557],{"class":1385},[1379,3110,1379],{"class":1389},[1379,3112,1545],{"class":1393},[1379,3114,1397],{"class":1385},[1379,3116,3059],{"class":1400},[1379,3118,3119],{"class":1385},">Wed\u003C/",[1379,3121,1379],{"class":1389},[1379,3123,1409],{"class":1385},[1379,3125,3126,3128,3130,3132,3134,3136,3139,3141],{"class":1381,"line":1602},[1379,3127,1557],{"class":1385},[1379,3129,1379],{"class":1389},[1379,3131,1545],{"class":1393},[1379,3133,1397],{"class":1385},[1379,3135,3059],{"class":1400},[1379,3137,3138],{"class":1385},">Thu\u003C/",[1379,3140,1379],{"class":1389},[1379,3142,1409],{"class":1385},[1379,3144,3145,3147,3149,3151,3153,3155,3158,3160],{"class":1381,"line":1617},[1379,3146,1557],{"class":1385},[1379,3148,1379],{"class":1389},[1379,3150,1545],{"class":1393},[1379,3152,1397],{"class":1385},[1379,3154,3059],{"class":1400},[1379,3156,3157],{"class":1385},">Fri\u003C/",[1379,3159,1379],{"class":1389},[1379,3161,1409],{"class":1385},[1379,3163,3164,3166,3168,3170,3172,3174,3177,3179],{"class":1381,"line":1630},[1379,3165,1557],{"class":1385},[1379,3167,1379],{"class":1389},[1379,3169,1545],{"class":1393},[1379,3171,1397],{"class":1385},[1379,3173,3059],{"class":1400},[1379,3175,3176],{"class":1385},">Sat\u003C/",[1379,3178,1379],{"class":1389},[1379,3180,1409],{"class":1385},[1379,3182,3183],{"class":1381,"line":1643},[1379,3184,3186],{"class":3185},"sJ8bj","  \u003C!-- [...] -->\n",[1379,3188,3189,3191,3193,3195,3197,3200,3203,3205],{"class":1381,"line":1931},[1379,3190,1557],{"class":1385},[1379,3192,1379],{"class":1389},[1379,3194,1545],{"class":1393},[1379,3196,1397],{"class":1385},[1379,3198,3199],{"class":1400},"\"cell day blank\"",[1379,3201,3202],{"class":1385},">\u003C/",[1379,3204,1379],{"class":1389},[1379,3206,1409],{"class":1385},[1379,3208,3209,3211,3213,3215,3217,3219,3221,3223],{"class":1381,"line":1941},[1379,3210,1557],{"class":1385},[1379,3212,1379],{"class":1389},[1379,3214,1545],{"class":1393},[1379,3216,1397],{"class":1385},[1379,3218,3199],{"class":1400},[1379,3220,3202],{"class":1385},[1379,3222,1379],{"class":1389},[1379,3224,1409],{"class":1385},[1379,3226,3227,3229,3231,3233,3235,3238,3241,3243,3246,3248,3250,3252],{"class":1381,"line":1962},[1379,3228,1557],{"class":1385},[1379,3230,1379],{"class":1389},[1379,3232,1545],{"class":1393},[1379,3234,1397],{"class":1385},[1379,3236,3237],{"class":1400},"\"cell day\"",[1379,3239,3240],{"class":1385},">\u003C",[1379,3242,1379],{"class":1389},[1379,3244,3245],{"class":1385},">1\u003C/",[1379,3247,1379],{"class":1389},[1379,3249,3202],{"class":1385},[1379,3251,1379],{"class":1389},[1379,3253,1409],{"class":1385},[1379,3255,3256,3258,3260,3262,3264,3266,3268,3270,3273,3275,3277,3279],{"class":1381,"line":1972},[1379,3257,1557],{"class":1385},[1379,3259,1379],{"class":1389},[1379,3261,1545],{"class":1393},[1379,3263,1397],{"class":1385},[1379,3265,3237],{"class":1400},[1379,3267,3240],{"class":1385},[1379,3269,1379],{"class":1389},[1379,3271,3272],{"class":1385},">2\u003C/",[1379,3274,1379],{"class":1389},[1379,3276,3202],{"class":1385},[1379,3278,1379],{"class":1389},[1379,3280,1409],{"class":1385},[1379,3282,3283,3285,3287,3289,3291,3293,3295,3297,3300,3302,3304,3306],{"class":1381,"line":1987},[1379,3284,1557],{"class":1385},[1379,3286,1379],{"class":1389},[1379,3288,1545],{"class":1393},[1379,3290,1397],{"class":1385},[1379,3292,3237],{"class":1400},[1379,3294,3240],{"class":1385},[1379,3296,1379],{"class":1389},[1379,3298,3299],{"class":1385},">3\u003C/",[1379,3301,1379],{"class":1389},[1379,3303,3202],{"class":1385},[1379,3305,1379],{"class":1389},[1379,3307,1409],{"class":1385},[1379,3309,3310],{"class":1381,"line":2001},[1379,3311,3186],{"class":3185},[1379,3313,3314,3316,3318,3320,3322,3324,3326,3328,3331,3333,3335,3337],{"class":1381,"line":2010},[1379,3315,1557],{"class":1385},[1379,3317,1379],{"class":1389},[1379,3319,1545],{"class":1393},[1379,3321,1397],{"class":1385},[1379,3323,3237],{"class":1400},[1379,3325,3240],{"class":1385},[1379,3327,1379],{"class":1389},[1379,3329,3330],{"class":1385},">29\u003C/",[1379,3332,1379],{"class":1389},[1379,3334,3202],{"class":1385},[1379,3336,1379],{"class":1389},[1379,3338,1409],{"class":1385},[1379,3340,3341,3343,3345,3347,3349,3351,3353,3355,3358,3360,3362,3364],{"class":1381,"line":2016},[1379,3342,1557],{"class":1385},[1379,3344,1379],{"class":1389},[1379,3346,1545],{"class":1393},[1379,3348,1397],{"class":1385},[1379,3350,3237],{"class":1400},[1379,3352,3240],{"class":1385},[1379,3354,1379],{"class":1389},[1379,3356,3357],{"class":1385},">30\u003C/",[1379,3359,1379],{"class":1389},[1379,3361,3202],{"class":1385},[1379,3363,1379],{"class":1389},[1379,3365,1409],{"class":1385},[1379,3367,3368,3370,3372],{"class":1381,"line":2022},[1379,3369,1595],{"class":1385},[1379,3371,1833],{"class":1389},[1379,3373,1409],{"class":1385},[11,3375,3376,3377,3379,3380,3383,3384,3386,3387,3390,3391,3393],{},"The days wrapped in ",[1196,3378,1379],{}," (e.g., ",[1196,3381,3382],{},"16",") are clickable, but the HTML communicates no such actionability. The idiomatic DOM equivalent would instead use ",[1196,3385,1390],{}," elements throughout, or at least assign suitable ",[1196,3388,3389],{},"role"," attributes like ",[1196,3392,1390],{},". A DOM snapshot pipeline leaves space for solving this problem: rewrite the virtual DOM with idiomatic elements.",[1371,3395,3397],{"className":1373,"code":3396,"language":1375,"meta":259,"style":259},"  \u003Cbutton class=\"cell day\">29\u003C/button>\n  \u003C!-- OR -->\n  \u003Cspan role=\"button\" class=\"cell day\">30\u003C/button>\n",[1196,3398,3399,3417,3422],{"__ignoreMap":259},[1379,3400,3401,3403,3405,3407,3409,3411,3413,3415],{"class":1381,"line":1382},[1379,3402,1557],{"class":1385},[1379,3404,1390],{"class":1389},[1379,3406,1545],{"class":1393},[1379,3408,1397],{"class":1385},[1379,3410,3237],{"class":1400},[1379,3412,3330],{"class":1385},[1379,3414,1390],{"class":1389},[1379,3416,1409],{"class":1385},[1379,3418,3419],{"class":1381,"line":1021},[1379,3420,3421],{"class":3185},"  \u003C!-- OR -->\n",[1379,3423,3424,3426,3428,3431,3433,3435,3437,3439,3441,3443,3445],{"class":1381,"line":1026},[1379,3425,1557],{"class":1385},[1379,3427,1379],{"class":1389},[1379,3429,3430],{"class":1393}," role",[1379,3432,1397],{"class":1385},[1379,3434,1952],{"class":1400},[1379,3436,1545],{"class":1393},[1379,3438,1397],{"class":1385},[1379,3440,3237],{"class":1400},[1379,3442,3357],{"class":1385},[1379,3444,1390],{"class":1389},[1379,3446,1409],{"class":1385},[70,3448,3450],{"id":3449},"choosing-between-snapshot-representations","Choosing Between Snapshot Representations",[11,3452,3453],{},"No snapshot representation fits all purposes; the two base representations alone – GUI and DOM (HTML) – come with markedly different advantages and limitations. A proper snapshot tool should therefore route and cascade:",[296,3455,3456,3468,3474,3480],{},[54,3457,3458,3461,3462,3464,3465,3467],{},[35,3459,3460],{},"Route Modality."," Fall back to GUI snapshots if the UI is based on ",[1196,3463,1296],{}," or cross-origin ",[1196,3466,1292],{}," elements; use DOM snapshots by default.",[54,3469,3470,3473],{},[35,3471,3472],{},"Route Representation."," Categorise task and translate DOM snapshot to the most suitable representation. Categorising the task can be done with increasing levels of machine learning – simple keyword matching, or deploying a language model.",[54,3475,3476,3479],{},[35,3477,3478],{},"Scope Escalation."," If elicited action suggestions are unusable, widen the snapshot scope and representation richness.",[54,3481,3482,3485],{},[35,3483,3484],{},"Cascade to Hybrid."," If elicited action suggestions are still unusable, augment the rich DOM snapshot with a GUI snapshot for maximum grounding.",[126,3487,3488,3507],{},[129,3489,3490],{},[132,3491,3492,3495,3498,3501,3504],{},[135,3493],{"align":3494},"left",[135,3496,3497],{},"GUI",[135,3499,3500],{},"grounded",[135,3502,3503],{},"DOM",[135,3505,3506],{},"transformed",[151,3508,3509,3531,3552,3575,3594,3615],{},[132,3510,3511,3514,3519,3523,3526],{},[156,3512,3513],{"align":3494},"LLM Input Tokens",[156,3515,3516],{},[35,3517,3518],{},"Low",[156,3520,3521],{},[35,3522,3518],{},[156,3524,3525],{},"High",[156,3527,3528,3530],{},[35,3529,3518],{}," to Moderate",[132,3532,3533,3536,3539,3544,3548],{},[156,3534,3535],{"align":3494},"Element Targeting",[156,3537,3538],{},"Absolute",[156,3540,3541],{},[35,3542,3543],{},"Relative",[156,3545,3546],{},[35,3547,3543],{},[156,3549,3550],{},[35,3551,3543],{},[132,3553,3554,3557,3563,3567,3571],{},[156,3555,3556],{"align":3494},"Snapshot Scope",[156,3558,3559,3562],{},[35,3560,3561],{},"Free","/Viewport",[156,3564,3565,3562],{},[35,3566,3561],{},[156,3568,3569],{},[35,3570,3561],{},[156,3572,3573],{},[35,3574,3561],{},[132,3576,3577,3580,3583,3585,3590],{},[156,3578,3579],{"align":3494},"Snapshot Context",[156,3581,3582],{},"UI",[156,3584,3582],{},[156,3586,3587],{},[35,3588,3589],{},"Full",[156,3591,3592],{},[35,3593,3589],{},[132,3595,3596,3601,3606,3610,3613],{},[156,3597,3598,3600],{"align":3494},[1196,3599,1292],{}," Context",[156,3602,3603],{},[35,3604,3605],{},"Viewport",[156,3607,3608],{},[35,3609,3605],{},[156,3611,3612],{},"Same-origin only",[156,3614,3612],{},[132,3616,3617,3621,3625,3629,3632],{},[156,3618,3619,3600],{"align":3494},[1196,3620,1296],{},[156,3622,3623],{},[35,3624,166],{},[156,3626,3627],{},[35,3628,166],{},[156,3630,3631],{},"No",[156,3633,3631],{},[70,3635,3637],{"id":3636},"using-snapshots-in-the-model-context","Using Snapshots in the Model Context",[11,3639,3640],{},"Creating a snapshot is one thing. Using it as context to a model prompt is the other. Usually, conversational AI maintains a full trace of history as model context. Since a snapshot is time-sensitive, however, only the latest snapshot can be targeted with actions by an LLM.",[11,3642,3643],{},"Whether to keep a trace of previous actions and model reasoning is a question in its own right. If web browsing is modelled as a history-independent task, the snapshot-to-actions agent loop could even be separated from the persistent conversational loop as transient, one-shot model prompts.",[70,3645,3647],{"id":3646},"timing-snapshots-in-the-web-client","Timing Snapshots in the Web Client",[11,3649,3650,3651,1116],{},"LLM-based web agents are much slower than humans in taking and processing snapshots. Not least for that reason, snapshot rounds should be kept to a minimum and timed so that a snapshot is taken as soon as the UI has reached a stable state. That implies a non-trivial challenge: defining what qualifies as a UI change, and at what intensity such a change counts as a state shift – beyond hard page transitions, e.g., in a single-page application",[1239,3652,3653],{},[15,3654,3658],{"href":3655,"ariaDescribedBy":3656,"dataFootnoteRef":259,"id":3657},"#user-content-fn-5",[1245],"user-content-fnref-5","5",[3660,3661],"hr",{},[11,3663,3664,3665,1116],{},"The snapshot is the central artefact in a web agent harness. Its representation is among the most significant factors for agent success",[1239,3666,3667],{},[15,3668,3672],{"href":3669,"ariaDescribedBy":3670,"dataFootnoteRef":259,"id":3671},"#user-content-fn-6",[1245],"user-content-fnref-6","6",[11,3674,3675,3676,3678],{},"At Webfuse, we develop a platform for agentic browsing. A focus of our research is on efficient snapshot representations. The Automation API can be used to create next-level snapshots: timed, rectified, across ",[1196,3677,1292],{}," boundaries, etc.",[1542,3680,3683,3688],{"className":3681,"dataFootnotes":259},[3682],"footnotes",[70,3684,3687],{"className":3685,"id":1245},[3686],"sr-only","Footnotes",[296,3689,3690,3705,3718,3730,3756,3768],{},[54,3691,3693,3697,3698],{"id":3692},"user-content-fn-1",[15,3694,3695],{"href":3695,"rel":3696},"https://arxiv.org/abs/2310.11441",[19]," ",[15,3699,3704],{"href":3700,"ariaLabel":3701,"className":3702,"dataFootnoteBackref":259},"#user-content-fnref-1","Back to reference 1",[3703],"data-footnote-backref","↩",[54,3706,3708,3709,3697,3713],{"id":3707},"user-content-fn-2","Our reimplementation of Browser Use's grounded GUI snapshots: ",[15,3710,3711],{"href":3711,"rel":3712},"https://github.com/webfuse-com/D2Snap/blob/main/snapshots/_bu.js",[19],[15,3714,3704],{"href":3715,"ariaLabel":3716,"className":3717,"dataFootnoteBackref":259},"#user-content-fnref-2","Back to reference 2",[3703],[54,3719,3721,3697,3725],{"id":3720},"user-content-fn-3",[15,3722,3723],{"href":3723,"rel":3724},"https://arxiv.org/abs/2210.03945",[19],[15,3726,3704],{"href":3727,"ariaLabel":3728,"className":3729,"dataFootnoteBackref":259},"#user-content-fnref-3","Back to reference 3",[3703],[54,3731,3733,3697,3737,3697,3742,3697,3749],{"id":3732},"user-content-fn-4",[15,3734,3735],{"href":3735,"rel":3736},"https://arxiv.org/abs/2508.04412",[19],[15,3738,3704],{"href":3739,"ariaLabel":3740,"className":3741,"dataFootnoteBackref":259},"#user-content-fnref-4","Back to reference 4",[3703],[15,3743,3704,3747],{"href":3744,"ariaLabel":3745,"className":3746,"dataFootnoteBackref":259},"#user-content-fnref-4-2","Back to reference 4-2",[3703],[1239,3748,1270],{},[15,3750,3704,3754],{"href":3751,"ariaLabel":3752,"className":3753,"dataFootnoteBackref":259},"#user-content-fnref-4-3","Back to reference 4-3",[3703],[1239,3755,1285],{},[54,3757,3759,3697,3763],{"id":3758},"user-content-fn-5",[15,3760,3761],{"href":3761,"rel":3762},"https://github.com/webfuse-com/DOMQuake",[19],[15,3764,3704],{"href":3765,"ariaLabel":3766,"className":3767,"dataFootnoteBackref":259},"#user-content-fnref-5","Back to reference 5",[3703],[54,3769,3771,3697,3775],{"id":3770},"user-content-fn-6",[15,3772,3773],{"href":3773,"rel":3774},"https://arxiv.org/abs/2504.01382",[19],[15,3776,3704],{"href":3777,"ariaLabel":3778,"className":3779,"dataFootnoteBackref":259},"#user-content-fnref-6","Back to reference 6",[3703],[3781,3782,3783],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .suv1-, html code.shiki .suv1-{--shiki-default:#E1E4E8;--shiki-dark:#E1E4E8}html pre.shiki code .sxg3X, html code.shiki .sxg3X{--shiki-default:#85E89D;--shiki-dark:#85E89D}html pre.shiki code .sFR8T, html code.shiki .sFR8T{--shiki-default:#B392F0;--shiki-dark:#B392F0}html pre.shiki code .s4wv1, html code.shiki .s4wv1{--shiki-default:#9ECBFF;--shiki-dark:#9ECBFF}html pre.shiki code .s8ozJ, html code.shiki .s8ozJ{--shiki-default:#79B8FF;--shiki-dark:#79B8FF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}",{"title":259,"searchDepth":1021,"depth":1021,"links":3785},[3786,3790,3796,3797,3798,3799,3800],{"id":1148,"depth":1021,"text":1149,"children":3787},[3788,3789],{"id":1171,"depth":1026,"text":1172},{"id":1211,"depth":1026,"text":1212},{"id":1273,"depth":1021,"text":1274,"children":3791},[3792,3793,3794,3795],{"id":1303,"depth":1026,"text":1304},{"id":1328,"depth":1026,"text":1329},{"id":2256,"depth":1026,"text":2257},{"id":2280,"depth":1026,"text":2281},{"id":3005,"depth":1021,"text":3006},{"id":3449,"depth":1021,"text":3450},{"id":3636,"depth":1021,"text":3637},{"id":3646,"depth":1021,"text":3647},{"id":1245,"depth":1021,"text":3687},"2025-08-28","Understand how web-based user interfaces can be serialised for use with LLM-based web agents.",{"homepage":1092,"relatedLinks":3804},[3805,3808],{"text":3806,"href":3807},"DOM Downsampling for LLM-Based Web Agents","/rnd/dom-downsampling-for-llm-based-web-agents",{"text":3809,"href":3810,"external":1092},"Webfuse Automation API","https://dev.webfuse.com/automation-api/","/blog/serialising-web-ui-state-for-llms-the-complete-guide",{"title":1106,"description":3802},{"loc":3811},"blog/1013.serialising-web-ui-state-for-llms-the-complete-guide",[1100,3816,3817,3818,3819],"web-agents","web-automation","llms","snapshots","X7R93b2XjyDfLllTDaNgFoo_c8AvF9oBDGzv4yiISFA",{"id":3822,"title":3823,"authorId":1107,"body":3824,"category":1100,"created":4300,"description":4301,"extension":1057,"faqs":1075,"featurePriority":1075,"head":1075,"hideCta":1092,"landingPath":1075,"meta":4302,"navigation":1092,"ogImage":1075,"path":4308,"robots":1075,"schemaOrg":1075,"seo":4309,"sitemap":4310,"stem":4311,"tags":4312,"__hash__":4314},"blog/blog/1011.a-gentle-introduction-to-ai-agents-for-the-web.md","A Gentle Introduction to AI Agents for the Web",{"type":8,"value":3825,"toc":4281},[3826,3844,3848,3856,3861,3864,3868,3871,3891,3895,3904,3908,3924,3929,3932,3936,3947,3951,3954,3958,3965,3969,3992,3996,4000,4003,4006,4010,4031,4034,4038,4042,4051,4062,4067,4071,4080,4083,4087,4090,4093,4097,4110,4113,4117,4126,4129,4133,4137,4140,4144,4148,4151,4225,4227,4230,4244],[11,3827,3828,3833,3834,3833,3838,3843],{},[15,3829,3832],{"href":3830,"rel":3831},"https://openai.com/index/introducing-operator/",[19],"Operator",", ",[15,3835,1261],{"href":3836,"rel":3837},"https://browser-use.com/",[19],[15,3839,3842],{"href":3840,"rel":3841},"https://www.skyvern.com/",[19],"Skyvern"," – we are currently witnessing an evolution of autonomous web browsing agents. The concept of agents is, in general, ubiquitous in the era of AI.",[70,3845,3847],{"id":3846},"what-is-an-agent","What is an Agent?",[11,3849,3850,3851,1116],{},"An agent is an entity that acts on behalf of another entity – its principal. Robotic agents are modelled after humans: they perceive the world and act in it to achieve a specified goal. Whilst humans perceive through their senses, agents perceive through sensors; whilst humans act through motor systems (e.g., fingers), agents act through actuators. This terminology has been transferred from hard- to software agents, particularly those premised on artificial intelligence (AI). An agent's purpose, that is, its goal, is clearly defined; the principal interfaces with that goal by assigning the agent tasks",[1239,3852,3853],{},[15,3854,1247],{"href":1243,"ariaDescribedBy":3855,"dataFootnoteRef":259,"id":1246},[1245],[23,3857],{"alt":3858,"loading":28,"src":3859,"provider":29,":width":3860},"Figurative comparison of component equivalents between a human and an AI agent","/blog/a-gentle-introduction-to-ai-agents-for-the-web/1.svg","430",[11,3862,3863],{},"The analogy is more than decorative: an agent takes over the tool-use expertise, leaving the principal with only the domain expertise. Traditional software applications like Photoshop demand both.",[118,3865,3867],{"id":3866},"what-makes-an-agent-intelligent","What Makes an Agent Intelligent?",[11,3869,3870],{},"For acting on behalf of a human, developing an agent could be as simple as making it follow a hardcoded sequence of actions. This describes process automation, but not an artificially intelligent agent. Whilst definitions for AI differ, agent intelligence comes from making informed decisions that are not hardcoded. Here are common – not necessarily mutually exclusive – high-level approaches to induce intelligence:",[51,3872,3873,3879,3885],{},[54,3874,3875,3878],{},[35,3876,3877],{},"Search"," – the agent has a complete or heuristic model of the world and the action space to anticipate the effects of actions and search for a goal state.",[54,3880,3881,3884],{},[35,3882,3883],{},"Reinforcement learning"," – the agent has an initial action space and iteratively discovers the world and effects of actions, learning an optimal policy to act in order to reach a goal state.",[54,3886,3887,3890],{},[35,3888,3889],{},"Neural networks"," – the agent has no model, only examples: a network is fitted to task–solution instances and generalises to unseen, yet related tasks.",[118,3892,3894],{"id":3893},"a-quick-note-on-multi-agent-systems","A Quick Note on Multi-Agent Systems",[11,3896,3897,3898,3903],{},"Agent design gets an additional edge when other entities act in the same world as the agent. Such entities can be considered agents on their own. Their beliefs and goals might affect the agent: cooperative agents share the designed agent's goal, whilst adversarial agents have an opposite goal. In a zero-sum game, an adversarial agent maximises the exact inverse of the designed agent's utility. Knowing this, the designed agent needs to anticipate intelligent decisions by the adversary (e.g., by applying the ",[15,3899,3902],{"href":3900,"rel":3901},"https://cs.stanford.edu/people/eroberts/courses/soco/projects/2003-04/intelligent-search/minimax.html",[19],"Minimax algorithm","), as opposed to random world changes. That said, web browsing – modelled after humans – is not a typical multi-agent scenario.",[118,3905,3907],{"id":3906},"the-agent-loop","The Agent Loop",[11,3909,3910,3915,3916,3923],{},[15,3911,3914],{"href":3912,"rel":3913},"https://www.nngroup.com/people/don-norman/",[19],"Don Norman","'s ",[15,3917,3920],{"href":3918,"rel":3919},"https://jnd.org/books/the-design-of-everyday-things-revised-and-expanded-edition/",[19],[84,3921,3922],{},"Seven Stages of Action"," is a model of the human cognition cycle that generalises to an agent's lifecycle – the agent loop: with a (1) goal, (2) intent to act, (3) plan actions, (4) execute the action plan – this alters the world – then (5) perceive the new world state, (6) interpret the perception, and finally (7) evaluate the interpretation – report task success or an irrevocable failure; continue the cycle if the world state not yet reflects a goal state.",[23,3925],{"alt":3926,"loading":28,"src":3927,"provider":29,":width":3928},"Don Norman's Seven Stages of Action – a model of human cognition","/blog/a-gentle-introduction-to-ai-agents-for-the-web/2.svg","500",[11,3930,3931],{},"Humans can run many of these cycles hierarchically, with increasing frequency the smaller the task. Take driving a car as an example: choosing a route runs on the order of minutes, choosing when to change lanes on seconds, and holding the car in its lane on fractions of a second. A cycle for holding the lane (1) is: intent to actively hold the lane (2), plan to keep the steering wheel straight, then watch the road (3), execute plan (4), observe the road (5), check whether the car is in the lane (6), if not, cycle again with an adjusted action plan (e.g., slightly rotate the steering wheel to the right) (7).",[118,3933,3935],{"id":3934},"errors-and-recovery","Errors and Recovery",[11,3937,3938,3939,3942,3943,3946],{},"Precisely speaking, \"to err is human\" means that errors are inevitable when heuristically navigating the world. Consequently, AI agents regularly run into errors, too. Norman's Seven Stages of Action make two error spaces apparent: when executing an action plan; the ",[84,3940,3941],{},"Gulf of Action (Execution)"," – how to act in the world? – and when perceiving world state; the ",[84,3944,3945],{},"Gulf of Perception (Evaluation)"," – how to understand the world?",[23,3948],{"alt":3949,"loading":28,"src":3950,"provider":29,":width":3928},"The Gulf of Action (Execution) and  Perception (Evaluation) in Don Norman's Seven Stages of Action","/blog/a-gentle-introduction-to-ai-agents-for-the-web/3.svg",[11,3952,3953],{},"There are two notable kinds of errors within the Gulf of Action: mistakes and slips. A mistake occurs when the action plan is wrong, although executing this action plan might go right in itself. A slip occurs when the action plan is right, but the execution goes wrong. Here is an example: stopping a manual-shift car without pressing the clutch. This error describes a mistake if the driver did not know about the need to press the clutch, e.g., because they only knew how to operate automatic-drive cars. It is, in turn, a slip if they simply forgot to press the clutch because of fatigue. Since cognition cycles are hierarchic, most errors can be recovered within a cycle. Hearing a choking engine sound, the driver might adjust the action plan and press the clutch late. Errors that cannot be recovered – picture the car's engine dying – require adjusting the goal, which can be seen as moving on to a new cycle.",[70,3955,3957],{"id":3956},"web-browsing-agents","Web Browsing Agents",[11,3959,3960,3961,3964],{},"The advent of large language models (LLMs) has recently opened a new perspective on developing web agents – falling back on the neural networks approach to intelligence. An LLM can serve as a plug-in reasoning backend: given a task and somehow encoded UI state, it can suggest input actions – type and target, e.g., click at ",[1196,3962,3963],{},"(200, 375)"," – that progress the given task. Moreover, it can decide whether a goal state has been reached. Providing a state-and-action history within context conveys to the model prior outcomes and contributes to in-context learning.",[118,3966,3968],{"id":3967},"web-browsing-is-all-about-the-journey","Web Browsing is All About the Journey",[11,3970,3971,3972,3975,3976,3981,3982,3987,3988,3991],{},"A web browsing task is any task that can be solved through a web-based user interface (UI). For instance: ",[84,3973,3974],{},"\"Book the cheapest flight from Amsterdam to New York tomorrow night\""," on ",[15,3977,3980],{"href":3978,"rel":3979},"https://www.klm.com",[19],"klm.com",". Application programming interfaces (APIs; e.g., ",[15,3983,3986],{"href":3984,"rel":3985},"https://www.ibm.com/think/topics/rest-apis",[19],"REST APIs",") are often restricted, incomplete, or unavailable. For that reason, web browsing agents today have to rely on human-facing web application UIs. Many agent use cases, such as for customer experience, even require a human in the loop. In this context, web browsing agents come at two levels: outcome-driven agents care only about reaching a goal state, which favours shortcuts like direct URL navigation or expert navigation like ",[1196,3989,3990],{},"Ctrl + F",", whilst journey-driven agents are supposed to browse a web application in human fashion, via the application-native UI.",[23,3993],{"alt":3994,"loading":28,"src":3995},"A depiction of a web journey over three web application UI states","/blog/a-gentle-introduction-to-ai-agents-for-the-web/4.png",[118,3997,3999],{"id":3998},"a-formal-model-for-web-browsing","A Formal Model for Web Browsing",[11,4001,4002],{},"The world of web application UIs is hard to formalise; a complete model spans the union of every reachable UI state in every web application on the web. For that reason, conventional AI has failed to produce capable generalist web agents. From a web agent design perspective, however, an abstract model of web browsing is sufficient.",[11,4004,4005],{},"Web browsing might afford a model that is history-independent. A human could certainly pick up a partially completed browsing task. For example, completing the payment process on an airline website for an already selected flight – assuming prior actions were correct with respect to the specified task (e.g., the selected destination).",[23,4007],{"alt":4008,"loading":28,"src":4009},"Screenshot of the payment page on klm.com that reflects outcomes of prior actions and affords next actions","/blog/a-gentle-introduction-to-ai-agents-for-the-web/5.png",[11,4011,4012,4013,4018,4019,4024,4025,4030],{},"Yet, many web applications lack ",[15,4014,4017],{"href":4015,"rel":4016},"https://www.nngroup.com/articles/ten-usability-heuristics/#toc-1-visibility-of-system-status-1",[19],"visibility of system status"," – against ",[15,4020,4023],{"href":4021,"rel":4022},"https://www.nngroup.com/people/jakob-nielsen/",[19],"Jakob Nielsen","'s first heuristic for UI usability. Visibility of system status comprises reflecting both outcomes of prior actions and affordances for next actions in the UI. With this concept, web browsing is best described as a partially observable Markov decision process (POMDP)",[1239,4026,4027],{},[15,4028,1270],{"href":1267,"ariaDescribedBy":4029,"dataFootnoteRef":259,"id":1269},[1245]," with, adapting to discovered UI states, where states can be tuples of UI state trajectories to contain history.",[11,4032,4033],{},"Reflex agents take actions that solely depend on the current world state. Planning agents, in turn, plan ahead. For most web browsing tasks, future UI states are roughly predictable, given that web journeys are similar across different web applications that help with similar tasks. In POMDP terms, the expected next UI state has a high probability, whilst unexpected states remain with a non-negligible probability. The further a web journey is progressed, the higher the probability of the next expected UI state gets: the flight selection step is likely followed by the passenger details step, which is even more likely followed by the payment step.",[23,4035],{"alt":4036,"loading":28,"src":4037},"Graph-based web journey representation of an unknown, adaptive, partially observable Markov decision process (POMDP)","/blog/a-gentle-introduction-to-ai-agents-for-the-web/6.png",[118,4039,4041],{"id":4040},"llm-sensors-actuators-agent","LLM + Sensors + Actuators = Agent",[11,4043,4044,4045,4050],{},"Conversationally satisfying simple information requests does not reasonably qualify an LLM as an agent – no more than it would a TF-IDF search system",[1239,4046,4047],{},[15,4048,1285],{"href":1282,"ariaDescribedBy":4049,"dataFootnoteRef":259,"id":1284},[1245],", or a colleague telling you the time. The web agent LLM backend is metaphorically the brain of the agent. It abstracts the world model and state transition function, and can decide whether a goal state was reached.",[11,4052,4053,4054,3833,4056,3833,4058,4061],{},"Turning an LLM into a web agent means, at the core, connecting it with sensors and actuators, for example, the ability to take DOM snapshots and dispatch clicks via CSS selectors. This is a specialised architecture of LLM tool calling. The set of available input actions is known from human computer use: ",[84,4055,1134],{},[84,4057,1137],{},[84,4059,4060],{},"scrolling",", etc. The agent loop is then a continuous cycle of serialising UI state for the model to elicit action suggestions under a goal derived from the user-specified task.",[23,4063],{"alt":4064,"loading":28,"src":4065,"provider":29,":width":4066},"Figurative depiction of components in the agent loop","/blog/a-gentle-introduction-to-ai-agents-for-the-web/7.svg","460",[118,4068,4070],{"id":4069},"snapshots-serialised-ui-state","Snapshots – Serialised UI State",[11,4072,4073,4074,4079],{},"There are plenty of ways to serialise the state of a web-based UI – referred to as a snapshot. A screenshot resembles what humans see when viewing a web page's graphical UI (GUI) at a given point in time. That said, a UI can be represented in versatile ways. LLMs have demonstrated the ability to interpret HTML and even navigate a UI encoded in this way",[1239,4075,4076],{},[15,4077,1778],{"href":1775,"ariaDescribedBy":4078,"dataFootnoteRef":259,"id":1777},[1245],". The document object model (DOM) is a web browser's runtime model of a web page. Serialised as HTML, to that effect, DOM snapshots represent a compelling alternative to GUI snapshots, offering certain advantages, such as processability (e.g., filtering). Filtering the DOM by pruning hidden, disabled, or redundant subtrees saves LLM communication costs and works towards more stable and predictable journeys.",[11,4081,4082],{},"Outcome-driven web agents have used UI-invasive snapshot measures, such as grounding LLMs with bounding boxes around actionable elements.",[23,4084],{"alt":4085,"loading":28,"src":4086},"Example of a screenshot (GUI snapshot) with coloured bounding boxes that visually ground an LLM","/blog/a-gentle-introduction-to-ai-agents-for-the-web/8.png",[11,4088,4089],{},"The ideal snapshot representation depends on the task. To get an idea, classify between navigation and information retrieval tasks: navigation tasks require snapshots to preserve actionable elements, whilst retrieval tasks require them to preserve text – both can be sourced from DOM snapshots. Accessibility tree snapshots are another DOM-sourced representation. Web UIs rarely afford merely a single task, so many of their components are noise in any given snapshot – which is not just a UI serialisation, but one scoped to the task at hand. GUI snapshots become a necessity when the target application is graphics-based, such as a canvas-based application like Google Maps.",[11,4091,4092],{},"A snapshot impacts the width of Norman's Gulf of Perception: if a relevant element is omitted or diffused, the model is hindered in identifying correct actions to progress a web browsing task. The Gulf of Action mostly grows when the snapshot-encoded UI state and the live UI state have irrecoverably drifted apart. The agent loop should be fast to prevent such drift. The Gulf of Perception is much wider by the nature of the problem: what snapshot representation can an LLM interpret, and which representation is best?",[118,4094,4096],{"id":4095},"targeting-action-suggestions","Targeting – Action Suggestions",[11,4098,4099,4100,3833,4102,4105,4106,4109],{},"An action is a user event, such as ",[84,4101,1189],{},[84,4103,4104],{},"type",", or ",[84,4107,4108],{},"scroll",", targeting a specific element in the UI. The set of available actions thereby is defined by the accepted user input types which are generally derived and standardised over the mouse and keyboard devices.",[11,4111,4112],{},"The type of snapshot representation provided to an LLM dictates how an element can be targeted for an action. Vision-based, geometric targeting is indirect and absolute, i.e., by coordinates. DOM-based targeting is direct and relative, e.g., by CSS selectors or object references. Relative targeting remains valid if the UI shifts between snapshot and action time. A shift can happen due to many reasons, for instance, a lazy-loaded image.",[118,4114,4116],{"id":4115},"the-agent-harness","The Agent Harness",[11,4118,4119,4120,4125],{},"Any custom logic that sits between the LLM backend and the web page UI (problem world) is considered an agent harness – often linguistically equated with the agent in general. Common aspects of a harness include task decomposition, model routing, snapshot creation, history management, error recovery, and timeouts. ",[15,4121,4124],{"href":4122,"rel":4123},"https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents",[19],"System prompts"," are also part of the harness, instructing a general-purpose LLM to model a web agent for a specific domain – right in context.",[11,4127,4128],{},"Designing a harness is where the POMDP and Seven Stages of Action models come into play. The POMDP model helps with context engineering, for instance, how much history to provide to the LLM. The Seven Stages of Action set out problem components and artefacts, for example, the LLM or snapshots, respectively.",[23,4130],{"alt":4131,"loading":28,"src":4132,"provider":29},"Don Norman's Seven Stages of Action mapped to the agentic web browsing problem","/blog/a-gentle-introduction-to-ai-agents-for-the-web/9.svg",[118,4134,4136],{"id":4135},"the-model-context-protocol","The Model Context Protocol",[11,4138,4139],{},"LLM providers commonly offer a structured output interface, which allows eliciting well-formed action suggestions from an LLM. A client can subsequently parse and drive these suggested actions in the live user interface. Through the model context protocol (MCP) – another web-based RPC standard – structured output can be bridged with tools that LLMs can call directly, without an extra client trip.",[23,4141],{"alt":4142,"loading":28,"src":4143,"provider":29,":width":3928},"Figurative depiction of components in the agent loop using the model context protocol (MCP)","/blog/a-gentle-introduction-to-ai-agents-for-the-web/10.svg",[118,4145,4147],{"id":4146},"web-browser-environments","Web Browser Environments",[11,4149,4150],{},"When connected to web perception and actuation tools via MCP, an ordinary AI chat UI already qualifies as an agent UI; the MCP integration decides where the actual web browser runs. There are several types of browser runtime environment:",[51,4152,4153,4168,4181,4193],{},[54,4154,4155,4158,4159,4162,4163],{},[35,4156,4157],{},"Agentic browser"," – the agent's UI embeds a browser.",[4160,4161],"br",{},"\nExample: ",[15,4164,4167],{"href":4165,"rel":4166},"https://www.perplexity.ai/comet",[19],"Perplexity Comet",[54,4169,4170,4173,4174,4162,4176],{},[35,4171,4172],{},"Cloud browser"," – the agent spins up a browser in the cloud.",[4160,4175],{},[15,4177,4180],{"href":4178,"rel":4179},"https://www.browserbase.com",[19],"Browserbase",[54,4182,4183,4186,4187,4162,4189],{},[35,4184,4185],{},"Local browser (3rd-party browser)"," – the agent spins up an isolated browser locally.",[4160,4188],{},[15,4190,4192],{"href":1122,"rel":4191},[19],"Chromium + Playwright",[54,4194,4195,4198,4199],{},[35,4196,4197],{},"Local browser (1st-party browser)"," – the agent is embedded within a web application running in a user's personal browser, e.g., in the form of a chatbot.\n",[51,4200,4201,4214],{},[54,4202,4203,4206,4207,4162,4209],{},[35,4204,4205],{},"Implemented (1st-party app)"," – served directly through the website.",[4160,4208],{},[15,4210,4213],{"href":4211,"rel":4212},"https://www.cognigy.com/platform/ai-agent-studio",[19],"Cognigy Studio",[54,4215,4216,4219,4220,4162,4222],{},[35,4217,4218],{},"Augmented (3rd-party app)"," – served indirectly through a website proxy.",[4160,4221],{},[15,4223,1130],{"href":1128,"rel":4224},[19],[3660,4226],{},[11,4228,4229],{},"Building AI agents has been approached from different angles. LLMs have opened up new perspectives on the design and development of AI agents, particularly web agents, which work in a problem domain that is hard to grasp with a formal model. Whilst higher-level agent methodology can borrow much from established research, LLMs have introduced new challenges, such as how to take UI state snapshots for eliciting input action suggestions from such models.",[11,4231,4232,4233,4238,4239,1116],{},"At Webfuse, we research agentic web browsing and develop a platform for agentic browsing. Our web application reverse proxy allows LLMs to perceive and act on the live web. Use cases include connecting a chat UI like ",[15,4234,4237],{"href":4235,"rel":4236},"https://claude.com/",[19],"Claude",", or shipping ",[15,4240,4243],{"href":4241,"rel":4242},"https://github.com/webfuse-com/extension-elevenlabs-mcp",[19],"perceive-and-act-ready chatbots",[1542,4245,4247,4250],{"className":4246,"dataFootnotes":259},[3682],[70,4248,3687],{"className":4249,"id":1245},[3686],[296,4251,4252,4261,4267,4273],{},[54,4253,4254,3697,4258],{"id":3692},[15,4255,4256],{"href":4256,"rel":4257},"https://inst.eecs.berkeley.edu/~cs188/textbook/textbook_full.pdf",[19],[15,4259,3704],{"href":3700,"ariaLabel":3701,"className":4260,"dataFootnoteBackref":259},[3703],[54,4262,4263,4264],{"id":3707},"A POMDP is a decision process in which the agent cannot see the exact world state, only partial observations of it, and so must choose actions against a probability distribution over the states it might be in. ",[15,4265,3704],{"href":3715,"ariaLabel":3716,"className":4266,"dataFootnoteBackref":259},[3703],[54,4268,4269,4270],{"id":3720},"TF-IDF (term frequency-inverse document frequency) is an information retrieval method that ranks documents by how often a query term occurs in each one, weighted down by how common that term is across the whole collection – so terms that are frequent here but rare elsewhere carry the most weight. ",[15,4271,3704],{"href":3727,"ariaLabel":3728,"className":4272,"dataFootnoteBackref":259},[3703],[54,4274,4275,3697,4278],{"id":3732},[15,4276,3723],{"href":3723,"rel":4277},[19],[15,4279,3704],{"href":3739,"ariaLabel":3740,"className":4280,"dataFootnoteBackref":259},[3703],{"title":259,"searchDepth":1021,"depth":1021,"links":4282},[4283,4289,4299],{"id":3846,"depth":1021,"text":3847,"children":4284},[4285,4286,4287,4288],{"id":3866,"depth":1026,"text":3867},{"id":3893,"depth":1026,"text":3894},{"id":3906,"depth":1026,"text":3907},{"id":3934,"depth":1026,"text":3935},{"id":3956,"depth":1021,"text":3957,"children":4290},[4291,4292,4293,4294,4295,4296,4297,4298],{"id":3967,"depth":1026,"text":3968},{"id":3998,"depth":1026,"text":3999},{"id":4040,"depth":1026,"text":4041},{"id":4069,"depth":1026,"text":4070},{"id":4095,"depth":1026,"text":4096},{"id":4115,"depth":1026,"text":4116},{"id":4135,"depth":1026,"text":4136},{"id":4146,"depth":1026,"text":4147},{"id":1245,"depth":1021,"text":3687},"2025-06-15","Get an idea of agents and how to build AI agents that browse the web: key concepts, terminology, and challenges.",{"homepage":1092,"relatedLinks":4303},[4304,4306,4307],{"text":4305,"href":3811},"Serialising Web UI State for LLMs",{"text":3806,"href":3807},{"text":3809,"href":3810,"external":1092},"/blog/a-gentle-introduction-to-ai-agents-for-the-web",{"title":3823,"description":4301},{"loc":4308},"blog/1011.a-gentle-introduction-to-ai-agents-for-the-web",[1100,4313,3816,3817,3818],"browser-agents","65hAoQNete8P3AgO-H42P0eBhVfIkl5vIfZZAkvL-9o",1791897078752]