[{"data":1,"prerenderedAt":3811},["ShallowReactive",2],{"/blog/serialising-web-ui-state-for-llms-the-complete-guide":3,"related-/blog/serialising-web-ui-state-for-llms-the-complete-guide":2753},{"id":4,"title":5,"authorId":6,"body":7,"category":2731,"created":2732,"description":2733,"extension":1822,"faqs":2734,"featurePriority":2734,"head":2734,"hideCta":1834,"landingPath":2734,"meta":2735,"navigation":1834,"ogImage":2734,"path":2743,"robots":2734,"schemaOrg":2734,"seo":2744,"sitemap":2745,"stem":2746,"tags":2747,"__hash__":2752},"blog/blog/1013.serialising-web-ui-state-for-llms-the-complete-guide.md","Serialising Web UI State for LLMs: The Complete Guide","thassilo-schiepanski",{"type":8,"value":9,"toc":2712},"minimark",[10,19,44,49,52,57,60,63,72,77,82,89,107,111,114,117,121,128,132,137,140,158,162,180,184,196,210,214,228,232,235,239,257,260,264,274,279,355,358,361,366,437,442,561,566,675,679,690,695,997,1002,1156,1165,1169,1180,1185,1189,1193,1196,1199,1202,1312,1317,1320,1324,1327,1332,1556,1561,1813,1818,1896,1901,1908,1917,1921,1924,1927,1930,1935,1944,1948,2288,2308,2361,2365,2368,2400,2404,2407,2562,2566,2569,2572,2576,2587,2590,2601,2607,2708],[11,12,13,14,18],"p",{},"LLMs have become a core component of AI web browsing agents. Previously, automating a web journey required formal models of both web browsing and web-based user interfaces (UIs). An LLM can serve as a plug-in reasoning backend: given serialised web UI state, it is expected to suggest input actions that progress a specified web browsing task. Serialised UI state is, more ergonomically, referred to as a ",[15,16,17],"strong",{},"snapshot",".",[11,20,21,22,29,30,35,36,29,40,43],{},"Snapshots and action suggestions are the two key artefacts along the agent loop. Action suggestions are elicited from the web agent LLM backend. Automated web browser environment, such as ",[23,24,28],"a",{"href":25,"rel":26},"https://playwright.dev",[27],"nofollow","Playwright"," or ",[23,31,34],{"href":32,"rel":33},"https://www.webfuse.com",[27],"Webfuse"," can be used to act out a suggestion, for example, ",[37,38,39],"em",{},"click",[37,41,42],{},"type",". The snapshot, on the other hand, is an artefact created by the agent harness, that is, the individual application component between the LLM and the web browser.",[45,46],"nuxt-picture",{"src":47,":alt":48},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/1.svg","{\"Figurative depiction of two key artefacts along the agent loop\":\"the snapshot towards the LLM and the input action towards the browser\",\"provider\":\"none\",\"loading\":\"lazy\",\"width\":460}",[11,50,51],{},"Creating snapshots for LLMs is a central challenge of developing efficient web agents. It is a trade-off between representation utility – how well models can interpret the such encoded UI – and cost LLM token costs. Indeed, snapshots are the major cost factor of an agentic web journey, accumulated over multiple rounds of the loop.",[53,54,56],"h2",{"id":55},"gui-snapshots","GUI Snapshots",[11,58,59],{},"Screenshots, i.e., GUI snapshots, resemble what humans see when viewing a web page at a given point in time. Early LLM-based web agents were primarily premised on GUI snapshots because of this evident equivalence. By design, the GUI surfaces only the UI context designated to be visible to the user.",[11,61,62],{},"The document object model (DOM) is a web browser's model of a web application UI's runtime state. With few exceptions, the DOM is the single source of truth regarding the UI. Most snapshot representations are thus, technically speaking, derivatives of a raw DOM snapshot. For the cases not reflected in the DOM, GUI snapshots are a first-class base representation.",[11,64,65,66,71],{},"Rendering a GUI snapshot from the DOM is, moreover, non-trivial; no client-side library manages to create GUI snapshots in reasonable quality or time – ",[23,67,70],{"href":68,"rel":69},"https://html2canvas.hertzen.com",[27],"HTML2Canvas"," demonstrates the limits.",[45,73],{"src":74,"alt":75,"loading":76},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/2.png","The same web-based user interface serialised via browser-native screenshot API and the client-side library HTML2Canvas showing significant geometric differences","lazy",[78,79,81],"h3",{"id":80},"vision-based-targeting","Vision-Based Targeting",[11,83,84,85,88],{},"The snapshot representation dictates how elements can be targeted in the live UI. Unambiguous targeting based on a visual representation must be either geometric (absolute), e.g., via point coordinates, or semantic (relative), e.g., via cues, such as ",[37,86,87],{},"\"Button with Text 'Shop'\"",". On its own, semantic targeting introduces a circular problem: which element has this text?",[11,90,91,92,95,96,98,99,102,103,18],{},"Consider the web browsing task: ",[37,93,94],{},"\"Navigate to the Apple online shop\"",". In combination with an action, in this case ",[37,97,39],{},", the agent could target the ",[37,100,101],{},"Shop"," button via a point coordinate, e.g., ",[104,105,106],"code",{},"(725, 750)",[45,108],{"src":109,"alt":110,"loading":76},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/3.png","A GUI snapshot (screenshot) visualising point coordinate targeting",[11,112,113],{},"When the UI shifts between snapshot and action time, geometric targeting suggestions might have become obsolete.",[11,115,116],{},"Depending on the web browser environment, screenshots might be constrained to the GUI viewport. The secondary task of scrolling is imperative with viewport-contrained GUI snapshots.",[78,118,120],{"id":119},"the-problem-with-gui-snapshots","The Problem with GUI Snapshots",[11,122,123,124,127],{},"Image data is downsampled to make it economically viable as input to LLMs. It reduces LLM input tokens to a few thousand for real-world-representative full-page screenshots. A common downsampling approach is averaging ",[37,125,126],{},"n×n","-tiles of pixels, as known from image downscaling. Side effects are: pixel-precise targeting does not map back to the live UI, and small geometric features are not preserved (e.g., small labels). Downsampling effects can be accounted for under certain circumstances, e.g., by providing a snapshot format that downsamples to its identity. However, GUI aspect ratio and detail might be beyond the agent's control.",[45,129],{"src":130,"alt":131,"loading":76},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/4.png","A GUI snapshot (screenshot) compared to its figuratively downsampled counterpart",[133,134,136],"h4",{"id":135},"grounded-gui-snapshots","Grounded GUI Snapshots",[11,138,139],{},"Geometric targeting remains unreliable with certain models and responsive UIs. To rely on semantic targeting instead, the agent must link relevant elements in the live UI to semantics in the screenshot. Relevance usually equates with actionability, but ultimately depends on the snapshot's purpose.",[11,141,142,145,146,157],{},[37,143,144],{},"Set-of-Mark"," prompting is the go-to technique to create grounded GUI snapshots",[147,148,149],"sup",{},[23,150,156],{"href":151,"ariaDescribedBy":152,"dataFootnoteRef":154,"id":155},"#user-content-fn-1",[153],"footnote-label","","user-content-fnref-1","1",". In this context, relevant elements are parsed from the live DOM and indexed with a unique numeric identifier. Each indexed element in the live UI is then enhanced with a coloured bounding box associated with the respective identifier.",[45,159],{"src":160,"alt":161,"loading":76},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/5.png","A GUI snapshot (screenshot) visualising grounded cue targeting",[11,163,164,171,172,18],{},[23,165,168],{"href":166,"rel":167},"https://browser-use.com",[27],[37,169,170],{},"Browser Use"," agents implement Set-of-Mark-grounded GUI snapshots",[147,173,174],{},[23,175,179],{"href":176,"ariaDescribedBy":177,"dataFootnoteRef":154,"id":178},"#user-content-fn-2",[153],"user-content-fnref-2","2",[53,181,183],{"id":182},"dom-snapshots","DOM Snapshots",[11,185,186,187,195],{},"The DOM is parsed from HTML. In its latest state it can be serialised back to HTML – including all runtime mutations. Research has supported LLMs' capabilities to interpret HTML, and even to navigate such encoded UIs",[147,188,189],{},[23,190,194],{"href":191,"ariaDescribedBy":192,"dataFootnoteRef":154,"id":193},"#user-content-fn-3",[153],"user-content-fnref-3","3",". For that reason, DOM snapshots are a compelling alternative to GUI snapshots. LLMs excel at code generation and review, so these capabilities are no surprise.",[11,197,198,199,202,203,206,207,209],{},"Cross-origin ",[104,200,201],{},"iframe"," elements and ",[104,204,205],{},"canvas"," elements are exceptions to the DOM-is-source-of-truth assumption: ",[104,208,205],{},"-drawn GUI is not reflected in the DOM, nor are external, self-contained DOM subtrees if no explicit policy allows it. Structurally, both are leaves in the serialisable DOM tree.",[78,211,213],{"id":212},"dom-based-targeting","DOM-Based Targeting",[11,215,216,217,220,221,224,225,18],{},"Programmatically, DOM elements can be targeted by different relative means. CSS selectors abstract targeting with a standalone language. Whilst CSS selectors can resolve to more than one element in the DOM – compare semantics of the HTML ",[104,218,219],{},"class"," and ",[104,222,223],{},"id"," attributes – every element has at least one unique CSS selector, constructed from nth-child node paths, e.g., ",[104,226,227],{},"main .section:nth-child(2) > .row:nth-of-type(5) > form button",[45,229],{"src":230,"alt":231,"loading":76},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/6.png","A DOM snapshot (HTML) visualising CSS selector targeting",[11,233,234],{},"A raw DOM snapshot is complete, beyond the GUI that renders in the spatially limited browser viewport. For a human-like visual browsing experience, scrolling can be abstracted by an indirect scroll-target-into-view mechanism.",[78,236,238],{"id":237},"the-problem-with-dom-snapshots","The Problem with DOM Snapshots",[11,240,241,242,247,248,256],{},"DOM snapshots of real-world web pages may come at a size of several megabytes, which converts to millions of LLM input tokens – we estimate a raw snapshot of ",[23,243,246],{"href":244,"rel":245},"https://edition.cnn.com",[27],"cnn.com"," at around 1.5M tokens. At this size, DOM snapshots over-utilise the context window of many LLMs, or even exceed it (such as of ",[23,249,252,255],{"href":250,"rel":251},"https://openai.com/index/gpt-4o-system-card/",[27],[37,253,254],{},"GPT-4o"," (OpenAI)","). Ultimately, raw DOM snapshots out-cost GUI snapshots.",[11,258,259],{},"Using DOM snapshots with LLM-based web agents requires size-reducing transformations, similar to image downsampling implemented behind LLM APIs.",[133,261,263],{"id":262},"extracted-dom-snapshots","Extracted DOM Snapshots",[11,265,266,267,270,271,273],{},"The naïve approach to use information from the DOM as a snapshot is element extraction: relevant elements are copied from the live DOM to an initially empty virtual DOM. For top-",[37,268,269],{},"k"," extractions – the ",[37,272,269],{}," highest-scoring elements – a snapshot of the virtual DOM comes at a fraction of the original DOM's size.",[11,275,276],{},[15,277,278],{},"Top-3 DOM Snapshot:",[280,281,285],"pre",{"className":282,"code":283,"language":284,"meta":154,"style":154},"language-html shiki shiki-themes github-dark github-dark","\u003Cbutton type=\"submit\">Submit\u003C/button>\n\u003Cbutton>More\u003C/button>\n\u003Ca href=\"/more\">Find out more\u003C/a>\n","html",[104,286,287,319,333],{"__ignoreMap":154},[288,289,292,296,300,304,307,311,314,316],"span",{"class":290,"line":291},"line",1,[288,293,295],{"class":294},"suv1-","\u003C",[288,297,299],{"class":298},"sxg3X","button",[288,301,303],{"class":302},"sFR8T"," type",[288,305,306],{"class":294},"=",[288,308,310],{"class":309},"s4wv1","\"submit\"",[288,312,313],{"class":294},">Submit\u003C/",[288,315,299],{"class":298},[288,317,318],{"class":294},">\n",[288,320,322,324,326,329,331],{"class":290,"line":321},2,[288,323,295],{"class":294},[288,325,299],{"class":298},[288,327,328],{"class":294},">More\u003C/",[288,330,299],{"class":298},[288,332,318],{"class":294},[288,334,336,338,340,343,345,348,351,353],{"class":290,"line":335},3,[288,337,295],{"class":294},[288,339,23],{"class":298},[288,341,342],{"class":302}," href",[288,344,306],{"class":294},[288,346,347],{"class":309},"\"/more\"",[288,349,350],{"class":294},">Find out more\u003C/",[288,352,23],{"class":298},[288,354,318],{"class":294},[11,356,357],{},"If done manually, element extraction presupposes correct assumptions about which elements and element properties count as relevant. ALternatively, element relevance scoring can be outsourced to an LLM, at the cost of another inference step.",[11,359,360],{},"Extraction disposes of deep hierarchy, which might be salient to the web agent LLM backend. From the below extracted DOM snapshot alone, the original content relationship is lost:",[11,362,363],{},[15,364,365],{},"Extracted DOM Snapshot:",[280,367,369],{"className":282,"code":368,"language":284,"meta":154,"style":154},"\u003Cstrong>MacBook Air\u003C/strong>\n\u003Cbutton>Add to cart\u003C/button>\n\u003Cp>Out of stock\u003C/p>\n\u003Ch2>MacBook Pro\u003C/h2>\n\u003Cbutton>Add to cart\u003C/button>\n",[104,370,371,384,397,410,424],{"__ignoreMap":154},[288,372,373,375,377,380,382],{"class":290,"line":291},[288,374,295],{"class":294},[288,376,15],{"class":298},[288,378,379],{"class":294},">MacBook Air\u003C/",[288,381,15],{"class":298},[288,383,318],{"class":294},[288,385,386,388,390,393,395],{"class":290,"line":321},[288,387,295],{"class":294},[288,389,299],{"class":298},[288,391,392],{"class":294},">Add to cart\u003C/",[288,394,299],{"class":298},[288,396,318],{"class":294},[288,398,399,401,403,406,408],{"class":290,"line":335},[288,400,295],{"class":294},[288,402,11],{"class":298},[288,404,405],{"class":294},">Out of stock\u003C/",[288,407,11],{"class":298},[288,409,318],{"class":294},[288,411,413,415,417,420,422],{"class":290,"line":412},4,[288,414,295],{"class":294},[288,416,53],{"class":298},[288,418,419],{"class":294},">MacBook Pro\u003C/",[288,421,53],{"class":298},[288,423,318],{"class":294},[288,425,427,429,431,433,435],{"class":290,"line":426},5,[288,428,295],{"class":294},[288,430,299],{"class":298},[288,432,392],{"class":294},[288,434,299],{"class":298},[288,436,318],{"class":294},[11,438,439],{},[15,440,441],{},"Original DOM Snapshot – Candidate A:",[280,443,445],{"className":282,"code":444,"language":284,"meta":154,"style":154},"\u003Csection class=\"product\">\n  \u003Cstrong>MacBook Air\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n  \u003Cp>Out of stock\u003C/p>  👈\n\u003C/section>\n\u003Csection class=\"product\">\n  \u003Cstrong>MacBook Pro\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n\u003C/section>\n",[104,446,447,464,477,489,502,511,526,539,552],{"__ignoreMap":154},[288,448,449,451,454,457,459,462],{"class":290,"line":291},[288,450,295],{"class":294},[288,452,453],{"class":298},"section",[288,455,456],{"class":302}," class",[288,458,306],{"class":294},[288,460,461],{"class":309},"\"product\"",[288,463,318],{"class":294},[288,465,466,469,471,473,475],{"class":290,"line":321},[288,467,468],{"class":294},"  \u003C",[288,470,15],{"class":298},[288,472,379],{"class":294},[288,474,15],{"class":298},[288,476,318],{"class":294},[288,478,479,481,483,485,487],{"class":290,"line":335},[288,480,468],{"class":294},[288,482,299],{"class":298},[288,484,392],{"class":294},[288,486,299],{"class":298},[288,488,318],{"class":294},[288,490,491,493,495,497,499],{"class":290,"line":412},[288,492,468],{"class":294},[288,494,11],{"class":298},[288,496,405],{"class":294},[288,498,11],{"class":298},[288,500,501],{"class":294},">  👈\n",[288,503,504,507,509],{"class":290,"line":426},[288,505,506],{"class":294},"\u003C/",[288,508,453],{"class":298},[288,510,318],{"class":294},[288,512,514,516,518,520,522,524],{"class":290,"line":513},6,[288,515,295],{"class":294},[288,517,453],{"class":298},[288,519,456],{"class":302},[288,521,306],{"class":294},[288,523,461],{"class":309},[288,525,318],{"class":294},[288,527,529,531,533,535,537],{"class":290,"line":528},7,[288,530,468],{"class":294},[288,532,15],{"class":298},[288,534,419],{"class":294},[288,536,15],{"class":298},[288,538,318],{"class":294},[288,540,542,544,546,548,550],{"class":290,"line":541},8,[288,543,468],{"class":294},[288,545,299],{"class":298},[288,547,392],{"class":294},[288,549,299],{"class":298},[288,551,318],{"class":294},[288,553,555,557,559],{"class":290,"line":554},9,[288,556,506],{"class":294},[288,558,453],{"class":298},[288,560,318],{"class":294},[11,562,563],{},[15,564,565],{},"Original DOM Snapshot – Candidate B:",[280,567,569],{"className":282,"code":568,"language":284,"meta":154,"style":154},"\u003Csection class=\"product\">\n  \u003Cstrong>MacBook Air\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n\u003C/section>\n\u003Csection class=\"product\">\n  \u003Cp>Out of stock\u003C/p>  👈\n  \u003Cstrong>MacBook Pro\u003C/strong>\n  \u003Cbutton>Add to cart\u003C/button>\n\u003C/section>\n",[104,570,571,585,597,609,617,631,643,655,667],{"__ignoreMap":154},[288,572,573,575,577,579,581,583],{"class":290,"line":291},[288,574,295],{"class":294},[288,576,453],{"class":298},[288,578,456],{"class":302},[288,580,306],{"class":294},[288,582,461],{"class":309},[288,584,318],{"class":294},[288,586,587,589,591,593,595],{"class":290,"line":321},[288,588,468],{"class":294},[288,590,15],{"class":298},[288,592,379],{"class":294},[288,594,15],{"class":298},[288,596,318],{"class":294},[288,598,599,601,603,605,607],{"class":290,"line":335},[288,600,468],{"class":294},[288,602,299],{"class":298},[288,604,392],{"class":294},[288,606,299],{"class":298},[288,608,318],{"class":294},[288,610,611,613,615],{"class":290,"line":412},[288,612,506],{"class":294},[288,614,453],{"class":298},[288,616,318],{"class":294},[288,618,619,621,623,625,627,629],{"class":290,"line":426},[288,620,295],{"class":294},[288,622,453],{"class":298},[288,624,456],{"class":302},[288,626,306],{"class":294},[288,628,461],{"class":309},[288,630,318],{"class":294},[288,632,633,635,637,639,641],{"class":290,"line":513},[288,634,468],{"class":294},[288,636,11],{"class":298},[288,638,405],{"class":294},[288,640,11],{"class":298},[288,642,501],{"class":294},[288,644,645,647,649,651,653],{"class":290,"line":528},[288,646,468],{"class":294},[288,648,15],{"class":298},[288,650,419],{"class":294},[288,652,15],{"class":298},[288,654,318],{"class":294},[288,656,657,659,661,663,665],{"class":290,"line":541},[288,658,468],{"class":294},[288,660,299],{"class":298},[288,662,392],{"class":294},[288,664,299],{"class":298},[288,666,318],{"class":294},[288,668,669,671,673],{"class":290,"line":554},[288,670,506],{"class":294},[288,672,453],{"class":298},[288,674,318],{"class":294},[133,676,678],{"id":677},"downsampled-dom-snapshots","Downsampled DOM Snapshots",[11,680,681,682,18],{},"Downsampling means reducing data point count whilst preserving the overall data shape and most of the relevant features. Picture an image being downsized: the depicted object remains recognisable to a large degree. The concept of image downsampling can be transferred to the DOM by locally 'averaging' DOM subtrees – trading fidelity for size",[147,683,684],{},[23,685,689],{"href":686,"ariaDescribedBy":687,"dataFootnoteRef":154,"id":688},"#user-content-fn-4",[153],"user-content-fnref-4","4",[11,691,692],{},[15,693,694],{},"Original DOM Snapshot:",[280,696,698],{"className":282,"code":697,"language":284,"meta":154,"style":154},"\u003Csection class=\"container\" tabindex=\"3\" required=\"true\" type=\"example\">\n  \u003Cdiv class=\"mx-auto\" data-topic=\"products\" required=\"false\">\n    \u003Ch1>Our Pizza\u003C/h1>\n    \u003Cdiv>\n      \u003Cdiv class=\"shadow-lg\">\n        \u003Ch2>Margherita\u003C/h2>\n        \u003Cp>\n         A simple classic: mozzarella, tomatoes and basil.\n         An everyday choice!\n        \u003C/p>\n        \u003Cbutton type=\"button\">Add\u003C/button>\n      \u003C/div>\n      \u003Cdiv class=\"shadow-lg\">\n        \u003Ch2>Capricciosa\u003C/h2>\n        \u003Cp>\n          A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n          A true favourite!\n        \u003C/p>\n        \u003Cbutton type=\"button\">Add\u003C/button>\n      \u003C/div>\n    \u003C/div>\n  \u003C/div>\n\u003C/section>\n",[104,699,700,738,769,784,792,808,822,830,835,840,850,871,881,896,910,919,925,931,940,959,968,978,988],{"__ignoreMap":154},[288,701,702,704,706,708,710,713,716,718,721,724,726,729,731,733,736],{"class":290,"line":291},[288,703,295],{"class":294},[288,705,453],{"class":298},[288,707,456],{"class":302},[288,709,306],{"class":294},[288,711,712],{"class":309},"\"container\"",[288,714,715],{"class":302}," tabindex",[288,717,306],{"class":294},[288,719,720],{"class":309},"\"3\"",[288,722,723],{"class":302}," required",[288,725,306],{"class":294},[288,727,728],{"class":309},"\"true\"",[288,730,303],{"class":302},[288,732,306],{"class":294},[288,734,735],{"class":309},"\"example\"",[288,737,318],{"class":294},[288,739,740,742,745,747,749,752,755,757,760,762,764,767],{"class":290,"line":321},[288,741,468],{"class":294},[288,743,744],{"class":298},"div",[288,746,456],{"class":302},[288,748,306],{"class":294},[288,750,751],{"class":309},"\"mx-auto\"",[288,753,754],{"class":302}," data-topic",[288,756,306],{"class":294},[288,758,759],{"class":309},"\"products\"",[288,761,723],{"class":302},[288,763,306],{"class":294},[288,765,766],{"class":309},"\"false\"",[288,768,318],{"class":294},[288,770,771,774,777,780,782],{"class":290,"line":335},[288,772,773],{"class":294},"    \u003C",[288,775,776],{"class":298},"h1",[288,778,779],{"class":294},">Our Pizza\u003C/",[288,781,776],{"class":298},[288,783,318],{"class":294},[288,785,786,788,790],{"class":290,"line":412},[288,787,773],{"class":294},[288,789,744],{"class":298},[288,791,318],{"class":294},[288,793,794,797,799,801,803,806],{"class":290,"line":426},[288,795,796],{"class":294},"      \u003C",[288,798,744],{"class":298},[288,800,456],{"class":302},[288,802,306],{"class":294},[288,804,805],{"class":309},"\"shadow-lg\"",[288,807,318],{"class":294},[288,809,810,813,815,818,820],{"class":290,"line":513},[288,811,812],{"class":294},"        \u003C",[288,814,53],{"class":298},[288,816,817],{"class":294},">Margherita\u003C/",[288,819,53],{"class":298},[288,821,318],{"class":294},[288,823,824,826,828],{"class":290,"line":528},[288,825,812],{"class":294},[288,827,11],{"class":298},[288,829,318],{"class":294},[288,831,832],{"class":290,"line":541},[288,833,834],{"class":294},"         A simple classic: mozzarella, tomatoes and basil.\n",[288,836,837],{"class":290,"line":554},[288,838,839],{"class":294},"         An everyday choice!\n",[288,841,843,846,848],{"class":290,"line":842},10,[288,844,845],{"class":294},"        \u003C/",[288,847,11],{"class":298},[288,849,318],{"class":294},[288,851,853,855,857,859,861,864,867,869],{"class":290,"line":852},11,[288,854,812],{"class":294},[288,856,299],{"class":298},[288,858,303],{"class":302},[288,860,306],{"class":294},[288,862,863],{"class":309},"\"button\"",[288,865,866],{"class":294},">Add\u003C/",[288,868,299],{"class":298},[288,870,318],{"class":294},[288,872,874,877,879],{"class":290,"line":873},12,[288,875,876],{"class":294},"      \u003C/",[288,878,744],{"class":298},[288,880,318],{"class":294},[288,882,884,886,888,890,892,894],{"class":290,"line":883},13,[288,885,796],{"class":294},[288,887,744],{"class":298},[288,889,456],{"class":302},[288,891,306],{"class":294},[288,893,805],{"class":309},[288,895,318],{"class":294},[288,897,899,901,903,906,908],{"class":290,"line":898},14,[288,900,812],{"class":294},[288,902,53],{"class":298},[288,904,905],{"class":294},">Capricciosa\u003C/",[288,907,53],{"class":298},[288,909,318],{"class":294},[288,911,913,915,917],{"class":290,"line":912},15,[288,914,812],{"class":294},[288,916,11],{"class":298},[288,918,318],{"class":294},[288,920,922],{"class":290,"line":921},16,[288,923,924],{"class":294},"          A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[288,926,928],{"class":290,"line":927},17,[288,929,930],{"class":294},"          A true favourite!\n",[288,932,934,936,938],{"class":290,"line":933},18,[288,935,845],{"class":294},[288,937,11],{"class":298},[288,939,318],{"class":294},[288,941,943,945,947,949,951,953,955,957],{"class":290,"line":942},19,[288,944,812],{"class":294},[288,946,299],{"class":298},[288,948,303],{"class":302},[288,950,306],{"class":294},[288,952,863],{"class":309},[288,954,866],{"class":294},[288,956,299],{"class":298},[288,958,318],{"class":294},[288,960,962,964,966],{"class":290,"line":961},20,[288,963,876],{"class":294},[288,965,744],{"class":298},[288,967,318],{"class":294},[288,969,971,974,976],{"class":290,"line":970},21,[288,972,973],{"class":294},"    \u003C/",[288,975,744],{"class":298},[288,977,318],{"class":294},[288,979,981,984,986],{"class":290,"line":980},22,[288,982,983],{"class":294},"  \u003C/",[288,985,744],{"class":298},[288,987,318],{"class":294},[288,989,991,993,995],{"class":290,"line":990},23,[288,992,506],{"class":294},[288,994,453],{"class":298},[288,996,318],{"class":294},[11,998,999],{},[15,1000,1001],{},"Downsampled DOM Snapshot:",[280,1003,1005],{"className":282,"code":1004,"language":284,"meta":154,"style":154},"\u003Csection class=\"container\" required=\"true\" type=\"example\">\n  \u003Cdiv class=\"mx-auto\" required=\"false\">\n    Our Pizza\n    \u003Cdiv>\n      Margherita\n      A simple classic: mozzarella, tomatoes and basil.\n      An everyday choice!\n      \u003Cbutton type=\"button\">Add\u003C/button>\n      Capricciosa\n      A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n      A true favourite!\n      \u003Cbutton type=\"button\">Add\u003C/button>\n    \u003C/div>\n  \u003C/div>\n\u003C/section>\n",[104,1006,1007,1033,1053,1058,1066,1071,1076,1081,1099,1104,1109,1114,1132,1140,1148],{"__ignoreMap":154},[288,1008,1009,1011,1013,1015,1017,1019,1021,1023,1025,1027,1029,1031],{"class":290,"line":291},[288,1010,295],{"class":294},[288,1012,453],{"class":298},[288,1014,456],{"class":302},[288,1016,306],{"class":294},[288,1018,712],{"class":309},[288,1020,723],{"class":302},[288,1022,306],{"class":294},[288,1024,728],{"class":309},[288,1026,303],{"class":302},[288,1028,306],{"class":294},[288,1030,735],{"class":309},[288,1032,318],{"class":294},[288,1034,1035,1037,1039,1041,1043,1045,1047,1049,1051],{"class":290,"line":321},[288,1036,468],{"class":294},[288,1038,744],{"class":298},[288,1040,456],{"class":302},[288,1042,306],{"class":294},[288,1044,751],{"class":309},[288,1046,723],{"class":302},[288,1048,306],{"class":294},[288,1050,766],{"class":309},[288,1052,318],{"class":294},[288,1054,1055],{"class":290,"line":335},[288,1056,1057],{"class":294},"    Our Pizza\n",[288,1059,1060,1062,1064],{"class":290,"line":412},[288,1061,773],{"class":294},[288,1063,744],{"class":298},[288,1065,318],{"class":294},[288,1067,1068],{"class":290,"line":426},[288,1069,1070],{"class":294},"      Margherita\n",[288,1072,1073],{"class":290,"line":513},[288,1074,1075],{"class":294},"      A simple classic: mozzarella, tomatoes and basil.\n",[288,1077,1078],{"class":290,"line":528},[288,1079,1080],{"class":294},"      An everyday choice!\n",[288,1082,1083,1085,1087,1089,1091,1093,1095,1097],{"class":290,"line":541},[288,1084,796],{"class":294},[288,1086,299],{"class":298},[288,1088,303],{"class":302},[288,1090,306],{"class":294},[288,1092,863],{"class":309},[288,1094,866],{"class":294},[288,1096,299],{"class":298},[288,1098,318],{"class":294},[288,1100,1101],{"class":290,"line":554},[288,1102,1103],{"class":294},"      Capricciosa\n",[288,1105,1106],{"class":290,"line":842},[288,1107,1108],{"class":294},"      A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[288,1110,1111],{"class":290,"line":852},[288,1112,1113],{"class":294},"      A true favourite!\n",[288,1115,1116,1118,1120,1122,1124,1126,1128,1130],{"class":290,"line":873},[288,1117,796],{"class":294},[288,1119,299],{"class":298},[288,1121,303],{"class":302},[288,1123,306],{"class":294},[288,1125,863],{"class":309},[288,1127,866],{"class":294},[288,1129,299],{"class":298},[288,1131,318],{"class":294},[288,1133,1134,1136,1138],{"class":290,"line":883},[288,1135,973],{"class":294},[288,1137,744],{"class":298},[288,1139,318],{"class":294},[288,1141,1142,1144,1146],{"class":290,"line":898},[288,1143,983],{"class":294},[288,1145,744],{"class":298},[288,1147,318],{"class":294},[288,1149,1150,1152,1154],{"class":290,"line":912},[288,1151,506],{"class":294},[288,1153,453],{"class":298},[288,1155,318],{"class":294},[11,1157,1158,1159,18],{},"When downsampling is defined over different types of nodes – for instance, elements, attributes, and text – it becomes a powerful technique to control DOM size in a snapshot pipeline. Adaptive implementations can moreover produce snapshots that fall below a hard token limit. Our evaluation revealed that moderate downsampling ratios comfortably fit the mean DOM snapshot of real-world-representative web pages within GPT-4o's context window – leaving headroom for an agent system prompt and a trace of reasoning and action history",[147,1160,1161],{},[23,1162,689],{"href":686,"ariaDescribedBy":1163,"dataFootnoteRef":154,"id":1164},[153],"user-content-fnref-4-2",[78,1166,1168],{"id":1167},"the-problem-with-dom-based-targeting","The Problem with DOM-Based Targeting",[11,1170,1171,1172,1175,1176,1179],{},"When DOM snapshots are transformation artefacts, relative CSS selectors like nth-child paths might not map back to the live DOM. There are two straightforward ways to account for breaking CSS selectors: (1) assigning unique IDs to relevant elements in the live UI that persist transformations (e.g., via a unique attribute ",[104,1173,1174],{},"UNIQUE-ID=\"27\"",", which can be selected via ",[104,1177,1178],{},"[UNIQUE-ID=\"27\"]","), and (2) maintaining a source map between elements in the live DOM and the transformed DOM. The latter approach does favourably not affect the snapshot directly.",[11,1181,1182],{},[15,1183,1184],{},"DOM Snapshot with Inlined IDs:",[45,1186],{"src":1187,"alt":1188,"loading":76},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/7.png","A DOM snapshot (HTML) visualising unique ID (attribute) targeting",[78,1190,1192],{"id":1191},"the-dom-based-snapshot-pipeline","The DOM-Based Snapshot Pipeline",[11,1194,1195],{},"LLMs are able to reason over HTML. Yet, HTML carries noise, such as obfuscated scripts or hidden subtrees. DOM snapshots should for that reason be seen as a base representation, rather than a target representation.",[11,1197,1198],{},"The concept of DOM downsampling can be utilised as a universal size control in a snapshot pipeline – ideally exposed as quality, the inverse property of downsampling. Moreover, the DOM can be pre-transformed by deterministic techniques, such as pruning hidden subtrees to surface only the tangible UI – a characteristic that is natural with GUI snapshots.",[11,1200,1201],{},"The pipeline approach traceably goes from the raw DOM to an abstract snapshot representation. For example:",[1203,1204,1205,1229,1235,1294,1300,1306],"ol",{},[1206,1207,1208,1211,1212,1217,1218,1221,1222,220,1225,1228],"li",{},[15,1209,1210],{},"Scope Subtree."," Scope in to a task-relevant subtree. An information retrieval task on ",[23,1213,1216],{"href":1214,"rel":1215},"https://www.wikipedia.org",[27],"wikipedia.org"," can scope into the ",[104,1219,1220],{},"main"," element – ",[104,1223,1224],{},"header",[104,1226,1227],{},"footer"," are irrelevant.",[1206,1230,1231,1234],{},[15,1232,1233],{},"Clone."," Clone the live DOM.",[1206,1236,1237,1240,1241],{},[15,1238,1239],{},"Filter."," Apply arbitrary filters to the cloned DOM (in memory). For example:\n",[1242,1243,1244,1282,1288],"ul",{},[1206,1245,1246,1249,1250],{},[15,1247,1248],{},"Prune Subtrees"," E.g.:\n",[1242,1251,1252,1258,1264],{},[1206,1253,1254,1257],{},[15,1255,1256],{},"Non-interactive Nodes",", e.g., hidden elements.",[1206,1259,1260,1263],{},[15,1261,1262],{},"Irrelevant Nodes",", e.g., comment nodes.",[1206,1265,1266,1269,1270,1272,1273,1275,1276,1279,1280,18],{},[15,1267,1268],{},"Redundant Nodes",", e.g., ",[104,1271,23],{}," in the ",[104,1274,1227],{}," that has a same-",[104,1277,1278],{},"href"," equivalent in the ",[104,1281,1224],{},[1206,1283,1284,1287],{},[15,1285,1286],{},"Sanitise."," E.g., sensitive information from content-editables.",[1206,1289,1290,1293],{},[15,1291,1292],{},"Clean."," E.g., remove stop words from text nodes.",[1206,1295,1296,1299],{},[15,1297,1298],{},"Serialise."," Serialise the cloned DOM to HTML.",[1206,1301,1302,1305],{},[15,1303,1304],{},"Downsample."," Downsample the HTML to control size.",[1206,1307,1308,1311],{},[15,1309,1310],{},"Translate."," Translate the downsampled HTML to any DOM-based representation, e.g., the accessibility tree.",[45,1313],{"src":1314,"alt":1315,"loading":76,"provider":1316},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/8.svg","Example of a DOM snapshot pipeline","none",[11,1318,1319],{},"A filtered DOM drastically increases stability of repeatedly run web journeys: if only a single button routes to a target page, the agent perceives an unambiguous action target element. The rule of thumb is: the less actionable options a snapshot transcribes, the better – given all relevant options are covered.",[133,1321,1323],{"id":1322},"arbitrary-snapshot-representations","Arbitrary Snapshot Representations",[11,1325,1326],{},"Any text-based snapshot representation can be derived from the single-source-of-truth DOM. Which representation is best highly depends on the given web browsing task. A high-level distinction is between (A) navigation and (B) information retrieval tasks. Navigation tasks do not require extensive text, but  actionable elements – if a snapshot omits a relevant button, the LLM is not able to use the encoded UI. Information retrieval tasks, on the other hand, require text, as the text contains topical information, whilst actionable elements do not matter.",[11,1328,1329],{},[15,1330,1331],{},"DOM Snapshot:",[280,1333,1335],{"className":282,"code":1334,"language":284,"meta":154,"style":154},"\u003Csection class=\"container\" required=\"true\" type=\"example\">\n  \u003Cdiv class=\"mx-auto\" required=\"false\">\n    \u003Ch1>Our Pizza\u003C/h1>\n    \u003Cdiv>\n      \u003Ch2>Margherita\u003C/h2>\n      \u003Cp>\n        A simple classic: mozzarella, tomatoes and basil.\n        An everyday choice!\n      \u003C/p>\n      \u003Ca href=\"add?p=margherita\" data-uid=\"1\">Add\u003C/a>\n      \u003Ch2>Capricciosa\u003C/h2>\n      \u003Cp>\n        A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n        A true favourite!\n      \u003C/p>\n      \u003Ca href=\"add?p=capricciosa\" data-uid=\"2\">Add\u003C/a>\n    \u003C/div>\n  \u003C/div>\n\u003C/section>\n",[104,1336,1337,1363,1383,1395,1403,1415,1423,1428,1433,1441,1468,1480,1488,1493,1498,1506,1532,1540,1548],{"__ignoreMap":154},[288,1338,1339,1341,1343,1345,1347,1349,1351,1353,1355,1357,1359,1361],{"class":290,"line":291},[288,1340,295],{"class":294},[288,1342,453],{"class":298},[288,1344,456],{"class":302},[288,1346,306],{"class":294},[288,1348,712],{"class":309},[288,1350,723],{"class":302},[288,1352,306],{"class":294},[288,1354,728],{"class":309},[288,1356,303],{"class":302},[288,1358,306],{"class":294},[288,1360,735],{"class":309},[288,1362,318],{"class":294},[288,1364,1365,1367,1369,1371,1373,1375,1377,1379,1381],{"class":290,"line":321},[288,1366,468],{"class":294},[288,1368,744],{"class":298},[288,1370,456],{"class":302},[288,1372,306],{"class":294},[288,1374,751],{"class":309},[288,1376,723],{"class":302},[288,1378,306],{"class":294},[288,1380,766],{"class":309},[288,1382,318],{"class":294},[288,1384,1385,1387,1389,1391,1393],{"class":290,"line":335},[288,1386,773],{"class":294},[288,1388,776],{"class":298},[288,1390,779],{"class":294},[288,1392,776],{"class":298},[288,1394,318],{"class":294},[288,1396,1397,1399,1401],{"class":290,"line":412},[288,1398,773],{"class":294},[288,1400,744],{"class":298},[288,1402,318],{"class":294},[288,1404,1405,1407,1409,1411,1413],{"class":290,"line":426},[288,1406,796],{"class":294},[288,1408,53],{"class":298},[288,1410,817],{"class":294},[288,1412,53],{"class":298},[288,1414,318],{"class":294},[288,1416,1417,1419,1421],{"class":290,"line":513},[288,1418,796],{"class":294},[288,1420,11],{"class":298},[288,1422,318],{"class":294},[288,1424,1425],{"class":290,"line":528},[288,1426,1427],{"class":294},"        A simple classic: mozzarella, tomatoes and basil.\n",[288,1429,1430],{"class":290,"line":541},[288,1431,1432],{"class":294},"        An everyday choice!\n",[288,1434,1435,1437,1439],{"class":290,"line":554},[288,1436,876],{"class":294},[288,1438,11],{"class":298},[288,1440,318],{"class":294},[288,1442,1443,1445,1447,1449,1451,1454,1457,1459,1462,1464,1466],{"class":290,"line":842},[288,1444,796],{"class":294},[288,1446,23],{"class":298},[288,1448,342],{"class":302},[288,1450,306],{"class":294},[288,1452,1453],{"class":309},"\"add?p=margherita\"",[288,1455,1456],{"class":302}," data-uid",[288,1458,306],{"class":294},[288,1460,1461],{"class":309},"\"1\"",[288,1463,866],{"class":294},[288,1465,23],{"class":298},[288,1467,318],{"class":294},[288,1469,1470,1472,1474,1476,1478],{"class":290,"line":852},[288,1471,796],{"class":294},[288,1473,53],{"class":298},[288,1475,905],{"class":294},[288,1477,53],{"class":298},[288,1479,318],{"class":294},[288,1481,1482,1484,1486],{"class":290,"line":873},[288,1483,796],{"class":294},[288,1485,11],{"class":298},[288,1487,318],{"class":294},[288,1489,1490],{"class":290,"line":883},[288,1491,1492],{"class":294},"        A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[288,1494,1495],{"class":290,"line":898},[288,1496,1497],{"class":294},"        A true favourite!\n",[288,1499,1500,1502,1504],{"class":290,"line":912},[288,1501,876],{"class":294},[288,1503,11],{"class":298},[288,1505,318],{"class":294},[288,1507,1508,1510,1512,1514,1516,1519,1521,1523,1526,1528,1530],{"class":290,"line":921},[288,1509,796],{"class":294},[288,1511,23],{"class":298},[288,1513,342],{"class":302},[288,1515,306],{"class":294},[288,1517,1518],{"class":309},"\"add?p=capricciosa\"",[288,1520,1456],{"class":302},[288,1522,306],{"class":294},[288,1524,1525],{"class":309},"\"2\"",[288,1527,866],{"class":294},[288,1529,23],{"class":298},[288,1531,318],{"class":294},[288,1533,1534,1536,1538],{"class":290,"line":927},[288,1535,973],{"class":294},[288,1537,744],{"class":298},[288,1539,318],{"class":294},[288,1541,1542,1544,1546],{"class":290,"line":933},[288,1543,983],{"class":294},[288,1545,744],{"class":298},[288,1547,318],{"class":294},[288,1549,1550,1552,1554],{"class":290,"line":942},[288,1551,506],{"class":294},[288,1553,453],{"class":298},[288,1555,318],{"class":294},[11,1557,1558],{},[15,1559,1560],{},"Accessibility Tree Snapshot:",[280,1562,1566],{"className":1563,"code":1564,"language":1565,"meta":154,"style":154},"language-yaml shiki shiki-themes github-dark github-dark","- document:\n  children:\n    - section:\n      children:\n        - heading:\n          name: \"Our Pizza\"\n          level: 1\n        - heading:\n          name: \"Margherita\"\n          level: 2\n        - paragraph:\n          children:\n            - text: \"A simple classic: mozzarella, tomatoes and basil. An everyday choice!\"\n        - link:\n          name: \"Add\"\n          url: \"add?p=margherita\"\n          data-uid: 1\n        - heading:\n          name: \"Capricciosa\"\n          level: 2\n        - paragraph:\n          children:\n            - text: \"A rich taste: mozzarella, ham, mushrooms, artichokes and olives. A true favourite!\"\n        - link:\n          name: \"Add\"\n          url: \"add?p=capricciosa\"\n          data-uid: 2\n","yaml",[104,1567,1568,1579,1586,1595,1602,1612,1623,1634,1642,1651,1660,1669,1676,1689,1698,1707,1717,1726,1734,1743,1751,1759,1765,1776,1785,1794,1804],{"__ignoreMap":154},[288,1569,1570,1573,1576],{"class":290,"line":291},[288,1571,1572],{"class":294},"- ",[288,1574,1575],{"class":298},"document",[288,1577,1578],{"class":294},":\n",[288,1580,1581,1584],{"class":290,"line":321},[288,1582,1583],{"class":298},"  children",[288,1585,1578],{"class":294},[288,1587,1588,1591,1593],{"class":290,"line":335},[288,1589,1590],{"class":294},"    - ",[288,1592,453],{"class":298},[288,1594,1578],{"class":294},[288,1596,1597,1600],{"class":290,"line":412},[288,1598,1599],{"class":298},"      children",[288,1601,1578],{"class":294},[288,1603,1604,1607,1610],{"class":290,"line":426},[288,1605,1606],{"class":294},"        - ",[288,1608,1609],{"class":298},"heading",[288,1611,1578],{"class":294},[288,1613,1614,1617,1620],{"class":290,"line":513},[288,1615,1616],{"class":298},"          name",[288,1618,1619],{"class":294},": ",[288,1621,1622],{"class":309},"\"Our Pizza\"\n",[288,1624,1625,1628,1630],{"class":290,"line":528},[288,1626,1627],{"class":298},"          level",[288,1629,1619],{"class":294},[288,1631,1633],{"class":1632},"s8ozJ","1\n",[288,1635,1636,1638,1640],{"class":290,"line":541},[288,1637,1606],{"class":294},[288,1639,1609],{"class":298},[288,1641,1578],{"class":294},[288,1643,1644,1646,1648],{"class":290,"line":554},[288,1645,1616],{"class":298},[288,1647,1619],{"class":294},[288,1649,1650],{"class":309},"\"Margherita\"\n",[288,1652,1653,1655,1657],{"class":290,"line":842},[288,1654,1627],{"class":298},[288,1656,1619],{"class":294},[288,1658,1659],{"class":1632},"2\n",[288,1661,1662,1664,1667],{"class":290,"line":852},[288,1663,1606],{"class":294},[288,1665,1666],{"class":298},"paragraph",[288,1668,1578],{"class":294},[288,1670,1671,1674],{"class":290,"line":873},[288,1672,1673],{"class":298},"          children",[288,1675,1578],{"class":294},[288,1677,1678,1681,1684,1686],{"class":290,"line":883},[288,1679,1680],{"class":294},"            - ",[288,1682,1683],{"class":298},"text",[288,1685,1619],{"class":294},[288,1687,1688],{"class":309},"\"A simple classic: mozzarella, tomatoes and basil. An everyday choice!\"\n",[288,1690,1691,1693,1696],{"class":290,"line":898},[288,1692,1606],{"class":294},[288,1694,1695],{"class":298},"link",[288,1697,1578],{"class":294},[288,1699,1700,1702,1704],{"class":290,"line":912},[288,1701,1616],{"class":298},[288,1703,1619],{"class":294},[288,1705,1706],{"class":309},"\"Add\"\n",[288,1708,1709,1712,1714],{"class":290,"line":921},[288,1710,1711],{"class":298},"          url",[288,1713,1619],{"class":294},[288,1715,1716],{"class":309},"\"add?p=margherita\"\n",[288,1718,1719,1722,1724],{"class":290,"line":927},[288,1720,1721],{"class":298},"          data-uid",[288,1723,1619],{"class":294},[288,1725,1633],{"class":1632},[288,1727,1728,1730,1732],{"class":290,"line":933},[288,1729,1606],{"class":294},[288,1731,1609],{"class":298},[288,1733,1578],{"class":294},[288,1735,1736,1738,1740],{"class":290,"line":942},[288,1737,1616],{"class":298},[288,1739,1619],{"class":294},[288,1741,1742],{"class":309},"\"Capricciosa\"\n",[288,1744,1745,1747,1749],{"class":290,"line":961},[288,1746,1627],{"class":298},[288,1748,1619],{"class":294},[288,1750,1659],{"class":1632},[288,1752,1753,1755,1757],{"class":290,"line":970},[288,1754,1606],{"class":294},[288,1756,1666],{"class":298},[288,1758,1578],{"class":294},[288,1760,1761,1763],{"class":290,"line":980},[288,1762,1673],{"class":298},[288,1764,1578],{"class":294},[288,1766,1767,1769,1771,1773],{"class":290,"line":990},[288,1768,1680],{"class":294},[288,1770,1683],{"class":298},[288,1772,1619],{"class":294},[288,1774,1775],{"class":309},"\"A rich taste: mozzarella, ham, mushrooms, artichokes and olives. A true favourite!\"\n",[288,1777,1779,1781,1783],{"class":290,"line":1778},24,[288,1780,1606],{"class":294},[288,1782,1695],{"class":298},[288,1784,1578],{"class":294},[288,1786,1788,1790,1792],{"class":290,"line":1787},25,[288,1789,1616],{"class":298},[288,1791,1619],{"class":294},[288,1793,1706],{"class":309},[288,1795,1797,1799,1801],{"class":290,"line":1796},26,[288,1798,1711],{"class":298},[288,1800,1619],{"class":294},[288,1802,1803],{"class":309},"\"add?p=capricciosa\"\n",[288,1805,1807,1809,1811],{"class":290,"line":1806},27,[288,1808,1721],{"class":298},[288,1810,1619],{"class":294},[288,1812,1659],{"class":1632},[11,1814,1815],{},[15,1816,1817],{},"Markdown Snapshot:",[280,1819,1823],{"className":1820,"code":1821,"language":1822,"meta":154,"style":154},"language-md shiki shiki-themes github-dark github-dark","# Our Pizza\n\n## Margherita\n\nA simple classic: mozzarella, tomatoes and basil.\nAn everyday choice!\n\n[Add](add?p=margherita)\n\n## Capricciosa\n\nA rich taste: mozzarella, ham, mushrooms, artichokes and olives.\nA true favourite!\n\n[Add](add?p=capricciosa)\n","md",[104,1824,1825,1830,1836,1841,1845,1850,1855,1859,1864,1868,1873,1877,1882,1887,1891],{"__ignoreMap":154},[288,1826,1827],{"class":290,"line":291},[288,1828,1829],{},"# Our Pizza\n",[288,1831,1832],{"class":290,"line":321},[288,1833,1835],{"emptyLinePlaceholder":1834},true,"\n",[288,1837,1838],{"class":290,"line":335},[288,1839,1840],{},"## Margherita\n",[288,1842,1843],{"class":290,"line":412},[288,1844,1835],{"emptyLinePlaceholder":1834},[288,1846,1847],{"class":290,"line":426},[288,1848,1849],{},"A simple classic: mozzarella, tomatoes and basil.\n",[288,1851,1852],{"class":290,"line":513},[288,1853,1854],{},"An everyday choice!\n",[288,1856,1857],{"class":290,"line":528},[288,1858,1835],{"emptyLinePlaceholder":1834},[288,1860,1861],{"class":290,"line":541},[288,1862,1863],{},"[Add](add?p=margherita)\n",[288,1865,1866],{"class":290,"line":554},[288,1867,1835],{"emptyLinePlaceholder":1834},[288,1869,1870],{"class":290,"line":842},[288,1871,1872],{},"## Capricciosa\n",[288,1874,1875],{"class":290,"line":852},[288,1876,1835],{"emptyLinePlaceholder":1834},[288,1878,1879],{"class":290,"line":873},[288,1880,1881],{},"A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[288,1883,1884],{"class":290,"line":883},[288,1885,1886],{},"A true favourite!\n",[288,1888,1889],{"class":290,"line":898},[288,1890,1835],{"emptyLinePlaceholder":1834},[288,1892,1893],{"class":290,"line":912},[288,1894,1895],{},"[Add](add?p=capricciosa)\n",[11,1897,1898],{},[15,1899,1900],{},"Custom Format Snapshot:",[280,1902,1906],{"className":1903,"code":1905,"language":1683},[1904],"language-text","[0] H1 \"Our Pizza\"\n[1] H2 \"Margherita\"\n[2] P \"A simple classic: mozzarella, tomatoes and basil.\\nAn everyday choice!\"\n[3] A \"Add\" \"add?p=margherita\"\n[4] H2 \"Capricciosa\"\n[5] P \"A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\\nA true...\"\n[6] A \"Add\" \"add?p=capricciosa\"\n",[104,1907,1905],{"__ignoreMap":154},[11,1909,1910,1911,18],{},"Browser Use agents implement hybrid snapshots: the grounded GUI snapshot is a supplement to a custom-format indexed-list DOM snapshot, similar to the above example. Evaluations show that the DOM snapshot representation generally conveys most of the salient UI features for the web agent LLM backend",[147,1912,1913],{},[23,1914,689],{"href":686,"ariaDescribedBy":1915,"dataFootnoteRef":154,"id":1916},[153],"user-content-fnref-4-3",[53,1918,1920],{"id":1919},"non-idiomatic-application-designs","Non-Idiomatic Application Designs",[11,1922,1923],{},"An actionable element might lack affordance – not communicating its actionability through its bare appearance. That is bad UI/UX design practice. The concept extends to the DOM through HTML semantics, which communicate the general purpose of an element to web browsers and developers rather than users. Generalising to web agent LLM backends, the concept of affordance aligns between the GUI and the DOM – the model is a UI representation-agnostic user.",[11,1925,1926],{},"Missing affordances should be rectified in a snapshot wherever they can be discovered. Discoverability is another circular problem: what is an indirect affordance for actionability? Ideally, the underlying actionability is tracked, such as by checking for attached action event listeners. However, there is no browser API for this practice.",[11,1928,1929],{},"As an example for bad affordance in the GUI, imagine an overly subtle 'Shop' button. Bounding boxes in grounded GUI snapshots are a workaround – provided the element is discoverable as actionable in the DOM in the first place.",[45,1931],{"src":1932,"alt":1933,"loading":76,":width":1934},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/9.png","Example of a fictional UI/UX bad practice (overly subtle button)","480",[11,1936,1937,1938,1943],{},"As an example for bad affordance in the DOM, look at this datepicker from ",[23,1939,1942],{"href":1940,"rel":1941},"https://www.webmd.com",[27],"webmd.com",":",[45,1945],{"src":1946,"alt":1947,"loading":76,":width":1934},"/blog/serialising-web-ui-state-for-llms-the-complete-guide/10.png","Example of an HTML bad practice (ignoring tag name semantics)",[280,1949,1951],{"className":282,"code":1950,"language":284,"meta":154,"style":154},"\u003Cdiv>\n  \u003Cspan class=\"cell day-header\">Sun\u003C/span>\n  \u003Cspan class=\"cell day-header\">Mon\u003C/span>\n  \u003Cspan class=\"cell day-header\">Tue\u003C/span>\n  \u003Cspan class=\"cell day-header\">Wed\u003C/span>\n  \u003Cspan class=\"cell day-header\">Thu\u003C/span>\n  \u003Cspan class=\"cell day-header\">Fri\u003C/span>\n  \u003Cspan class=\"cell day-header\">Sat\u003C/span>\n  \u003C!-- [...] -->\n  \u003Cspan class=\"cell day blank\">\u003C/span>\n  \u003Cspan class=\"cell day blank\">\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>1\u003C/span>\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>2\u003C/span>\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>3\u003C/span>\u003C/span>\n  \u003C!-- [...] -->\n  \u003Cspan class=\"cell day\">\u003Cspan>29\u003C/span>\u003C/span>\n  \u003Cspan class=\"cell day\">\u003Cspan>30\u003C/span>\u003C/span>\n\u003C/div>\n",[104,1952,1953,1961,1981,2000,2019,2038,2057,2076,2095,2101,2121,2139,2168,2195,2222,2226,2253,2280],{"__ignoreMap":154},[288,1954,1955,1957,1959],{"class":290,"line":291},[288,1956,295],{"class":294},[288,1958,744],{"class":298},[288,1960,318],{"class":294},[288,1962,1963,1965,1967,1969,1971,1974,1977,1979],{"class":290,"line":321},[288,1964,468],{"class":294},[288,1966,288],{"class":298},[288,1968,456],{"class":302},[288,1970,306],{"class":294},[288,1972,1973],{"class":309},"\"cell day-header\"",[288,1975,1976],{"class":294},">Sun\u003C/",[288,1978,288],{"class":298},[288,1980,318],{"class":294},[288,1982,1983,1985,1987,1989,1991,1993,1996,1998],{"class":290,"line":335},[288,1984,468],{"class":294},[288,1986,288],{"class":298},[288,1988,456],{"class":302},[288,1990,306],{"class":294},[288,1992,1973],{"class":309},[288,1994,1995],{"class":294},">Mon\u003C/",[288,1997,288],{"class":298},[288,1999,318],{"class":294},[288,2001,2002,2004,2006,2008,2010,2012,2015,2017],{"class":290,"line":412},[288,2003,468],{"class":294},[288,2005,288],{"class":298},[288,2007,456],{"class":302},[288,2009,306],{"class":294},[288,2011,1973],{"class":309},[288,2013,2014],{"class":294},">Tue\u003C/",[288,2016,288],{"class":298},[288,2018,318],{"class":294},[288,2020,2021,2023,2025,2027,2029,2031,2034,2036],{"class":290,"line":426},[288,2022,468],{"class":294},[288,2024,288],{"class":298},[288,2026,456],{"class":302},[288,2028,306],{"class":294},[288,2030,1973],{"class":309},[288,2032,2033],{"class":294},">Wed\u003C/",[288,2035,288],{"class":298},[288,2037,318],{"class":294},[288,2039,2040,2042,2044,2046,2048,2050,2053,2055],{"class":290,"line":513},[288,2041,468],{"class":294},[288,2043,288],{"class":298},[288,2045,456],{"class":302},[288,2047,306],{"class":294},[288,2049,1973],{"class":309},[288,2051,2052],{"class":294},">Thu\u003C/",[288,2054,288],{"class":298},[288,2056,318],{"class":294},[288,2058,2059,2061,2063,2065,2067,2069,2072,2074],{"class":290,"line":528},[288,2060,468],{"class":294},[288,2062,288],{"class":298},[288,2064,456],{"class":302},[288,2066,306],{"class":294},[288,2068,1973],{"class":309},[288,2070,2071],{"class":294},">Fri\u003C/",[288,2073,288],{"class":298},[288,2075,318],{"class":294},[288,2077,2078,2080,2082,2084,2086,2088,2091,2093],{"class":290,"line":541},[288,2079,468],{"class":294},[288,2081,288],{"class":298},[288,2083,456],{"class":302},[288,2085,306],{"class":294},[288,2087,1973],{"class":309},[288,2089,2090],{"class":294},">Sat\u003C/",[288,2092,288],{"class":298},[288,2094,318],{"class":294},[288,2096,2097],{"class":290,"line":554},[288,2098,2100],{"class":2099},"sJ8bj","  \u003C!-- [...] -->\n",[288,2102,2103,2105,2107,2109,2111,2114,2117,2119],{"class":290,"line":842},[288,2104,468],{"class":294},[288,2106,288],{"class":298},[288,2108,456],{"class":302},[288,2110,306],{"class":294},[288,2112,2113],{"class":309},"\"cell day blank\"",[288,2115,2116],{"class":294},">\u003C/",[288,2118,288],{"class":298},[288,2120,318],{"class":294},[288,2122,2123,2125,2127,2129,2131,2133,2135,2137],{"class":290,"line":852},[288,2124,468],{"class":294},[288,2126,288],{"class":298},[288,2128,456],{"class":302},[288,2130,306],{"class":294},[288,2132,2113],{"class":309},[288,2134,2116],{"class":294},[288,2136,288],{"class":298},[288,2138,318],{"class":294},[288,2140,2141,2143,2145,2147,2149,2152,2155,2157,2160,2162,2164,2166],{"class":290,"line":873},[288,2142,468],{"class":294},[288,2144,288],{"class":298},[288,2146,456],{"class":302},[288,2148,306],{"class":294},[288,2150,2151],{"class":309},"\"cell day\"",[288,2153,2154],{"class":294},">\u003C",[288,2156,288],{"class":298},[288,2158,2159],{"class":294},">1\u003C/",[288,2161,288],{"class":298},[288,2163,2116],{"class":294},[288,2165,288],{"class":298},[288,2167,318],{"class":294},[288,2169,2170,2172,2174,2176,2178,2180,2182,2184,2187,2189,2191,2193],{"class":290,"line":883},[288,2171,468],{"class":294},[288,2173,288],{"class":298},[288,2175,456],{"class":302},[288,2177,306],{"class":294},[288,2179,2151],{"class":309},[288,2181,2154],{"class":294},[288,2183,288],{"class":298},[288,2185,2186],{"class":294},">2\u003C/",[288,2188,288],{"class":298},[288,2190,2116],{"class":294},[288,2192,288],{"class":298},[288,2194,318],{"class":294},[288,2196,2197,2199,2201,2203,2205,2207,2209,2211,2214,2216,2218,2220],{"class":290,"line":898},[288,2198,468],{"class":294},[288,2200,288],{"class":298},[288,2202,456],{"class":302},[288,2204,306],{"class":294},[288,2206,2151],{"class":309},[288,2208,2154],{"class":294},[288,2210,288],{"class":298},[288,2212,2213],{"class":294},">3\u003C/",[288,2215,288],{"class":298},[288,2217,2116],{"class":294},[288,2219,288],{"class":298},[288,2221,318],{"class":294},[288,2223,2224],{"class":290,"line":912},[288,2225,2100],{"class":2099},[288,2227,2228,2230,2232,2234,2236,2238,2240,2242,2245,2247,2249,2251],{"class":290,"line":921},[288,2229,468],{"class":294},[288,2231,288],{"class":298},[288,2233,456],{"class":302},[288,2235,306],{"class":294},[288,2237,2151],{"class":309},[288,2239,2154],{"class":294},[288,2241,288],{"class":298},[288,2243,2244],{"class":294},">29\u003C/",[288,2246,288],{"class":298},[288,2248,2116],{"class":294},[288,2250,288],{"class":298},[288,2252,318],{"class":294},[288,2254,2255,2257,2259,2261,2263,2265,2267,2269,2272,2274,2276,2278],{"class":290,"line":927},[288,2256,468],{"class":294},[288,2258,288],{"class":298},[288,2260,456],{"class":302},[288,2262,306],{"class":294},[288,2264,2151],{"class":309},[288,2266,2154],{"class":294},[288,2268,288],{"class":298},[288,2270,2271],{"class":294},">30\u003C/",[288,2273,288],{"class":298},[288,2275,2116],{"class":294},[288,2277,288],{"class":298},[288,2279,318],{"class":294},[288,2281,2282,2284,2286],{"class":290,"line":933},[288,2283,506],{"class":294},[288,2285,744],{"class":298},[288,2287,318],{"class":294},[11,2289,2290,2291,2293,2294,2297,2298,2300,2301,2304,2305,2307],{},"The days wrapped in ",[104,2292,288],{}," (e.g., ",[104,2295,2296],{},"16",") are clickable, but the HTML communicates no such actionability. The idiomatic DOM equivalent would, throughout, instead use ",[104,2299,299],{}," elements, or at least assign a suitable ",[104,2302,2303],{},"role"," attributes like ",[104,2306,299],{},". A DOM snapshot pipeline leaves space for solving this problem: rewrite the virtual DOM with idiomatic elements.",[280,2309,2311],{"className":282,"code":2310,"language":284,"meta":154,"style":154},"  \u003Cbutton class=\"cell day\">29\u003C/button>\n  \u003C!-- OR -->\n  \u003Cspan role=\"button\" class=\"cell day\">30\u003C/button>\n",[104,2312,2313,2331,2336],{"__ignoreMap":154},[288,2314,2315,2317,2319,2321,2323,2325,2327,2329],{"class":290,"line":291},[288,2316,468],{"class":294},[288,2318,299],{"class":298},[288,2320,456],{"class":302},[288,2322,306],{"class":294},[288,2324,2151],{"class":309},[288,2326,2244],{"class":294},[288,2328,299],{"class":298},[288,2330,318],{"class":294},[288,2332,2333],{"class":290,"line":321},[288,2334,2335],{"class":2099},"  \u003C!-- OR -->\n",[288,2337,2338,2340,2342,2345,2347,2349,2351,2353,2355,2357,2359],{"class":290,"line":335},[288,2339,468],{"class":294},[288,2341,288],{"class":298},[288,2343,2344],{"class":302}," role",[288,2346,306],{"class":294},[288,2348,863],{"class":309},[288,2350,456],{"class":302},[288,2352,306],{"class":294},[288,2354,2151],{"class":309},[288,2356,2271],{"class":294},[288,2358,299],{"class":298},[288,2360,318],{"class":294},[53,2362,2364],{"id":2363},"choosing-between-snapshot-representations","Choosing Between Snapshot Representations",[11,2366,2367],{},"No snapshot representation fits all purposes; the two base representations alone – GUI and DOM (HTML) – come with markedly different advantages and limitations. A proper snapshot tool should therefore route and cascade:",[1203,2369,2370,2382,2388,2394],{},[1206,2371,2372,2375,2376,2378,2379,2381],{},[15,2373,2374],{},"Route Modality."," Fall back to GUI snapshots if the UI is based on ",[104,2377,205],{}," or cross-origin ",[104,2380,201],{}," elements; use DOM snapshots by default.",[1206,2383,2384,2387],{},[15,2385,2386],{},"Route Representation."," Categorise task and translate DOM snapshot to most suitable representation. Categorising the task can be done with increasing levels of machine learning – simple keyword matching, or deploying a language model.",[1206,2389,2390,2393],{},[15,2391,2392],{},"Scope Escalation."," If elicited action suggestions are unusable, widen the snapshot scope and representation richness.",[1206,2395,2396,2399],{},[15,2397,2398],{},"Cascade to Hybrid."," If elicited action suggestions are still unusable, augment the rich DOM snapshot with a GUI snapshot for maximum grounding.",[78,2401,2403],{"id":2402},"gui-snapshots-vs-dom-snapshots","GUI Snapshots vs DOM Snapshots",[11,2405,2406],{},"Bold marks the more favourable characteristic.",[2408,2409,2410,2432],"table",{},[2411,2412,2413],"thead",{},[2414,2415,2416,2420,2423,2426,2429],"tr",{},[2417,2418],"th",{"align":2419},"left",[2417,2421,2422],{},"GUI",[2417,2424,2425],{},"grounded",[2417,2427,2428],{},"DOM",[2417,2430,2431],{},"transformed",[2433,2434,2435,2458,2479,2502,2521,2542],"tbody",{},[2414,2436,2437,2441,2446,2450,2453],{},[2438,2439,2440],"td",{"align":2419},"LLM Input Tokens",[2438,2442,2443],{},[15,2444,2445],{},"Low",[2438,2447,2448],{},[15,2449,2445],{},[2438,2451,2452],{},"High",[2438,2454,2455,2457],{},[15,2456,2445],{}," to Moderate",[2414,2459,2460,2463,2466,2471,2475],{},[2438,2461,2462],{"align":2419},"Element Targeting",[2438,2464,2465],{},"Coordinates",[2438,2467,2468],{},[15,2469,2470],{},"Relative",[2438,2472,2473],{},[15,2474,2470],{},[2438,2476,2477],{},[15,2478,2470],{},[2414,2480,2481,2484,2490,2494,2498],{},[2438,2482,2483],{"align":2419},"Snapshot Scope",[2438,2485,2486,2489],{},[15,2487,2488],{},"Free","/Viewport",[2438,2491,2492,2489],{},[15,2493,2488],{},[2438,2495,2496],{},[15,2497,2488],{},[2438,2499,2500],{},[15,2501,2488],{},[2414,2503,2504,2507,2510,2512,2517],{},[2438,2505,2506],{"align":2419},"Snapshot Context",[2438,2508,2509],{},"UI",[2438,2511,2509],{},[2438,2513,2514],{},[15,2515,2516],{},"Full",[2438,2518,2519],{},[15,2520,2516],{},[2414,2522,2523,2528,2533,2537,2540],{},[2438,2524,2525,2527],{"align":2419},[104,2526,201],{}," Context",[2438,2529,2530],{},[15,2531,2532],{},"Viewport",[2438,2534,2535],{},[15,2536,2532],{},[2438,2538,2539],{},"Same-origin only",[2438,2541,2539],{},[2414,2543,2544,2548,2553,2557,2560],{},[2438,2545,2546,2527],{"align":2419},[104,2547,205],{},[2438,2549,2550],{},[15,2551,2552],{},"Yes",[2438,2554,2555],{},[15,2556,2552],{},[2438,2558,2559],{},"No",[2438,2561,2559],{},[53,2563,2565],{"id":2564},"using-snapshots-in-the-model-context","Using Snapshots in the Model Context",[11,2567,2568],{},"Creating a snapshot is one thing. Using it as context to a model prompt is the other. Usually, conversational AI maintains a full trace of history as model context. Since a snapshot is time-sensitive, however, only the latest snapshot can be targeted with actions by the model.",[11,2570,2571],{},"Whether to keep a trace of previous actions and model reasoning is a question in its own right. If web browsing is modelled as a history-independent task, the snapshot-to-actions agent loop could even be separated from the persistent conversational loop, so it becomes a one-shot model prompt.",[53,2573,2575],{"id":2574},"timing-snapshots-in-the-web-client","Timing Snapshots in the Web Client",[11,2577,2578,2579,18],{},"LLM-based web agents are much slower than humans in taking and processing snapshots. Not least for that reason, snapshot rounds should be kept to a minimum and timed so that a snapshot is taken as soon as the UI has reached a stable state. That implies a non-trivial challenge: defining what qualifies as a UI change, and at what intensity such a change counts as a state shift – beyond hard page transitions, e.g., in a single-page application",[147,2580,2581],{},[23,2582,2586],{"href":2583,"ariaDescribedBy":2584,"dataFootnoteRef":154,"id":2585},"#user-content-fn-5",[153],"user-content-fnref-5","5",[2588,2589],"hr",{},[11,2591,2592,2593,18],{},"The snapshot is the central artefact in a web agent harness. Its representation is among the most significant factors for agent success",[147,2594,2595],{},[23,2596,2600],{"href":2597,"ariaDescribedBy":2598,"dataFootnoteRef":154,"id":2599},"#user-content-fn-6",[153],"user-content-fnref-6","6",[11,2602,2603,2604,2606],{},"At Webfuse, we develop a platform for agentic browsing. A focus of our research is on efficient snapshot representations. The Automation API can be used to create next-level snapshots: timed, rectified, across ",[104,2605,201],{}," boundaries, etc.",[453,2608,2611,2616],{"className":2609,"dataFootnotes":154},[2610],"footnotes",[53,2612,2615],{"className":2613,"id":153},[2614],"sr-only","Footnotes",[1203,2617,2618,2633,2646,2658,2684,2696],{},[1206,2619,2621,2625,2626],{"id":2620},"user-content-fn-1",[23,2622,2623],{"href":2623,"rel":2624},"https://arxiv.org/abs/2310.11441",[27]," ",[23,2627,2632],{"href":2628,"ariaLabel":2629,"className":2630,"dataFootnoteBackref":154},"#user-content-fnref-1","Back to reference 1",[2631],"data-footnote-backref","↩",[1206,2634,2636,2637,2625,2641],{"id":2635},"user-content-fn-2","Our reimplementation of Browser Use's grounded GUI snapshots: ",[23,2638,2639],{"href":2639,"rel":2640},"https://github.com/webfuse-com/D2Snap/blob/main/snapshots/_bu.js",[27],[23,2642,2632],{"href":2643,"ariaLabel":2644,"className":2645,"dataFootnoteBackref":154},"#user-content-fnref-2","Back to reference 2",[2631],[1206,2647,2649,2625,2653],{"id":2648},"user-content-fn-3",[23,2650,2651],{"href":2651,"rel":2652},"https://arxiv.org/abs/2210.03945",[27],[23,2654,2632],{"href":2655,"ariaLabel":2656,"className":2657,"dataFootnoteBackref":154},"#user-content-fnref-3","Back to reference 3",[2631],[1206,2659,2661,2625,2665,2625,2670,2625,2677],{"id":2660},"user-content-fn-4",[23,2662,2663],{"href":2663,"rel":2664},"https://arxiv.org/abs/2508.04412",[27],[23,2666,2632],{"href":2667,"ariaLabel":2668,"className":2669,"dataFootnoteBackref":154},"#user-content-fnref-4","Back to reference 4",[2631],[23,2671,2632,2675],{"href":2672,"ariaLabel":2673,"className":2674,"dataFootnoteBackref":154},"#user-content-fnref-4-2","Back to reference 4-2",[2631],[147,2676,179],{},[23,2678,2632,2682],{"href":2679,"ariaLabel":2680,"className":2681,"dataFootnoteBackref":154},"#user-content-fnref-4-3","Back to reference 4-3",[2631],[147,2683,194],{},[1206,2685,2687,2625,2691],{"id":2686},"user-content-fn-5",[23,2688,2689],{"href":2689,"rel":2690},"https://github.com/webfuse-com/DOMQuake",[27],[23,2692,2632],{"href":2693,"ariaLabel":2694,"className":2695,"dataFootnoteBackref":154},"#user-content-fnref-5","Back to reference 5",[2631],[1206,2697,2699,2625,2703],{"id":2698},"user-content-fn-6",[23,2700,2701],{"href":2701,"rel":2702},"https://arxiv.org/abs/2504.01382",[27],[23,2704,2632],{"href":2705,"ariaLabel":2706,"className":2707,"dataFootnoteBackref":154},"#user-content-fnref-6","Back to reference 6",[2631],[2709,2710,2711],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .suv1-, html code.shiki .suv1-{--shiki-default:#E1E4E8;--shiki-dark:#E1E4E8}html pre.shiki code .sxg3X, html code.shiki .sxg3X{--shiki-default:#85E89D;--shiki-dark:#85E89D}html pre.shiki code .sFR8T, html code.shiki .sFR8T{--shiki-default:#B392F0;--shiki-dark:#B392F0}html pre.shiki code .s4wv1, html code.shiki .s4wv1{--shiki-default:#9ECBFF;--shiki-dark:#9ECBFF}html pre.shiki code .s8ozJ, html code.shiki .s8ozJ{--shiki-default:#79B8FF;--shiki-dark:#79B8FF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}",{"title":154,"searchDepth":321,"depth":321,"links":2713},[2714,2718,2724,2725,2728,2729,2730],{"id":55,"depth":321,"text":56,"children":2715},[2716,2717],{"id":80,"depth":335,"text":81},{"id":119,"depth":335,"text":120},{"id":182,"depth":321,"text":183,"children":2719},[2720,2721,2722,2723],{"id":212,"depth":335,"text":213},{"id":237,"depth":335,"text":238},{"id":1167,"depth":335,"text":1168},{"id":1191,"depth":335,"text":1192},{"id":1919,"depth":321,"text":1920},{"id":2363,"depth":321,"text":2364,"children":2726},[2727],{"id":2402,"depth":335,"text":2403},{"id":2564,"depth":321,"text":2565},{"id":2574,"depth":321,"text":2575},{"id":153,"depth":321,"text":2615},"ai-agents","2025-08-28","Understand how web-based user interfaces can be serialised for use with LLM-based web agents.",null,{"homepage":1834,"relatedLinks":2736},[2737,2740],{"text":2738,"href":2739},"DOM Downsampling for LLM-Based Web Agents","/rnd/dom-downsampling-for-llm-based-web-agents",{"text":2741,"href":2742,"external":1834},"Webfuse Automation API","https://dev.webfuse.com/automation-api/","/blog/serialising-web-ui-state-for-llms-the-complete-guide",{"title":5,"description":2733},{"loc":2743},"blog/1013.serialising-web-ui-state-for-llms-the-complete-guide",[2731,2748,2749,2750,2751],"web-agents","web-automation","llms","snapshots","AeGP0KPR9tR7LnBSgsIeXvjdntE_vgTIeI7xmMt52f8",[2754,3334],{"id":2755,"title":2756,"authorId":2757,"body":2758,"category":3307,"created":3308,"description":3309,"extension":1822,"faqs":3310,"featurePriority":2734,"head":2734,"hideCta":3323,"landingPath":3324,"meta":3325,"navigation":1834,"ogImage":2734,"path":3326,"robots":2734,"schemaOrg":2734,"seo":3327,"sitemap":3328,"stem":3329,"tags":3330,"__hash__":3333},"blog/blog/1018.top-5-voice-ai-agents-for-website-integration-in-2026.md","Top 5 Voice AI Agents for Website Integration in 2026","salome-koshadze",{"type":8,"value":2759,"toc":3281},[2760,2809,2812,2825,2831,2834,2837,2843,2850,2857,2863,2866,2869,2889,2895,2898,2901,2907,2910,2916,2921,2930,2936,2939,2942,2968,2974,2977,2980,2983,2989,2995,3003,3009,3012,3030,3033,3039,3042,3062,3068,3071,3074,3080,3085,3094,3100,3103,3117,3123,3126,3146,3152,3159,3162,3165,3171,3176,3183,3189,3192,3212,3218,3225,3228,3231,3234,3240,3243,3246,3278],[2761,2762,2763,2774,2806],"tldr-box",{},[11,2764,2765,2766,2769,2770,2773],{},"Voice AI is evolving from basic chatbots to agentic systems that can execute tasks directly on websites. The AI agent market is projected to reach ",[15,2767,2768],{},"$50.31 billion by 2030",", with ",[15,2771,2772],{},"40% of enterprise applications"," expected to use task-specific agents by 2026. This guide compares the top 5 platforms for 2026:",[1242,2775,2776,2782,2788,2794,2800],{},[1206,2777,2778,2781],{},[15,2779,2780],{},"ElevenLabs"," - Best for realistic, emotionally expressive voices with 400+ integrations",[1206,2783,2784,2787],{},[15,2785,2786],{},"Deepgram"," - Optimized for speed with \u003C250ms latency and unified API",[1206,2789,2790,2793],{},[15,2791,2792],{},"Vapi"," - Maximum flexibility for developers to mix and match AI models",[1206,2795,2796,2799],{},[15,2797,2798],{},"Google Dialogflow"," - Enterprise-grade solution integrated with Google Cloud",[1206,2801,2802,2805],{},[15,2803,2804],{},"Voiceflow"," - Visual, collaborative platform for team-based agent design",[11,2807,2808],{},"Each platform offers unique strengths depending on your priorities: voice quality, speed, flexibility, enterprise scale, or team collaboration.",[11,2810,2811],{},"For years, websites have been silent partners in our digital tasks. We click, we type, and they respond in a predictable, structured manner. That one-way interaction is undergoing a major redesign, shifting towards a collaborative, conversational model. By 2026, the use of voice AI agents that you can talk to and direct on a website is projected to become a widespread feature for businesses aiming to offer more intuitive and efficient user experiences.",[11,2813,2814,2815,2818,2819,2824],{},"This new wave of technology moves past simple chatbots. We are now looking at the integration of ",[37,2816,2817],{},"agentic"," voice AI. This means the AI can perform tasks and execute actions on the user's behalf directly on the webpage. Imagine telling a website, \"Book me a flight to New York for next Tuesday, and find a hotel near Central Park,\" and watch it happen without needing to navigate menus or fill out forms. This capability is rapidly becoming a reality. The global market for AI agents was valued at USD 5.40 billion in 2024 and is ",[23,2820,2823],{"href":2821,"rel":2822},"https://www.grandviewresearch.com/industry-analysis/ai-agents-market-report",[27],"projected to reach USD 50.31 billion"," by 2030.",[2826,2827],"article-signup-cta",{"eyebrow":2828,"heading":2829,"highlight":2830},"A voice agent that cannot see the page can only talk.","Webfuse gives it the live session.","live",[11,2832,2833],{},"This shift results in benefits such as increased speed and convenience for users, along with higher engagement and new avenues for customer support for businesses. The technology making this possible has seen considerable advancements. Lower latency in speech recognition and text-to-speech, paired with highly capable Large Language Models (LLMs), allows for real-time, human-like conversations. By 2026, it is anticipated that 40% of enterprise applications will use task-specific AI agents.",[11,2835,2836],{},"So, how do we get past basic chatbots and build these highly capable voice agents for website integration? Several platforms provide the tools to make this happen. Let's look at some of these major players expected to lead the way in 2026:",[53,2838,2840],{"id":2839},"elevenlabs-the-intersection-of-lifelike-voice-and-agentic-action",[15,2841,2842],{},"ElevenLabs: The Intersection of Lifelike Voice and Agentic Action",[45,2844],{":height":2845,":width":2846,"alt":2847,"format":2848,"loading":76,"src":2849},"1037","1999","Elevenlabs screenshot","webp","/blog/top-5-voice-ai-agents-for-website-integration-in-2026/1.png",[11,2851,2852,2856],{},[23,2853,2780],{"href":2854,"rel":2855},"https://elevenlabs.io/",[27]," has built a major reputation on one thing: generating some of the most realistic and emotionally nuanced AI voices available. But the platform is expanding beyond high-quality audio. It now provides a complete conversational AI platform designed to create voice agents that can be integrated directly into websites. This positions it as a key player for 2026, offering a unique combination of highly expressive voice synthesis and the backend intelligence to perform tasks.",[78,2858,2860],{"id":2859},"from-voice-synthesis-to-on-site-action",[15,2861,2862],{},"From Voice Synthesis to On-Site Action",[11,2864,2865],{},"The core strength of ElevenLabs lies in its ability to produce audio that is nearly indistinguishable from human speech. Users have a high degree of control over voice attributes, including pitch, speed, and emotional expression, across a library of over 1200 voices in more than 29 languages.",[11,2867,2868],{},"What  sets the platform apart are its agentic capabilities, which make it suitable for modern website integration. Let's look at some of these:",[1242,2870,2871,2877,2883],{},[1206,2872,2873,2876],{},[15,2874,2875],{},"Real-Time, Low-Latency API:"," For a voice conversation to feel natural, the response must be immediate. ElevenLabs has optimized its system for low latency, with its streaming APIs capable of delivering audio in under 100 milliseconds. This is fast enough to support real-time, interactive conversations without awkward delays.",[1206,2878,2879,2882],{},[15,2880,2881],{},"A Massive Integration Library:"," An agent is only as useful as the actions it can perform. ElevenLabs provides over 400 pre-configured integrations with a wide range of external systems. This allows an agent on your website to connect directly to CRMs like Salesforce, scheduling tools like Calendly, and communication platforms like Slack to execute tasks mid-conversation. For example, a user could ask the agent to book a meeting, and the agent could access calendar availability and confirm the appointment without the user ever leaving the page.",[1206,2884,2885,2888],{},[15,2886,2887],{},"Custom Knowledge and Intelligence:"," You can ground the agent in your specific data by uploading documents or connecting it to your website's content. Using Retrieval-Augmented Generation (RAG), the agent can pull from these sources to provide accurate, up-to-date answers, acting as an expert on your products or services. You can also connect your own Large Language Model (LLM), such as models from Google or OpenAI, to tailor its reasoning capabilities.",[78,2890,2892],{"id":2891},"simple-deployment-on-your-website",[15,2893,2894],{},"Simple Deployment on Your Website",[11,2896,2897],{},"Getting an ElevenLabs agent onto a website is a direct process. The platform provides a code snippet that can be embedded into your site's HTML. For popular platforms like WordPress or Webflow, this is as simple as adding a custom HTML block or an embed element. This accessibility means that a fully functional, voice-driven agent can be deployed in minutes rather than months.",[11,2899,2900],{},"The agent appears as a widget on the page, which users can interact with through voice or text. From a single dashboard, you can configure the agent's personality, set its first message, and monitor conversation transcripts to see how it's performing.",[78,2902,2904],{"id":2903},"a-usage-based-model",[15,2905,2906],{},"A Usage-Based Model",[11,2908,2909],{},"ElevenLabs operates on a usage-based pricing model, typically measured in characters or credits. This structure includes a free tier, allowing for experimentation and small-scale projects. Paid plans scale up based on the volume of characters generated and offer access to more advanced features like professional voice cloning and higher-quality audio outputs. This approach allows businesses to start small and scale their usage as the value of the voice agent is proven.",[53,2911,2913],{"id":2912},"deepgram-engineered-for-conversational-speed",[15,2914,2915],{},"Deepgram: Engineered for Conversational Speed",[45,2917],{":height":2918,":width":2846,"alt":2919,"format":2848,"loading":76,"src":2920},"1049","Deepgram screenshot","/blog/top-5-voice-ai-agents-for-website-integration-in-2026/2.png",[11,2922,2923,2924,2929],{},"While ElevenLabs puts the quality of the voice at the forefront, ",[23,2925,2928],{"href":2926,"rel":2927},"https://deepgram.com/",[27],"Deepgram's"," major strength is its foundation in speed and accuracy. Originally known for its highly performant speech-to-text (STT) services, Deepgram has expanded its offerings to provide an end-to-end platform for building real-time voice AI agents. For website integration where responsiveness is a major factor, Deepgram presents a highly compelling option.",[78,2931,2933],{"id":2932},"the-need-for-speed-in-voice-ai",[15,2934,2935],{},"The Need for Speed in Voice AI",[11,2937,2938],{},"For a voice agent on a website to feel interactive and not clunky, the time between a user speaking and the agent responding must be minimal. Any noticeable delay breaks the illusion of a natural conversation. This is where Deepgram directs its focus. The company has engineered its entire system to minimize latency, reporting response times of under 250 milliseconds. This speed creates a conversational flow that feels immediate and human-like.",[11,2940,2941],{},"This is achieved by building a complete, in-house technology stack. Instead of relying on a chain of different services for transcription, language processing, and voice synthesis, Deepgram handles it all. Let's break down what makes it a strong contender for agentic website integration.",[1242,2943,2944,2950,2956,2962],{},[1206,2945,2946,2949],{},[15,2947,2948],{},"A Unified API for Voice:"," Deepgram provides a single, unified API that manages the entire conversational loop. This includes industry-leading speech-to-text, access to language models for intelligence, and their own text-to-speech engine, Aura. This simplifies the development process, as developers do not need to piece together multiple services.",[1206,2951,2952,2955],{},[15,2953,2954],{},"Highly Accurate Transcription:"," The accuracy of the agent's understanding begins with the transcription. Deepgram's models are known for their high accuracy across a wide range of accents and dialects. The platform also includes features like smart formatting and punctuation to make the transcribed text more reliable for the language model to interpret.",[1206,2957,2958,2961],{},[15,2959,2960],{},"Developer-Centric Tools:"," Deepgram is built with developers in mind. It offers Software Development Kits (SDKs) for popular programming languages like Python and Node.js. This makes it easier to integrate Deepgram's voice capabilities into a custom front-end application on a website, giving developers full control over the user interface and experience.",[1206,2963,2964,2967],{},[15,2965,2966],{},"Conversational Intelligence Features:"," Beyond just transcription, Deepgram can provide deeper insights into the conversation. It can detect sentiment, identify topics being discussed, and even summarize conversations. For a website agent, this information can be used to route a user to the correct department or to understand customer satisfaction in real time.",[78,2969,2971],{"id":2970},"building-a-custom-experience",[15,2972,2973],{},"Building a Custom Experience",[11,2975,2976],{},"Unlike platforms that offer a pre-built widget, integrating Deepgram into a website typically involves a more custom development approach. Developers use Deepgram's APIs and SDKs to build a unique voice interface tailored to their specific needs. This offers a high degree of flexibility in how the agent looks, feels, and operates.",[11,2978,2979],{},"For example, a developer could build an interactive product guide where a user can ask questions about different features shown on the screen. The website's front end would capture the user's audio, send it to Deepgram's API, and then receive both the transcribed text and the synthesized audio response to play back to the user. Because the API can also connect to other tools, the agent could then take actions like adding a product to the cart or scheduling a demo.",[11,2981,2982],{},"Deepgram's pricing is consumption-based, billed per second of audio processed. This model allows for scalability, with costs directly tied to the amount of usage the voice agent receives.",[53,2984,2986],{"id":2985},"vapi-the-developers-toolkit-for-composable-voice-ai",[15,2987,2988],{},"Vapi: The Developer's Toolkit for Composable Voice AI",[45,2990],{":height":2991,":width":2992,"alt":2993,"format":2848,"loading":76,"src":2994},"1938","3840","Vapi screenshot","/blog/top-5-voice-ai-agents-for-website-integration-in-2026/3.png",[11,2996,2997,2998,3002],{},"Where other platforms provide an all-in-one system, ",[23,2999,2792],{"href":3000,"rel":3001},"https://vapi.ai/",[27]," positions itself differently. It is a highly configurable platform built specifically for developers who want to construct their own voice AI agents by combining best-in-class technologies.Instead of offering its own custom models for every step, Vapi acts as a coordination layer, handling the complex infrastructure required to make different services for speech-to-text, language processing, and text-to-speech work together in real-time.",[78,3004,3006],{"id":3005},"a-focus-on-coordination-not-creation",[15,3007,3008],{},"A Focus on Coordination, Not Creation",[11,3010,3011],{},"The core philosophy behind Vapi is flexibility. Developers are not locked into a single ecosystem. They can choose the components that best fit their needs. For instance, a developer could build a voice agent that uses:",[1242,3013,3014,3019,3025],{},[1206,3015,3016,3018],{},[15,3017,2786],{}," for its fast and accurate speech-to-text.",[1206,3020,3021,3024],{},[15,3022,3023],{},"OpenAI's GPT-4o or Anthropic's Claude 3"," for advanced reasoning and intelligence.",[1206,3026,3027,3029],{},[15,3028,2780],{}," for its highly realistic and expressive voice output.",[11,3031,3032],{},"Vapi's platform manages the complex flow of data between these services, ensuring the conversation happens with very low latency. This \"bring your own models\" approach is ideal for teams that want to fine-tune every aspect of their agent's performance and personality.",[78,3034,3036],{"id":3035},"key-features-for-building-capable-agents",[15,3037,3038],{},"Key Features for Building Capable Agents",[11,3040,3041],{},"Vapi's developer-first focus is evident in its feature set, which is geared towards creating highly functional and intelligent voice agents for websites.",[1242,3043,3044,3050,3056],{},[1206,3045,3046,3049],{},[15,3047,3048],{},"Powerful Tool Calling:"," This is a major feature for creating true agentic behavior. Tool calling allows the AI assistant to connect to and use external APIs during a conversation. For example, a voice agent on an e-commerce site could use a tool to check inventory levels, process a payment through Stripe, or create a shipping label by calling the shipping provider's API-all based on a user's spoken request.",[1206,3051,3052,3055],{},[15,3053,3054],{},"Simplified Real-Time Infrastructure:"," Handling real-time voice communication over the web can be complex. Vapi abstracts this away by managing WebSocket connections and the streaming of audio data. This frees up developers to focus on the agent's logic and capabilities rather than the underlying plumbing.",[1206,3057,3058,3061],{},[15,3059,3060],{},"Web Integration via SDK and Widget:"," Vapi offers multiple ways to get an agent onto a website. For maximum control, developers can use the JavaScript SDK to build a completely custom voice interface. For quicker deployment, Vapi also provides an embeddable web widget that can be added to a site with a single line of code, offering a floating chat interface that supports both voice and text.",[78,3063,3065],{"id":3064},"built-for-custom-workflows",[15,3066,3067],{},"Built for Custom Workflows",[11,3069,3070],{},"Integrating Vapi into a website is a process designed for technical teams. The platform is API-native, meaning every feature is exposed through an API for extensive configuration. Developers can define their assistant's parameters, set up custom prompts, and connect to their own back-end systems to pull data or trigger actions. This makes it possible to create highly bespoke voice experiences that are deeply integrated with a website's existing functionality.",[11,3072,3073],{},"Vapi's pricing is usage-based, typically charging a small fee per minute for coordinating the conversation, in addition to the costs of the third-party STT, LLM, and TTS models you choose to use. This model offers transparency and allows businesses to scale their costs directly with their usage.",[53,3075,3077],{"id":3076},"google-cloud-dialogflow-the-enterprise-grade-conversational-engine",[15,3078,3079],{},"Google Cloud Dialogflow: The Enterprise-Grade Conversational Engine",[45,3081],{":height":3082,":width":2992,"alt":3083,"format":2848,"loading":76,"src":3084},"1932","Google Dialogflow screenshot","/blog/top-5-voice-ai-agents-for-website-integration-in-2026/4.png",[11,3086,3087,3088,3093],{},"When we talk about building highly scalable and complex voice AI agents, Google Cloud's ",[23,3089,3092],{"href":3090,"rel":3091},"https://cloud.google.com/dialogflow",[27],"Dialogflow"," is a major part of the conversation. As Google's native platform for natural language understanding, it's designed to build conversational interfaces for everything from mobile apps to large-scale contact centers. For website integration, its primary advantage is its deep connection to the wider Google Cloud Platform (GCP), offering access to some of the most powerful AI and data tools available.",[78,3095,3097],{"id":3096},"two-flavors-es-for-simplicity-cx-for-complexity",[15,3098,3099],{},"Two Flavors: ES for Simplicity, CX for Complexity",[11,3101,3102],{},"Dialogflow comes in two main versions: ES (Essentials) and CX (Customer Experience).",[1242,3104,3105,3111],{},[1206,3106,3107,3110],{},[15,3108,3109],{},"Dialogflow ES"," is the original version, suitable for smaller or less complex agents. It uses a flat structure of \"intents\" to understand user requests, which is effective for straightforward conversations but can become difficult to manage in larger agents.",[1206,3112,3113,3116],{},[15,3114,3115],{},"Dialogflow CX"," is the newer, advanced offering designed for large and very complex agents. It uses a state machine approach, organizing conversations into \"flows\" and \"pages.\" This gives developers clear control over the conversational path, making it much easier to design, visualize, and maintain intricate, multi-turn dialogues. For building true agentic experiences on a website, Dialogflow CX is generally the more suitable choice.",[78,3118,3120],{"id":3119},"the-advantage-of-the-google-ecosystem",[15,3121,3122],{},"The Advantage of the Google Ecosystem",[11,3124,3125],{},"The real strength of using Dialogflow is that it doesn't exist in a vacuum. It seamlessly integrates with other Google Cloud services, allowing developers to build highly intelligent and capable agents.",[1242,3127,3128,3134,3140],{},[1206,3129,3130,3133],{},[15,3131,3132],{},"Vertex AI Integration:"," You can connect your Dialogflow agent to Google's Vertex AI platform. This opens up the ability to use state-of-the-art generative AI models for more dynamic, intelligent responses and to ground the agent in your company's own data.",[1206,3135,3136,3139],{},[15,3137,3138],{},"Google Cloud Functions:"," For executing actions, Dialogflow agents can trigger Cloud Functions. This allows the agent to run serverless code in response to a user's request, enabling it to interact with databases, call third-party APIs, or perform almost any back-end task.",[1206,3141,3142,3145],{},[15,3143,3144],{},"Contact Center AI (CCAI):"," Dialogflow is a core component of Google's CCAI platform. This means an agent built for a website can be part of a much larger customer service strategy, with the ability to hand off conversations to human agents with full context.",[78,3147,3149],{"id":3148},"website-integration-through-messenger",[15,3150,3151],{},"Website Integration Through Messenger",[11,3153,3154,3155,3158],{},"Google provides a direct way to embed a Dialogflow agent onto a website using ",[15,3156,3157],{},"Dialogflow Messenger",". This integration provides a simple, customizable chat widget that can be added to any webpage by embedding a small snippet of HTML code.",[11,3160,3161],{},"Through the Dialogflow console, you can configure the look and feel of the widget and enable it. For more advanced use cases, developers can use Dialogflow's REST APIs to build a completely custom user interface, giving them full control over the conversational experience on their site.",[11,3163,3164],{},"Dialogflow is priced on a pay-as-you-go basis, with costs determined by the version (CX or ES) and the number of requests. Voice sessions, which include both audio input and output, are billed per second of use. New customers often receive trial credits to help get started with the platform.",[53,3166,3168],{"id":3167},"voiceflow-the-collaborative-canvas-for-ai-agent-design",[15,3169,3170],{},"Voiceflow: The Collaborative Canvas for AI Agent Design",[45,3172],{":height":3173,":width":2846,"alt":3174,"format":2848,"loading":76,"src":3175},"1089","Voiceflow screenshot","/blog/top-5-voice-ai-agents-for-website-integration-in-2026/5.png",[11,3177,3178,3182],{},[23,3179,2804],{"href":3180,"rel":3181},"https://www.voiceflow.com/",[27]," enters the landscape with a different approach, focusing on the collaborative design and development of AI agents. It provides a visual, low-code platform where entire teams including designers, writers, and developers can work together to build complex conversational experiences. For website integration, this means that the logic and flow of the agent can be mapped out and prototyped in a highly intuitive, drag-and-drop environment before being deployed.",[78,3184,3186],{"id":3185},"visualizing-the-conversation",[15,3187,3188],{},"Visualizing the Conversation",[11,3190,3191],{},"The main feature of Voiceflow is its visual canvas. Instead of writing code to define conversational logic, you build it using blocks and connectors. This makes it much easier to visualize the user's journey, account for different conversational paths, and identify potential dead ends. This visual-first method brings several major benefits to building a website agent.",[1242,3193,3194,3200,3206],{},[1206,3195,3196,3199],{},[15,3197,3198],{},"Rapid Prototyping and Iteration:"," You can design, test, and refine a complete conversational flow directly within the Voiceflow canvas. The built-in prototyper lets you interact with your agent as you build it, making it fast to spot issues and make improvements without writing any deployment code.",[1206,3201,3202,3205],{},[15,3203,3204],{},"A Central Hub for Team Collaboration:"," Voiceflow's canvas acts as a single source of truth for the AI agent. Product managers can map out the high-level logic, UX writers can craft the dialogue, and developers can jump in to configure the technical integrations, all within the same shared workspace.",[1206,3207,3208,3211],{},[15,3209,3210],{},"Turning Design into Action:"," The visual design is not just a blueprint; it is the agent's executable logic. To make the agent truly agentic, developers can add API blocks or custom code snippets directly into the canvas. This allows the agent to fetch data from external sources, connect to services like a CRM or booking system, and perform actions on behalf of the user.",[78,3213,3215],{"id":3214},"from-canvas-to-website",[15,3216,3217],{},"From Canvas to Website",[11,3219,3220,3221,3224],{},"Voiceflow provides a straightforward path to get the agent you've designed onto your website. The primary method is through the ",[15,3222,3223],{},"Voiceflow Web Chat",", an embeddable widget that can be installed on any site with a single block of code.",[11,3226,3227],{},"This widget is highly customizable, allowing you to change its appearance to match your brand. It supports both voice and text input, giving users the flexibility to interact in the way they prefer. Once the widget is live, any changes you make to the agent's design on the Voiceflow canvas are updated in real-time, allowing for continuous improvement without needing to redeploy the code.",[11,3229,3230],{},"For teams wanting a more deeply integrated or custom front-end experience, Voiceflow also offers a Dialog Manager API. This allows developers to use Voiceflow as the conversational backend while building a completely bespoke user interface for their website.",[11,3232,3233],{},"Voiceflow's pricing is structured in tiers, with a free plan for individuals and small projects, a pro plan for growing teams, and an enterprise plan for large organizations that require advanced features like dedicated support and security reviews. This makes the platform accessible for a wide range of use cases, from simple informational bots to highly capable, task-performing agents.",[53,3235,3237],{"id":3236},"choosing-the-right-voice-ai-agent-for-your-website",[15,3238,3239],{},"Choosing the Right Voice AI Agent for Your Website",[11,3241,3242],{},"The transition from static, clickable websites to dynamic, conversational partners represents a major evolution in user experience. The five platforms we've examined each provide a different set of tools to build these agentic voice experiences. The suitable choice for your project will depend on your team's technical skills, your project's complexity, and your primary goals.",[11,3244,3245],{},"Here is a breakdown to help you identify which platform aligns best with your needs:",[1242,3247,3248,3254,3260,3266,3272],{},[1206,3249,3250,3253],{},[15,3251,3252],{},"Go with ElevenLabs if..."," your top priority is delivering the most realistic and emotionally expressive voice possible. It is a strong choice when the quality of the audio experience is a key part of your brand, and you want to get started quickly using a large library of pre-built integrations.",[1206,3255,3256,3259],{},[15,3257,3258],{},"Go with Deepgram if..."," you are building a custom voice application where conversational speed is the most important factor. Its unified, high-performance system is engineered for the lowest possible latency, making it ideal for developers who need to ensure conversations feel natural and immediate.",[1206,3261,3262,3265],{},[15,3263,3264],{},"Go with Vapi if..."," you are a developer-focused team that requires maximum flexibility. Vapi's integration platform lets you mix and match your preferred models for speech-to-text, language processing, and text-to-speech, giving you full control to build a \"best-of-breed\" agent.",[1206,3267,3268,3271],{},[15,3269,3270],{},"Go with Google Cloud Dialogflow if..."," you are an enterprise, particularly one already integrated into the Google Cloud ecosystem. Its CX version is built to handle highly complex, large-scale conversational agents that need to connect with other enterprise systems and data platforms.",[1206,3273,3274,3277],{},[15,3275,3276],{},"Go with Voiceflow if..."," your project is a collaborative effort between designers, writers, and developers. Its visual, low-code canvas makes it the ideal environment for teams to design, prototype, and manage the logic of an AI agent together before deploying it to a website.",[11,3279,3280],{},"Looking ahead, the capabilities of these voice agents are poised for even greater advancement. We can anticipate deeper website integration, where agents can not only converse but also actively guide users by highlighting elements, navigating pages, and filling out forms. As the underlying language models continue to become more intelligent, the web will move closer to becoming a collection of truly interactive and helpful conversational partners.",{"title":154,"searchDepth":321,"depth":321,"links":3282},[3283,3288,3292,3297,3302,3306],{"id":2839,"depth":321,"text":2842,"children":3284},[3285,3286,3287],{"id":2859,"depth":335,"text":2862},{"id":2891,"depth":335,"text":2894},{"id":2903,"depth":335,"text":2906},{"id":2912,"depth":321,"text":2915,"children":3289},[3290,3291],{"id":2932,"depth":335,"text":2935},{"id":2970,"depth":335,"text":2973},{"id":2985,"depth":321,"text":2988,"children":3293},[3294,3295,3296],{"id":3005,"depth":335,"text":3008},{"id":3035,"depth":335,"text":3038},{"id":3064,"depth":335,"text":3067},{"id":3076,"depth":321,"text":3079,"children":3298},[3299,3300,3301],{"id":3096,"depth":335,"text":3099},{"id":3119,"depth":335,"text":3122},{"id":3148,"depth":335,"text":3151},{"id":3167,"depth":321,"text":3170,"children":3303},[3304,3305],{"id":3185,"depth":335,"text":3188},{"id":3214,"depth":335,"text":3217},{"id":3236,"depth":321,"text":3239},"voice-ai","2025-10-10","The top 5 voice AI platforms for website integration in 2026: ElevenLabs, Deepgram, Vapi, Dialogflow and Voiceflow, compared.",[3311,3314,3317,3320],{"question":3312,"answer":3313},"What are the top voice AI agent platforms for website integration in 2026?","The article compares five platforms: ElevenLabs, Deepgram, Vapi, Google Dialogflow, and Voiceflow. ElevenLabs focuses on realistic voice with 400+ integrations, Deepgram on speed with a unified API, Vapi on developer flexibility, Dialogflow on enterprise scale within Google Cloud, and Voiceflow on collaborative visual design. Each suits different priorities such as voice quality, latency, flexibility, or team workflow.",{"question":3315,"answer":3316},"Which voice AI platform has the lowest latency?","Deepgram is engineered specifically for speed, reporting response times under 250 milliseconds by running a complete in-house technology stack. ElevenLabs also targets low latency, with streaming APIs capable of delivering audio in under 100 milliseconds. Both aim to keep voice conversations feeling immediate and natural.",{"question":3318,"answer":3319},"What makes Vapi different from all-in-one voice AI platforms?","Vapi acts as a coordination layer rather than providing its own models for every step, letting developers mix and match best-in-class services. A team could combine Deepgram for speech-to-text, GPT-4o or Claude 3 for reasoning, and ElevenLabs for voice output. Vapi manages the real-time data flow between these services and handles WebSocket connections.",{"question":3321,"answer":3322},"How do you add a voice AI agent to a website?","Most platforms offer an embeddable widget installed with a small snippet of HTML or a single line of code, such as ElevenLabs' code snippet, Dialogflow Messenger, and Voiceflow Web Chat. For more control, developers can use SDKs or APIs to build a fully custom interface, as with Deepgram's SDKs or Vapi's JavaScript SDK. Widget deployment can take minutes rather than months.",false,"/use-case/voice-agents",{},"/blog/top-5-voice-ai-agents-for-website-integration-in-2026",{"title":2756,"description":3309},{"loc":3326},"blog/1018.top-5-voice-ai-agents-for-website-integration-in-2026",[2731,3331,3332,2748,2749],"browser-agents","voice-agents","lyPv7nB7Lq9ewvl28IelqhQuQNiLmZ2fYTcEhQZ7e0U",{"id":3335,"title":3336,"authorId":6,"body":3337,"category":2731,"created":3797,"description":3798,"extension":1822,"faqs":2734,"featurePriority":2734,"head":2734,"hideCta":1834,"landingPath":2734,"meta":3799,"navigation":1834,"ogImage":2734,"path":3805,"robots":2734,"schemaOrg":2734,"seo":3806,"sitemap":3807,"stem":3808,"tags":3809,"__hash__":3810},"blog/blog/1011.a-gentle-introduction-to-ai-agents-for-the-web.md","A Gentle Introduction to AI Agents for the Web",{"type":8,"value":3338,"toc":3779},[3339,3357,3361,3369,3374,3377,3381,3384,3404,3408,3417,3421,3437,3442,3445,3449,3460,3464,3467,3471,3478,3482,3505,3509,3513,3516,3519,3523,3544,3547,3551,3555,3564,3577,3582,3586,3595,3598,3601,3605,3608,3611,3615,3624,3627,3631,3635,3638,3642,3646,3649,3723,3725,3728,3742],[11,3340,3341,3346,3347,3346,3351,3356],{},[23,3342,3345],{"href":3343,"rel":3344},"https://openai.com/index/introducing-operator/",[27],"Operator",", ",[23,3348,170],{"href":3349,"rel":3350},"https://browser-use.com/",[27],[23,3352,3355],{"href":3353,"rel":3354},"https://www.skyvern.com/",[27],"Skyvern"," – we are currently witnessing an evolution of autonomous web browsing agents. The concept of agents is, in general, ubiquitous in the era of AI.",[53,3358,3360],{"id":3359},"what-is-an-agent","What is an Agent?",[11,3362,3363,3364,18],{},"An agent is an entity that acts on behalf of another entity – its principal. Robotic agents are modelled after humans: they perceive the world and act in it to achieve a specified goal. Whilst humans perceive through their senses, agents perceive through sensors; whilst humans act through motor systems (e.g., fingers), agents act through actuators. This terminology has been transferred from hard- to software agents, particularly those premised on artificial intelligence (AI). An agent's purpose, that is, its goal, is clearly defined; the principal interfaces with that goal by assigning the agent tasks",[147,3365,3366],{},[23,3367,156],{"href":151,"ariaDescribedBy":3368,"dataFootnoteRef":154,"id":155},[153],[45,3370],{"alt":3371,"loading":76,"src":3372,"provider":1316,":width":3373},"Figurative comparison of component equivalents between a human and an AI agent","/blog/a-gentle-introduction-to-ai-agents-for-the-web/1.svg","430",[11,3375,3376],{},"The analogy is more than decorative: an agent takes over the tool-use expertise, leaving the principal with only the domain expertise. Traditional software applications like Photoshop demand both.",[78,3378,3380],{"id":3379},"what-makes-an-agent-intelligent","What Makes an Agent Intelligent?",[11,3382,3383],{},"For acting on behalf of a human, developing an agent could be as simple as making it follow a hardcoded sequence of actions. This describes process automation, but not an artificially intelligent agent. Whilst definitions for AI differ, agent intelligence comes from making informed decisions that are not hardcoded. Here are common – not necessarily mutually exclusive – high-level approaches to induce intelligence:",[1242,3385,3386,3392,3398],{},[1206,3387,3388,3391],{},[15,3389,3390],{},"Search"," – the agent has a complete or heuristic model of the world and the action space to anticipate the effects of actions and search for a goal state.",[1206,3393,3394,3397],{},[15,3395,3396],{},"Reinforcement learning"," – the agent has an initial action space and iteratively discovers the world and effects of actions, learning an optimal policy to act in order to reach a goal state.",[1206,3399,3400,3403],{},[15,3401,3402],{},"Neural networks"," – the agent has no model, only examples: a network is fitted to task–solution instances and generalises to unseen, yet related tasks.",[78,3405,3407],{"id":3406},"a-quick-note-on-multi-agent-systems","A Quick Note on Multi-Agent Systems",[11,3409,3410,3411,3416],{},"Agent design gets an additional edge when other entities act in the same world as the agent. Such entities can be considered agents on their own. Their beliefs and goals might affect the agent: cooperative agents share the designed agent's goal, whilst adversarial agents have an opposite goal. In a zero-sum game, an adversarial agent maximises the exact inverse of the designed agent's utility. Knowing this, the designed agent needs to anticipate intelligent decisions by the adversary (e.g., by applying the ",[23,3412,3415],{"href":3413,"rel":3414},"https://cs.stanford.edu/people/eroberts/courses/soco/projects/2003-04/intelligent-search/minimax.html",[27],"Minimax algorithm","), as opposed to random world changes. That said, web browsing – modelled after humans – is not a typical multi-agent scenario.",[78,3418,3420],{"id":3419},"the-agent-loop","The Agent Loop",[11,3422,3423,3428,3429,3436],{},[23,3424,3427],{"href":3425,"rel":3426},"https://www.nngroup.com/people/don-norman/",[27],"Don Norman","'s ",[23,3430,3433],{"href":3431,"rel":3432},"https://jnd.org/books/the-design-of-everyday-things-revised-and-expanded-edition/",[27],[37,3434,3435],{},"Seven Stages of Action"," is a model of the human cognition cycle that generalises to an agent's lifecycle – the agent loop: with a (1) goal, (2) intent to act, (3) plan actions, (4) execute the action plan – this alters the world – then (5) perceive the new world state, (6) interpret the perception, and finally (7) evaluate the interpretation – report task success, or an irrevocable failure, or continue the cycle if the world state not yet reflects a goal state.",[45,3438],{"alt":3439,"loading":76,"src":3440,"provider":1316,":width":3441},"Don Norman's Seven Stages of Action – a model of human cognition","/blog/a-gentle-introduction-to-ai-agents-for-the-web/2.svg","500",[11,3443,3444],{},"Humans can run many of these cycles hierarchically, with increasing frequency the smaller the task. Take driving a car as an example: choosing a route runs on the order of minutes, choosing when to change lanes on seconds, and holding the car in its lane on fractions of a second. A cycle for holding the lane (1) is: intent to actively hold the lane (2), plan to keep the steering wheel straight, then watch the road (3), execute plan (4), observe the road (5), check whether the car is in the lane (6), if not, cycle again with an adjusted action plan (e.g., slightly rotate the steering wheel to the right) (7).",[78,3446,3448],{"id":3447},"errors-and-recovery","Errors and Recovery",[11,3450,3451,3452,3455,3456,3459],{},"Precisely speaking, \"to err is human\" means that errors are inevitable when heuristically navigating the world. Consequently, AI agents regularly run into errors, too. Norman's Seven Stages of Action make two error spaces apparent: when executing an action plan; the ",[37,3453,3454],{},"Gulf of Action (Execution)"," – how to act in the world? – and when perceiving world state; the ",[37,3457,3458],{},"Gulf of Perception (Evaluation)"," – how to understand the world?",[45,3461],{"alt":3462,"loading":76,"src":3463,"provider":1316,":width":3441},"The Gulf of Action (Execution) and  Perception (Evaluation) in Don Norman's Seven Stages of Action","/blog/a-gentle-introduction-to-ai-agents-for-the-web/3.svg",[11,3465,3466],{},"There are two notable kinds of errors within the Gulf of Action: mistakes and slips. A mistake occurs when the action plan is wrong, although executing this action plan might go right in itself. A slip occurs when the action plan is right, but the execution goes wrong. Here is an example: stopping a manual-shift car without pressing the clutch. This error describes a mistake if the driver did not know about the need to press the clutch, e.g., because they only knew how to operate automatic-drive cars. It is, in turn, a slip if they simply forgot to press the clutch because of fatigue. Since cognition cycles are hierarchic, most errors can be recovered within a cycle. Hearing a choking engine sound, the driver might adjust the action plan and press the clutch late. Errors that cannot be recovered – picture the car's engine dying – require adjusting the goal, which can be seen as moving on to a new cycle.",[53,3468,3470],{"id":3469},"web-browsing-agents","Web Browsing Agents",[11,3472,3473,3474,3477],{},"The advent of large language models (LLMs) has recently opened a new perspective on developing web agents – falling back on the neural networks approach to intelligence. An LLM can serve as a plug-in reasoning backend: given a task and somehow encoded UI state, it can suggest input actions – type and target, e.g., click at ",[104,3475,3476],{},"(200, 375)"," – that progress the given task. Moreover, it can decide whether a goal state has been reached. Providing a state-and-action history within context conveys to the model prior outcomes and contributes to in-context learning.",[78,3479,3481],{"id":3480},"web-browsing-is-all-about-the-journey","Web Browsing is All About the Journey",[11,3483,3484,3485,3488,3489,3494,3495,3500,3501,3504],{},"A web browsing task is any task that can be solved through a web-based user interface (UI). For instance: ",[37,3486,3487],{},"\"Book the cheapest flight from Amsterdam to New York tomorrow night\""," on ",[23,3490,3493],{"href":3491,"rel":3492},"https://www.klm.com",[27],"klm.com",". Application programming interfaces (APIs; e.g., ",[23,3496,3499],{"href":3497,"rel":3498},"https://www.ibm.com/think/topics/rest-apis",[27],"REST APIs",") are often restricted, incomplete, or unavailable. For that reason, web browsing agents today have to rely on human-facing web application UIs. Many agent use cases, such as for customer experience, even require a human in the loop. In this context, web browsing agents come at two levels: outcome-driven agents care only about reaching a goal state, which favours shortcuts like direct URL navigation or expert navigation like ",[104,3502,3503],{},"Ctrl + F",", whilst journey-driven agents are supposed to browse a web application in human fashion, via the application-native UI.",[45,3506],{"alt":3507,"loading":76,"src":3508},"A depiction of a web journey over three web application UI states","/blog/a-gentle-introduction-to-ai-agents-for-the-web/4.png",[78,3510,3512],{"id":3511},"a-formal-model-for-web-browsing","A Formal Model for Web Browsing",[11,3514,3515],{},"The world of web application UIs is hard to formalise; a complete model spans the union of every reachable UI state in every web application on the web. For that reason, conventional AI has failed to produce capable generalist web agents. From a web agent design perspective, however, an abstract model of web browsing is sufficient.",[11,3517,3518],{},"Web browsing might afford a model that is history-independent: a human could certainly pick up a partially completed browsing task, completing the payment process on an airline website for an already selected flight – assuming prior actions were correct with respect to the specified task (e.g., the selected destination).",[45,3520],{"alt":3521,"loading":76,"src":3522},"Screenshot of the payment page on klm.com that reflects outcomes of prior actions and affords next actions","/blog/a-gentle-introduction-to-ai-agents-for-the-web/5.png",[11,3524,3525,3526,3531,3532,3537,3538,3543],{},"Yet, many web applications lack ",[23,3527,3530],{"href":3528,"rel":3529},"https://www.nngroup.com/articles/ten-usability-heuristics/#toc-1-visibility-of-system-status-1",[27],"visibility of system status"," – against ",[23,3533,3536],{"href":3534,"rel":3535},"https://www.nngroup.com/people/jakob-nielsen/",[27],"Jakob Nielsen","'s first heuristic for UI usability. Visibility of system status comprises reflecting both outcomes of prior actions and affordances for next actions in the UI. With this concept, web browsing is best described as a partially observable Markov decision process (POMDP)",[147,3539,3540],{},[23,3541,179],{"href":176,"ariaDescribedBy":3542,"dataFootnoteRef":154,"id":178},[153]," with, adapting to discovered UI states, where states can be tuples of UI state trajectories to contain history.",[11,3545,3546],{},"Reflex agents take actions that solely depend on the current world state. Planning agents, in turn, plan ahead. For most web browsing tasks, future UI states are roughly predictable, given that web journeys are similar across different web applications that solve similar tasks. In POMDP terms, the expected next UI state has a high probability, whilst unexpected states remain with a non-negligible probability. The further a web journey is progressed, the higher the probability of the next expected UI state gets: the flight selection step is likely followed by the passenger details step, which is even more likely followed by the payment step.",[45,3548],{"alt":3549,"loading":76,"src":3550},"Graph-based web journey representation of an unknown, adaptive, partially observable Markov decision process (POMDP)","/blog/a-gentle-introduction-to-ai-agents-for-the-web/6.png",[78,3552,3554],{"id":3553},"llm-sensors-actuators-agent","LLM + Sensors + Actuators = Agent",[11,3556,3557,3558,3563],{},"Conversationally satisfying simple information requests does not reasonably qualify an LLM as an agent – no more than it would a TF-IDF search system",[147,3559,3560],{},[23,3561,194],{"href":191,"ariaDescribedBy":3562,"dataFootnoteRef":154,"id":193},[153],", or a colleague telling you the time. The web agent LLM backend is metaphorically the brain of the agent. It abstracts the world model and state transition function, and can decide whether a goal state was reached.",[11,3565,3566,3567,3346,3570,3346,3573,3576],{},"Turning an LLM into a web agent means, at the core, connecting it with sensors and actuators, for example, the ability to take DOM snapshots and dispatch clicks via CSS selectors. This is a specialised architecture of LLM tool calling. The set of available input actions is known from human computer use: ",[37,3568,3569],{},"clicking",[37,3571,3572],{},"typing",[37,3574,3575],{},"scrolling",", etc. The agent loop is then a continuous cycle of serialising UI state for the model to elicit action suggestions under a goal derived from the user-specified task.",[45,3578],{"alt":3579,"loading":76,"src":3580,"provider":1316,":width":3581},"Figurative depiction of components in the agent loop","/blog/a-gentle-introduction-to-ai-agents-for-the-web/7.svg","460",[78,3583,3585],{"id":3584},"snapshots-serialised-ui-state","Snapshots – Serialised UI State",[11,3587,3588,3589,3594],{},"There are plenty of ways to serialise the state of a web-based UI – referred to as a snapshot. A screenshot resembles what humans see when viewing a web page's graphical UI (GUI) at a given point in time. LLMs have demonstrated the ability to interpret HTML and even navigate a UI encoded in this way",[147,3590,3591],{},[23,3592,689],{"href":686,"ariaDescribedBy":3593,"dataFootnoteRef":154,"id":688},[153],". The document object model (DOM) is a web browser's runtime model of a web page. Serialised as HTML, DOM snapshots represent a compelling alternative to GUI snapshots, offering certain advantages, such as processability (e.g., filtering). Filtering the DOM by pruning hidden, disabled, or redundant subtrees saves LLM communication costs and works towards more stable and predictable journeys.",[11,3596,3597],{},"The type of snapshot representation provided to an LLM dictates how an element can be targeted for an action. Vision-based targeting is indirect, i.e., by coordinates. DOM-based targeting is direct, e.g., by CSS selectors or object references. GUI snapshots become a necessity when the target application is graphics-based, such as a canvas-based application like Google Maps.",[11,3599,3600],{},"Outcome-driven web agents have used UI-invasive snapshot measures, such as grounding LLMs with bounding boxes around actionable elements.",[45,3602],{"alt":3603,"loading":76,"src":3604},"Example of a screenshot (GUI snapshot) with coloured bounding boxes that visually ground an LLM","/blog/a-gentle-introduction-to-ai-agents-for-the-web/8.png",[11,3606,3607],{},"The ideal snapshot representation depends on the task. To get an idea, classify between navigation and information retrieval tasks: navigation tasks require snapshots to preserve actionable elements, whilst retrieval tasks require them to preserve text – both can be sourced from DOM snapshots. Accessibility tree snapshots are another DOM-sourced representation. Web UIs rarely afford merely a single task, so many of their components are noise in any given snapshot – which is not just a UI serialisation, but one scoped to the task at hand.",[11,3609,3610],{},"For web agents, actions are expressive, so the Gulf of Action mostly grows when the snapshot-encoded UI state and the live UI state have irrecoverably drifted apart. The agent loop should be fast to prevent such drift. The Gulf of Perception is much wider by the nature of the problem: what snapshot representation can an LLM interpret, and which representation is best?",[78,3612,3614],{"id":3613},"the-agent-harness","The Agent Harness",[11,3616,3617,3618,3623],{},"Any custom logic that sits between the LLM backend and the web page UI (problem world) is considered an agent harness. Common aspects of a harness include task decomposition, model routing, snapshot creation, history management, error recovery, and timeouts. ",[23,3619,3622],{"href":3620,"rel":3621},"https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents",[27],"System prompts"," are also part of the harness, instructing a general-purpose LLM to model a web agent for a specific domain – right in context.",[11,3625,3626],{},"Designing a harness is where the POMDP and Seven Stages of Action models come into play. The POMDP model helps with context engineering, for instance, how much history to provide to the LLM. The Seven Stages of Action set out problem components and artefacts, for example, the LLM or snapshots, respectively.",[45,3628],{"alt":3629,"loading":76,"src":3630,"provider":1316},"Don Norman's Seven Stages of Action mapped to the agentic web browsing problem","/blog/a-gentle-introduction-to-ai-agents-for-the-web/9.svg",[78,3632,3634],{"id":3633},"the-model-context-protocol","The Model Context Protocol",[11,3636,3637],{},"LLM providers commonly offer a structured output interface, which allows eliciting well-formed action suggestions from an LLM. A client can subsequently parse and drive these suggested actions in the live user interface. Through the model context protocol (MCP) – another web-based RPC standard – structured output can be bridged with tools that LLMs can call directly, without an extra client trip.",[45,3639],{"alt":3640,"loading":76,"src":3641,"provider":1316,":width":3441},"Figurative depiction of components in the agent loop using the model context protocol (MCP)","/blog/a-gentle-introduction-to-ai-agents-for-the-web/10.svg",[78,3643,3645],{"id":3644},"web-browser-environments","Web Browser Environments",[11,3647,3648],{},"When connected to web perception and actuation tools via MCP, an ordinary AI chat UI already qualifies as an agent UI; the MCP integration decides where the actual web browser runs. There are several types of browser runtime environment:",[1242,3650,3651,3666,3679,3691],{},[1206,3652,3653,3656,3657,3660,3661],{},[15,3654,3655],{},"Agentic browser"," – the agent's UI embeds a browser.",[3658,3659],"br",{},"\nExample: ",[23,3662,3665],{"href":3663,"rel":3664},"https://www.perplexity.ai/comet",[27],"Perplexity Comet",[1206,3667,3668,3671,3672,3660,3674],{},[15,3669,3670],{},"Cloud browser"," – the agent spins up a browser in the cloud.",[3658,3673],{},[23,3675,3678],{"href":3676,"rel":3677},"https://www.browserbase.com",[27],"Browserbase",[1206,3680,3681,3684,3685,3660,3687],{},[15,3682,3683],{},"Local browser (3rd-party browser)"," – the agent spins up an isolated browser locally.",[3658,3686],{},[23,3688,3690],{"href":25,"rel":3689},[27],"Chromium + Playwright",[1206,3692,3693,3696,3697],{},[15,3694,3695],{},"Local browser (1st-party browser)"," – the agent is embedded within a web application running in a user's personal browser, e.g., in the form of a chatbot.\n",[1242,3698,3699,3712],{},[1206,3700,3701,3704,3705,3660,3707],{},[15,3702,3703],{},"Implemented (1st-party app)"," – served directly through the website.",[3658,3706],{},[23,3708,3711],{"href":3709,"rel":3710},"https://www.cognigy.com/platform/ai-agent-studio",[27],"Cognigy Studio",[1206,3713,3714,3717,3718,3660,3720],{},[15,3715,3716],{},"Augmented (3rd-party app)"," – served indirectly through a website proxy.",[3658,3719],{},[23,3721,34],{"href":32,"rel":3722},[27],[2588,3724],{},[11,3726,3727],{},"Building AI agents has been approached from different angles. LLMs have opened up new perspectives on the design and development of AI agents, particularly web agents, which work in a problem domain that is hard to grasp with a formal model. Whilst higher-level agent methodology can borrow much from established research, LLMs have introduced new challenges, such as how to take UI state snapshots for eliciting input action suggestions from such models.",[11,3729,3730,3731,3736,3737,18],{},"At Webfuse, we research agentic web browsing and develop a platform for agentic browsing. Our web application reverse proxy allows LLMs to perceive and act on the live web. Use cases include connecting a chat UI like ",[23,3732,3735],{"href":3733,"rel":3734},"https://claude.com/",[27],"Claude",", or shipping ",[23,3738,3741],{"href":3739,"rel":3740},"https://github.com/webfuse-com/extension-elevenlabs-mcp",[27],"perceive-and-act-ready chatbots",[453,3743,3745,3748],{"className":3744,"dataFootnotes":154},[2610],[53,3746,2615],{"className":3747,"id":153},[2614],[1203,3749,3750,3759,3765,3771],{},[1206,3751,3752,2625,3756],{"id":2620},[23,3753,3754],{"href":3754,"rel":3755},"https://inst.eecs.berkeley.edu/~cs188/textbook/textbook_full.pdf",[27],[23,3757,2632],{"href":2628,"ariaLabel":2629,"className":3758,"dataFootnoteBackref":154},[2631],[1206,3760,3761,3762],{"id":2635},"A POMDP is a decision process in which the agent cannot see the exact world state, only partial observations of it, and so must choose actions against a probability distribution over the states it might be in. ",[23,3763,2632],{"href":2643,"ariaLabel":2644,"className":3764,"dataFootnoteBackref":154},[2631],[1206,3766,3767,3768],{"id":2648},"TF-IDF (term frequency-inverse document frequency) is an information retrieval method that ranks documents by how often a query term occurs in each one, weighted down by how common that term is across the whole collection – so terms that are frequent here but rare elsewhere carry the most weight. ",[23,3769,2632],{"href":2655,"ariaLabel":2656,"className":3770,"dataFootnoteBackref":154},[2631],[1206,3772,3773,2625,3776],{"id":2660},[23,3774,2651],{"href":2651,"rel":3775},[27],[23,3777,2632],{"href":2667,"ariaLabel":2668,"className":3778,"dataFootnoteBackref":154},[2631],{"title":154,"searchDepth":321,"depth":321,"links":3780},[3781,3787,3796],{"id":3359,"depth":321,"text":3360,"children":3782},[3783,3784,3785,3786],{"id":3379,"depth":335,"text":3380},{"id":3406,"depth":335,"text":3407},{"id":3419,"depth":335,"text":3420},{"id":3447,"depth":335,"text":3448},{"id":3469,"depth":321,"text":3470,"children":3788},[3789,3790,3791,3792,3793,3794,3795],{"id":3480,"depth":335,"text":3481},{"id":3511,"depth":335,"text":3512},{"id":3553,"depth":335,"text":3554},{"id":3584,"depth":335,"text":3585},{"id":3613,"depth":335,"text":3614},{"id":3633,"depth":335,"text":3634},{"id":3644,"depth":335,"text":3645},{"id":153,"depth":321,"text":2615},"2025-06-15","Get an idea of agents and how to build AI agents that browse the web: key concepts, terminology, and challenges.",{"homepage":1834,"relatedLinks":3800},[3801,3803,3804],{"text":3802,"href":2743},"Serialising Web UI State for LLMs",{"text":2738,"href":2739},{"text":2741,"href":2742,"external":1834},"/blog/a-gentle-introduction-to-ai-agents-for-the-web",{"title":3336,"description":3798},{"loc":3805},"blog/1011.a-gentle-introduction-to-ai-agents-for-the-web",[2731,3331,2748,2749,2750],"qj5myg3Lr85323PUpPylNvbUtnv8IFDkHGMAbfnCcqg",1787685712953]