[{"data":1,"prerenderedAt":938},["ShallowReactive",2],{"/rnd/dom-downsampling-for-llm-based-web-agents":3},{"id":4,"title":5,"authorId":6,"body":7,"created":918,"description":919,"extension":920,"head":921,"meta":922,"navigation":923,"ogImage":921,"paperAbstract":924,"paperAuthors":925,"paperContributors":927,"paperDate":930,"paperLink":931,"paperTitle":932,"path":933,"robots":921,"schemaOrg":921,"seo":934,"sitemap":935,"stem":936,"__hash__":937},"rnd/rnd/0.dom-downsampling-for-llm-based-web-agents.md","DOM Downsampling for LLM-Based Web Agents","thassilo-schiepanski",{"type":8,"value":9,"toc":907},"minimark",[10,16,25,32,37,54,59,62,85,90,93,424,427,444,598,603,667,671,690,694,698,707,711,715,719,722,726,729,733,752,814,817,870,879,903],[11,12],"nuxt-picture",{"alt":13,"format":14,"src":15},"Example of downsampling on an image and equivalently the DOM","png","/rnd/0.dom-downsampling-for-llm-based-web-agents/1.png",[17,18,19,20,24],"p",{},"Large language models (LLMs) have recently sparked an evolution of autonomous web browsing agents. In this paradigm, an LLM functions as a plug-in reasoning backend: provided with serialised user interface (UI) state (referred to as a ",[21,22,23],"em",{},"snapshot","), it is expected to suggest input actions that incrementally solve a given web browsing task using the underlying UI. The central challenge is how best to compile a snapshot to elicit accurate input suggestions from an LLM, with few input tokens.",[17,26,27,28,31],{},"The graphical UI (GUI) is commonly serialised as screenshots. Image input to multimodal LLMs is downsampled – something like averaging ",[21,29,30],{},"n","-tiles of pixels; as a result, vision capabilities lack pixel precision. Web agents have thus progressively relied on grounded GUI snapshots: screenshots enhanced with visual cues associated with text-based grounding.",[11,33],{"alt":34,"format":14,"src":35,"loading":36},"Grounded GUI snapshot following Set-of-Mark prompting","/rnd/0.dom-downsampling-for-llm-based-web-agents/2.png","lazy",[17,38,39,40,47,48,53],{},"Image input comes with wall clock time overhead. Depending on the use case, grounding in the GUI is undesirable. Document object model (DOM) snapshots represent a compelling alternative to GUI snapshots. Capabilities of LLMs to interpret HTML and even navigate an encoded UI have previously been demonstrated. However, their excessive input token footprint precludes reliable implementation with web agents to date. ",[41,42,46],"a",{"href":43,"rel":44},"https://openai.com/index/gpt-4o-system-card/",[45],"nofollow","GPT-4o",", a multimodal, general-purpose model, comes with a context window of 128K tokens. At an estimate of 1.5M tokens, a DOM snapshot of ",[41,49,52],{"href":50,"rel":51},"https://edition.cnn.com",[45],"CNN's homepage"," would exceed the model's context window.",[55,56,58],"h2",{"id":57},"introducing-dom-downsampling","Introducing: DOM Downsampling",[17,60,61],{},"At Webfuse, we are developing a platform for agentic browsing. For our perception API, we've been researching techniques to facilitate an LLM's interpretation of snapshots. Our first goal: reliably and comfortably fitting DOM snapshots within the context window of a frontier LLM, whilst preserving the overall DOM structure and encoded UI features.",[17,63,64,65,67,68,71,72,76,77,80,81,84],{},"We transfer the concept of downsampling to the DOM: analogously, averaging ",[21,66,30],{},"-node DOM subtrees – trading fidelity for size. Desirably, the result remains a valid DOM that can be further processed, e.g., translated to an accessibility tree. As a first-of-its-kind approach, we propose the algorithm ",[21,69,70],{},"D2Snap"," (",[73,74,75],"ins",{},"D","ownsample ",[73,78,79],{},"DOM"," ",[73,82,83],{},"Snap","shots).",[86,87,89],"h3",{"id":88},"examples-of-dom-downsampling","Examples of DOM Downsampling",[17,91,92],{},"Below is an example of an HTML-serialised DOM subtree:",[94,95,100],"pre",{"className":96,"code":97,"language":98,"meta":99,"style":99},"language-html shiki shiki-themes github-dark github-dark","\u003Csection class=\"container\" tabindex=\"3\" required=\"true\" type=\"example\">\n  \u003Cdiv class=\"mx-auto\" data-topic=\"products\" required=\"false\">\n    \u003Ch1>Our Pizza\u003C/h1>\n    \u003Cdiv>\n      \u003Cdiv class=\"shadow-lg\">\n        \u003Ch2>Margherita\u003C/h2>\n        \u003Cp>\n         A simple classic: mozzarella, tomatoes and basil.\n         An everyday choice!\n        \u003C/p>\n        \u003Cbutton type=\"button\">Add\u003C/button>\n      \u003C/div>\n      \u003Cdiv class=\"shadow-lg\">\n        \u003Ch2>Capricciosa\u003C/h2>\n        \u003Cp>\n          A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n          A true favourite!\n        \u003C/p>\n        \u003Cbutton type=\"button\">Add\u003C/button>\n      \u003C/div>\n    \u003C/div>\n  \u003C/div>\n\u003C/section>\n","html","",[101,102,103,154,187,203,212,229,244,253,259,265,275,297,307,322,336,345,351,357,366,385,394,404,414],"code",{"__ignoreMap":99},[104,105,108,112,116,120,123,127,130,132,135,138,140,143,146,148,151],"span",{"class":106,"line":107},"line",1,[104,109,111],{"class":110},"suv1-","\u003C",[104,113,115],{"class":114},"sxg3X","section",[104,117,119],{"class":118},"sFR8T"," class",[104,121,122],{"class":110},"=",[104,124,126],{"class":125},"s4wv1","\"container\"",[104,128,129],{"class":118}," tabindex",[104,131,122],{"class":110},[104,133,134],{"class":125},"\"3\"",[104,136,137],{"class":118}," required",[104,139,122],{"class":110},[104,141,142],{"class":125},"\"true\"",[104,144,145],{"class":118}," type",[104,147,122],{"class":110},[104,149,150],{"class":125},"\"example\"",[104,152,153],{"class":110},">\n",[104,155,157,160,163,165,167,170,173,175,178,180,182,185],{"class":106,"line":156},2,[104,158,159],{"class":110},"  \u003C",[104,161,162],{"class":114},"div",[104,164,119],{"class":118},[104,166,122],{"class":110},[104,168,169],{"class":125},"\"mx-auto\"",[104,171,172],{"class":118}," data-topic",[104,174,122],{"class":110},[104,176,177],{"class":125},"\"products\"",[104,179,137],{"class":118},[104,181,122],{"class":110},[104,183,184],{"class":125},"\"false\"",[104,186,153],{"class":110},[104,188,190,193,196,199,201],{"class":106,"line":189},3,[104,191,192],{"class":110},"    \u003C",[104,194,195],{"class":114},"h1",[104,197,198],{"class":110},">Our Pizza\u003C/",[104,200,195],{"class":114},[104,202,153],{"class":110},[104,204,206,208,210],{"class":106,"line":205},4,[104,207,192],{"class":110},[104,209,162],{"class":114},[104,211,153],{"class":110},[104,213,215,218,220,222,224,227],{"class":106,"line":214},5,[104,216,217],{"class":110},"      \u003C",[104,219,162],{"class":114},[104,221,119],{"class":118},[104,223,122],{"class":110},[104,225,226],{"class":125},"\"shadow-lg\"",[104,228,153],{"class":110},[104,230,232,235,237,240,242],{"class":106,"line":231},6,[104,233,234],{"class":110},"        \u003C",[104,236,55],{"class":114},[104,238,239],{"class":110},">Margherita\u003C/",[104,241,55],{"class":114},[104,243,153],{"class":110},[104,245,247,249,251],{"class":106,"line":246},7,[104,248,234],{"class":110},[104,250,17],{"class":114},[104,252,153],{"class":110},[104,254,256],{"class":106,"line":255},8,[104,257,258],{"class":110},"         A simple classic: mozzarella, tomatoes and basil.\n",[104,260,262],{"class":106,"line":261},9,[104,263,264],{"class":110},"         An everyday choice!\n",[104,266,268,271,273],{"class":106,"line":267},10,[104,269,270],{"class":110},"        \u003C/",[104,272,17],{"class":114},[104,274,153],{"class":110},[104,276,278,280,283,285,287,290,293,295],{"class":106,"line":277},11,[104,279,234],{"class":110},[104,281,282],{"class":114},"button",[104,284,145],{"class":118},[104,286,122],{"class":110},[104,288,289],{"class":125},"\"button\"",[104,291,292],{"class":110},">Add\u003C/",[104,294,282],{"class":114},[104,296,153],{"class":110},[104,298,300,303,305],{"class":106,"line":299},12,[104,301,302],{"class":110},"      \u003C/",[104,304,162],{"class":114},[104,306,153],{"class":110},[104,308,310,312,314,316,318,320],{"class":106,"line":309},13,[104,311,217],{"class":110},[104,313,162],{"class":114},[104,315,119],{"class":118},[104,317,122],{"class":110},[104,319,226],{"class":125},[104,321,153],{"class":110},[104,323,325,327,329,332,334],{"class":106,"line":324},14,[104,326,234],{"class":110},[104,328,55],{"class":114},[104,330,331],{"class":110},">Capricciosa\u003C/",[104,333,55],{"class":114},[104,335,153],{"class":110},[104,337,339,341,343],{"class":106,"line":338},15,[104,340,234],{"class":110},[104,342,17],{"class":114},[104,344,153],{"class":110},[104,346,348],{"class":106,"line":347},16,[104,349,350],{"class":110},"          A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[104,352,354],{"class":106,"line":353},17,[104,355,356],{"class":110},"          A true favourite!\n",[104,358,360,362,364],{"class":106,"line":359},18,[104,361,270],{"class":110},[104,363,17],{"class":114},[104,365,153],{"class":110},[104,367,369,371,373,375,377,379,381,383],{"class":106,"line":368},19,[104,370,234],{"class":110},[104,372,282],{"class":114},[104,374,145],{"class":118},[104,376,122],{"class":110},[104,378,289],{"class":125},[104,380,292],{"class":110},[104,382,282],{"class":114},[104,384,153],{"class":110},[104,386,388,390,392],{"class":106,"line":387},20,[104,389,302],{"class":110},[104,391,162],{"class":114},[104,393,153],{"class":110},[104,395,397,400,402],{"class":106,"line":396},21,[104,398,399],{"class":110},"    \u003C/",[104,401,162],{"class":114},[104,403,153],{"class":110},[104,405,407,410,412],{"class":106,"line":406},22,[104,408,409],{"class":110},"  \u003C/",[104,411,162],{"class":114},[104,413,153],{"class":110},[104,415,417,420,422],{"class":106,"line":416},23,[104,418,419],{"class":110},"\u003C/",[104,421,115],{"class":114},[104,423,153],{"class":110},[17,425,426],{},"To get an idea, here are increasingly more aggressive downsampling results.",[17,428,429],{},[430,431,432,433,443],"strong",{},"Δ",[434,435,436],"sup",{},[41,437,442],{"href":438,"ariaDescribedBy":439,"dataFootnoteRef":99,"id":441},"#user-content-fn-1",[440],"footnote-label","user-content-fnref-1","1"," 68%:",[94,445,447],{"className":96,"code":446,"language":98,"meta":99,"style":99},"\u003Csection class=\"container\" required=\"true\" type=\"example\">\n  \u003Cdiv class=\"mx-auto\" required=\"false\">\n    # Our Pizza\n    \u003Cdiv>\n      ## Margherita\n      A simple classic: mozzarella, tomatoes and basil.\n      An everyday choice!\n      \u003Cbutton type=\"button\">Add\u003C/button>\n      ## Capricciosa\n      A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n      A true favourite!\n      \u003Cbutton type=\"button\">Add\u003C/button>\n    \u003C/div>\n  \u003C/div>\n\u003C/section>\n",[101,448,449,475,495,500,508,513,518,523,541,546,551,556,574,582,590],{"__ignoreMap":99},[104,450,451,453,455,457,459,461,463,465,467,469,471,473],{"class":106,"line":107},[104,452,111],{"class":110},[104,454,115],{"class":114},[104,456,119],{"class":118},[104,458,122],{"class":110},[104,460,126],{"class":125},[104,462,137],{"class":118},[104,464,122],{"class":110},[104,466,142],{"class":125},[104,468,145],{"class":118},[104,470,122],{"class":110},[104,472,150],{"class":125},[104,474,153],{"class":110},[104,476,477,479,481,483,485,487,489,491,493],{"class":106,"line":156},[104,478,159],{"class":110},[104,480,162],{"class":114},[104,482,119],{"class":118},[104,484,122],{"class":110},[104,486,169],{"class":125},[104,488,137],{"class":118},[104,490,122],{"class":110},[104,492,184],{"class":125},[104,494,153],{"class":110},[104,496,497],{"class":106,"line":189},[104,498,499],{"class":110},"    # Our Pizza\n",[104,501,502,504,506],{"class":106,"line":205},[104,503,192],{"class":110},[104,505,162],{"class":114},[104,507,153],{"class":110},[104,509,510],{"class":106,"line":214},[104,511,512],{"class":110},"      ## Margherita\n",[104,514,515],{"class":106,"line":231},[104,516,517],{"class":110},"      A simple classic: mozzarella, tomatoes and basil.\n",[104,519,520],{"class":106,"line":246},[104,521,522],{"class":110},"      An everyday choice!\n",[104,524,525,527,529,531,533,535,537,539],{"class":106,"line":255},[104,526,217],{"class":110},[104,528,282],{"class":114},[104,530,145],{"class":118},[104,532,122],{"class":110},[104,534,289],{"class":125},[104,536,292],{"class":110},[104,538,282],{"class":114},[104,540,153],{"class":110},[104,542,543],{"class":106,"line":261},[104,544,545],{"class":110},"      ## Capricciosa\n",[104,547,548],{"class":106,"line":267},[104,549,550],{"class":110},"      A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[104,552,553],{"class":106,"line":277},[104,554,555],{"class":110},"      A true favourite!\n",[104,557,558,560,562,564,566,568,570,572],{"class":106,"line":299},[104,559,217],{"class":110},[104,561,282],{"class":114},[104,563,145],{"class":118},[104,565,122],{"class":110},[104,567,289],{"class":125},[104,569,292],{"class":110},[104,571,282],{"class":114},[104,573,153],{"class":110},[104,575,576,578,580],{"class":106,"line":309},[104,577,399],{"class":110},[104,579,162],{"class":114},[104,581,153],{"class":110},[104,583,584,586,588],{"class":106,"line":324},[104,585,409],{"class":110},[104,587,162],{"class":114},[104,589,153],{"class":110},[104,591,592,594,596],{"class":106,"line":338},[104,593,419],{"class":110},[104,595,115],{"class":114},[104,597,153],{"class":110},[17,599,600],{},[430,601,602],{},"Δ 35%:",[94,604,606],{"className":96,"code":605,"language":98,"meta":99,"style":99},"# Our Pizza\n## Margherita\nA simple classic: mozzarella, tomatoes and basil.\nAn everyday choice!\n\u003Cbutton>Add\u003C/button>\n## Capricciosa\nA rich taste: mozzarella, ham, mushrooms, artichokes and olives.\nA true favourite!\n\u003Cbutton>Add\u003C/button>\n",[101,607,608,613,618,623,628,640,645,650,655],{"__ignoreMap":99},[104,609,610],{"class":106,"line":107},[104,611,612],{"class":110},"# Our Pizza\n",[104,614,615],{"class":106,"line":156},[104,616,617],{"class":110},"## Margherita\n",[104,619,620],{"class":106,"line":189},[104,621,622],{"class":110},"A simple classic: mozzarella, tomatoes and basil.\n",[104,624,625],{"class":106,"line":205},[104,626,627],{"class":110},"An everyday choice!\n",[104,629,630,632,634,636,638],{"class":106,"line":214},[104,631,111],{"class":110},[104,633,282],{"class":114},[104,635,292],{"class":110},[104,637,282],{"class":114},[104,639,153],{"class":110},[104,641,642],{"class":106,"line":231},[104,643,644],{"class":110},"## Capricciosa\n",[104,646,647],{"class":106,"line":246},[104,648,649],{"class":110},"A rich taste: mozzarella, ham, mushrooms, artichokes and olives.\n",[104,651,652],{"class":106,"line":255},[104,653,654],{"class":110},"A true favourite!\n",[104,656,657,659,661,663,665],{"class":106,"line":261},[104,658,111],{"class":110},[104,660,282],{"class":114},[104,662,292],{"class":110},[104,664,282],{"class":114},[104,666,153],{"class":110},[86,668,670],{"id":669},"d2snap-reliably-fits-within-the-models-context","D2Snap Reliably Fits Within the Model's Context",[17,672,673,674,681,682,689],{},"Every D2Snap-downsampled DOM snapshot produced by our reference configuration (",[675,676,70,677],"b",{},[678,679,680],"sub",{},".9, .3, .6",") from our evaluation dataset fits within GPT-4o's context window. Our dataset is sampled from ",[41,683,686],{"href":684,"rel":685},"https://arxiv.org/abs/2504.01382",[45],[21,687,688],{},"Online-Mind2Web"," (includes, e.g., nba.com).",[11,691],{"alt":692,"format":14,"src":693,"loading":36},"D2Snap evaluation results showing that all downsampled DOM snapshots fit within GPT-4o's context window","/rnd/0.dom-downsampling-for-llm-based-web-agents/3.png",[86,695,697],{"id":696},"d2snap-does-not-compromise-snapshot-utility","D2Snap Does Not Compromise Snapshot Utility",[17,699,700,701,706],{},"Used with a minimal web agent, D2Snap-downsampled DOM snapshots of our reference configuration attain a per-web-page task success rate (73%, excluding a deficit (one-sided 95%) beyond 11%pt, n = 52) that is not inferior to the success rate achieved with baseline grounded GUI snapshots (67%), which follow ",[41,702,705],{"href":703,"rel":704},"https://github.com/browser-use/browser-use",[45],"Browser Use","'s Set-of-Mark approach. The mean snapshot token count is at only 16.5% of the model's context size, leaving headroom for a prompt history. The per-web-page success rate is computed over LLM-suggested action target sets compared to annotated, coherent ground-truth action target sets in the snapshot dataset reference.",[11,708],{"alt":709,"format":14,"src":710,"loading":36},"An instance of a per-task snapshot trajectory with ground-truth target annotations","/rnd/0.dom-downsampling-for-llm-based-web-agents/4.png",[11,712],{"alt":713,"format":14,"src":714,"loading":36},"D2Snap evaluation results showing agent success rates with snapshots produced by different parametric configurations compared to baseline snapshot representations","/rnd/0.dom-downsampling-for-llm-based-web-agents/5.png",[86,716,718],{"id":717},"html-conveys-ui-feature-salience","HTML Conveys UI Feature Salience",[17,720,721],{},"We isolate downsampling ratios across redundant nodes of three types: elements, attributes, and text. Preserving HTML in a snapshot appears to have a positive impact on the snapshot utility: D2Snap snapshots attain the highest success rates with low retention of element-inherent hierarchy, high retention of semantic attributes, and moderate retention of text. Fully flattening the DOM to text – virtually Markdown with inlined actionable elements – attains a significantly lower success rate (46%).",[86,723,725],{"id":724},"images-appear-to-add-little-to-snapshot-utility","Images Appear to Add Little to Snapshot Utility",[17,727,728],{},"The agent's success rate with baseline grounding text as a standalone snapshot (62%), i.e., without screenshots, is close to the rate achieved with the baseline snapshot representation (67%), which suggests that image input adds little to snapshot utility. This insight challenges the use of image input, as image input imposes additional costs, including latency overhead.",[55,730,732],{"id":731},"dom-downsampling-implementation-and-interface","DOM Downsampling – Implementation and Interface",[17,734,735,736,741,742,747,748,751],{},"The ",[41,737,740],{"href":738,"rel":739},"https://dev.webfuse.com/automation-api/#see-perception-scope",[45],"Webfuse Automation Perception API"," provides a ",[41,743,746],{"href":744,"rel":745},"https://dev.webfuse.com/automation-api/#seedomsnapshot",[45],"DOM snapshot method",". We enhanced it with a ",[101,749,750],{},"quality"," parameter, behind which we implemented D2Snap-based DOM downsampling. Quality (fidelity) is the inverse of downsampling: the lower the quality, the lower the size of the DOM snapshot – at the cost of minor DOM features.",[94,753,757],{"className":754,"code":755,"language":756,"meta":99,"style":99},"language-ts shiki shiki-themes github-dark github-dark","await browser.webfuseSession\n  .automation\n  .see\n  .domSnapshot({\n    quality: 0.3,\n    webfuseIDs: true\n  });\n","ts",[101,758,759,768,773,778,789,801,809],{"__ignoreMap":99},[104,760,761,765],{"class":106,"line":107},[104,762,764],{"class":763},"sOPea","await",[104,766,767],{"class":110}," browser.webfuseSession\n",[104,769,770],{"class":106,"line":156},[104,771,772],{"class":110},"  .automation\n",[104,774,775],{"class":106,"line":189},[104,776,777],{"class":110},"  .see\n",[104,779,780,783,786],{"class":106,"line":205},[104,781,782],{"class":110},"  .",[104,784,785],{"class":118},"domSnapshot",[104,787,788],{"class":110},"({\n",[104,790,791,794,798],{"class":106,"line":214},[104,792,793],{"class":110},"    quality: ",[104,795,797],{"class":796},"s8ozJ","0.3",[104,799,800],{"class":110},",\n",[104,802,803,806],{"class":106,"line":231},[104,804,805],{"class":110},"    webfuseIDs: ",[104,807,808],{"class":796},"true\n",[104,810,811],{"class":106,"line":246},[104,812,813],{"class":110},"  });\n",[17,815,816],{},"Alternatively, quality can be adjusted adaptively given an estimated LLM input token limit:",[94,818,820],{"className":754,"code":819,"language":756,"meta":99,"style":99},"await browser.webfuseSession\n  .automation\n  .see\n  .domSnapshot({\n    maxTokens: 2**15,\n    webfuseIDs: true\n  });\n",[101,821,822,828,832,836,844,860,866],{"__ignoreMap":99},[104,823,824,826],{"class":106,"line":107},[104,825,764],{"class":763},[104,827,767],{"class":110},[104,829,830],{"class":106,"line":156},[104,831,772],{"class":110},[104,833,834],{"class":106,"line":189},[104,835,777],{"class":110},[104,837,838,840,842],{"class":106,"line":205},[104,839,782],{"class":110},[104,841,785],{"class":118},[104,843,788],{"class":110},[104,845,846,849,852,855,858],{"class":106,"line":214},[104,847,848],{"class":110},"    maxTokens: ",[104,850,851],{"class":796},"2",[104,853,854],{"class":763},"**",[104,856,857],{"class":796},"15",[104,859,800],{"class":110},[104,861,862,864],{"class":106,"line":231},[104,863,805],{"class":110},[104,865,808],{"class":796},[104,867,868],{"class":106,"line":246},[104,869,813],{"class":110},[17,871,872,873,878],{},"Obviously, CSS selectors between the downsampled DOM and its original, the live DOM, might be different. For that reason, we introduced ",[41,874,877],{"href":875,"rel":876},"https://dev.webfuse.com/automation-api/#webfuse-ids-",[45],"Webfuse IDs",": snapshot-exclusive attributes that associate every actionable element with a unique ID. This ID can later be used with Webfuse's Automation Actuation API for targeting an element in the live DOM.",[115,880,883,888],{"className":881,"dataFootnotes":99},[882],"footnotes",[55,884,887],{"className":885,"id":440},[886],"sr-only","Footnotes",[889,890,891],"ol",{},[892,893,895,896],"li",{"id":894},"user-content-fn-1","Δ denotes the size fraction compared to the original DOM (across HTML serialisations). ",[41,897,902],{"href":898,"ariaLabel":899,"className":900,"dataFootnoteBackref":99},"#user-content-fnref-1","Back to reference 1",[901],"data-footnote-backref","↩",[904,905,906],"style",{},"html pre.shiki code .suv1-, html code.shiki .suv1-{--shiki-default:#E1E4E8;--shiki-dark:#E1E4E8}html pre.shiki code .sxg3X, html code.shiki .sxg3X{--shiki-default:#85E89D;--shiki-dark:#85E89D}html pre.shiki code .sFR8T, html code.shiki .sFR8T{--shiki-default:#B392F0;--shiki-dark:#B392F0}html pre.shiki code .s4wv1, html code.shiki .s4wv1{--shiki-default:#9ECBFF;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sOPea, html code.shiki .sOPea{--shiki-default:#F97583;--shiki-dark:#F97583}html pre.shiki code .s8ozJ, html code.shiki .s8ozJ{--shiki-default:#79B8FF;--shiki-dark:#79B8FF}",{"title":99,"searchDepth":156,"depth":156,"links":908},[909,916,917],{"id":57,"depth":156,"text":58,"children":910},[911,912,913,914,915],{"id":88,"depth":189,"text":89},{"id":669,"depth":189,"text":670},{"id":696,"depth":189,"text":697},{"id":717,"depth":189,"text":718},{"id":724,"depth":189,"text":725},{"id":731,"depth":156,"text":732},{"id":440,"depth":156,"text":887},"2025-08-18","We introduce DOM downsampling, a size-reducing technique designed for pre-processing DOM snapshots for use with LLM-based web agents.","md",null,{},true,"The advent of large language models (LLMs) has sparked an evolution of autonomous web browsing agents: given a web browsing task and serialised user interface (UI) state, an LLM is expected to suggest input actions that incrementally solve the given task. The central challenge lies in serialising UI state for LLMs. Web agents have increasingly relied on grounded graphical UI (GUI) snapshots - screenshots augmented with visual cues - favoured for their modest input token footprint. Document Object Model (DOM) snapshots, serialised as HTML, represent a compelling alternative that leverages previously demonstrated HTML interpretation capabilities of LLMs. Their excessive token footprint, however, has precluded reliable deployment with web agents to date. We propose D2Snap, an algorithm to downsample the DOM, premised on preserving actionability and actionability-discriminating features. We evaluate D2Snap-downsampled DOM snapshots using a snapshot-variant web agent (GPT-4o) on a dataset sampled from Online-Mind2Web. Whilst 42% of raw DOM snapshots exceed the model context window (128 x 10^3 tokens), all D2Snap-downsampled DOM snapshots of our reference configuration fit, at a mean context utilisation of 16.5%. Against the 67% success rate of a grounded GUI snapshot baseline, our configuration attains 73% (+5.8%pt; 95% CI -13.6 to +26.0%pt; McNemar, p = 0.47), excluding a deficit (one-sided 95%) beyond 11%pt. Image input moreover appears to add little to snapshot utility; grounding text alone attains 62% (-5.8%pt; McNemar, p = 0.37).",[926],"Thassilo M. Schiepanski",[928,929],"Nicholas Piël","Yauhen Shulitski","2025-08-06","https://arxiv.org/abs/2508.04412","Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents","/rnd/dom-downsampling-for-llm-based-web-agents",{"title":5,"description":919},{"loc":933},"rnd/0.dom-downsampling-for-llm-based-web-agents","75o2CU-FTQMm2mZCQjeXSJhIpK4Gw5IU-Ciqj6UivWw",1787685714735]