Website Research

Node type: research:website
Category: Research

Description

Wire a domain in and get back everything an LLM needs to write about that brand without visiting the site: the brand identity, the catalogue, every image with the page it came from, and the readable text of each page it crawled.

It calls no model, so it costs no credits. The nodes that interpret the dump (an ad-angle scout, a product picker) are ordinary LLM nodes you wire downstream.

Canvas ports

These appear as port handles on the left side of the node.

ID	Label	Type	Required
url	Website URL	TEXT	yes

example.com, www.example.com/collections/all and https://example.com/x?y=1 all work. The page you give is the seed; the crawl stays on its origin.

Sidebar config

These render as form fields in the right-side config panel when the node is selected.

ID	Label	Type	Default	Notes
max_pages	Max Pages	NUMBER	12	Hard cap 25. Not a port: it is a budget, not data.

The url port is the real input. When nothing is wired to it, the node card carries an inline URL field so you can type one and run without building a text node first; the field goes read-only as soon as the port is connected, so the card never shows a value the run will not use.

Outputs

ID	Label	Type
summary	Summary	TEXT
brand	Brand	JSON
products	Products	JSON
images	Images	JSON
pages	Pages	JSON

summary is markdown, and it is the port you wire into an LLM: brand line, platform, the page list with a short gist each, a product table, the image inventory, and a MISSING section naming everything the site did not publish.

brand is { name, domain, logo, ogImage, socials, palette, platform }. products is { title, price, url, images[], source }, where price is { amount, currency } or null and currency is filled in ONLY when the site published a code (a bare $ is at least four currencies, so it stays null). images is every image with { url, alt, page, width, height }.

source says how a product was found, and it is the difference between a catalogue and a sample:

source	What was read
products_json	Shopify's /products.json: real prices, real image dimensions.
page	The product page itself was fetched: JSON-LD, og:type=product, or a price on the page.
link	Only a LINK on a listing page. Title and image come from the tile; price is always null.

The summary's product table carries a Source column and a By source line above it, so a model reading the dump knows which rows it has a price for and which it does not.

What it does not do

• It never guesses. A field the site did not publish comes back null and gets a line in MISSING. A dump that quietly invents a brand name reads exactly like one that read a real og:site_name.
• It never downloads an image to measure it. Dimensions come from HTML attributes, JSON-LD or Shopify's products.json. Unknown dimensions stay null, never zero.
• It does not run JavaScript. A site that renders its catalogue client-side comes back with the pages it served plus every product it LINKS to (source: 'link'). Those rows carry no price and no product copy, MISSING says how many they are, and fetching their pages is a separate step.
• It does not sample colours from images. The palette is what the HTML itself declares: inline styles, theme-color, and CSS custom properties.

How it crawls

• Same origin only, breadth-first. On an ordinary site the order is: the seed page, then /products*, /collections*, /shop*, /pricing*, /about*, /faq*, then everything else. Under a small page budget that order is the result.
• On a storefront the order flips to the catalogue. When the seed page carries more than 20 product-shaped links, or a <nav> naming at least two aisles (/c/, /category/, /collections/, /shop, /women, /men), category pages go first, product pages second, and the corporate pages (/about, /faq, /blog, /terms, …) go last. A store that uses /c/ and /p/ instead of Shopify's paths otherwise ranks its whole catalogue level with its About page, and a 6-page budget never reaches a product.
• Product links are counted even when the catalogue is JavaScript. On every page it fetches, the crawl collects anchors whose path looks like a product (/p/, /product/, /products/, /dp/, /item/, /pd/, -p-12345, /prod12345, or a slug-12345 under a category path). On a storefront only, it also takes any anchor wrapping an image inside a container whose class or id is a whole product… or tile token; that rule is off everywhere else, because "an image inside something classed card" describes every blog index built with Tailwind or Bootstrap. Those come back with source: 'link', a title from the anchor text (price and "Quick view" chrome stripped), else the image's alt, else the de-slugged path, and the tile's image. They are capped at 200, deduped by path against each other and against the products that were actually read, and MISSING says how many of the total were link-only. This is what makes a client-rendered store report its catalogue instead of reporting zero.
• Shopify stores answer /products.json, which is the only place a store hands over real image dimensions without a download. It is read in pages of 250 up to 1000 products, and a store with more than that gets a TRUNCATED line in MISSING rather than being reported as a complete catalogue. The price quoted is the cheapest variant. A Webflow store 404s there, and the products are read off the rendered product pages instead.
• robots.txt is fetched once per run and its Disallow rules are honoured for WireflowBot and *. Skipped paths are listed in the summary.
• Per fetch: an 8 second budget that covers DNS, every redirect hop and the body read (a server that dribbles bytes forever is abandoned, not waited on), a 2 MB body ceiling, and at most 3 redirects with the host policy re-checked on every hop. Requests identify as WireflowBot/1.0 (+https://wireflow.ai).
• Private, loopback, link-local, CGNAT and internal-only hosts are refused two ways: on what the URL says, and on what the hostname resolves to. Both run on the seed and on every redirect. A public domain whose A record points at an internal address is refused, which matters here because the node hands the fetched body back to you. The resolved check fails closed: if the server cannot resolve names on a given code path, the node refuses a hostname rather than falling back to the literal check (a bare public IP still works, since there is no lookup to be missing).
• A seed that redirects is followed and its landing origin becomes the crawl origin, so example.com to www.example.com reads the whole site. A redirect to a different site is named in MISSING.

---

Auto-generated from the Wireflow node registry.