# Web Data

GET /lookup/webdata: Loads a web page in a real browser and returns everything Deepinfo extracts from it, in one call: detected technologies, page metadata…

Source: https://docs.deepinfo.com/reference/lookup/web-data/

Last updated: 2026-09-27

---
`GET https://api.deepinfo.com/v1/lookup/webdata`

Loads a web page in a real browser and returns everything Deepinfo extracts from it, in one call: detected technologies, page metadata (title, description, Open Graph tags, JSON-LD), the HTML source and visible text, links, scripts, trackers (Google Analytics, AdSense and Tag Manager IDs), e-mail addresses, favicons, `robots.txt`, and the HTTP response headers, cookies and redirects. Add `screenshot=true` to capture a screenshot as well.

Use it to profile a website in depth, for example to compare a look-alike domain's page with your own or to find the analytics IDs it shares with other sites. For technologies only, [Technology](/reference/lookup/technology/) is lighter; for an image only, use [Screenshot](/reference/lookup/screenshot/).

A request takes several seconds (about 10 s in our tests).

## Authentication

Send your API key in the `apikey` request header.

## Query Parameters

| Parameter | Required | Description | Example |
|---|---|---|---|
| `url` | Required | Web page to analyze, e.g. `https://www.deepinfo.com`. | `https://www.deepinfo.com` |
| `screenshot` | Optional | Also capture a JPEG screenshot of the rendered page. The response then includes a `screenshot` object. Default `false`. | `true` |
| `wait_until` | Optional | Page load events to wait for before the page is parsed, comma-separated. Ignored when `screenshot` is `true`: the page is then always parsed after `load`. One or more of `load`, `domcontentloaded`, `networkidle0`, `networkidle2`. Default `networkidle2,load,domcontentloaded`. | `networkidle2,load,domcontentloaded` |
| `max_file_size` | Optional | Maximum size of `html.source_code`, in bytes. For a larger page, `source_code` holds a placeholder message instead of the HTML. Range `1`–`33554432`. Default `8388608`. | `8388608` |
| `proxy` | Optional | Proxy to load the page through, as a URL with an explicit port, e.g. `http://user:password@host:8080`. The proxy host must be a public IP address or an allowed domain. |  |

## Response Fields

| Field | Description |
|---|---|
| `url` | The requested URL |
| `technology.stacks[]` | One entry per detected technology |
| `technology.stacks[].slug` | Identifier of the technology |
| `technology.stacks[].name` | Name of the technology |
| `technology.stacks[].description` | What the technology is |
| `technology.stacks[].categories` | Categories it belongs to |
| `technology.stacks[].confidence` | Detection confidence (0–100) |
| `technology.stacks[].version` | Detected version; empty when the site does not reveal it |
| `technology.stacks[].clean_version` | The version as a number, or `null` |
| `technology.stacks[].cpe` | CPE name |
| `technology.stacks[].alternative_cpe_names` | Other CPE names of the technology |
| `technology.stacks[].website` | The vendor's website |
| `technology.stacks[].icon` | Icon file name |
| `html.meta.title` | Page title |
| `html.meta.name` | Site name given in the page metadata |
| `html.meta.description` | Meta description |
| `html.meta.keywords` | Meta keywords |
| `html.meta.language` | Language of the page |
| `html.meta.language_alternatives` | Other language versions the page declares |
| `html.meta.encoding` | Character encoding |
| `html.meta.noindex_status` | Whether the page asks search engines not to index it |
| `html.meta.canonical_url` | Canonical URL |
| `html.meta.og[]` | Open Graph and Twitter card tags, one entry per tag |
| `html.meta.og[].name` | Tag name |
| `html.meta.og[].value` | Tag value |
| `html.meta.json_ld` | Structured data blocks, in expanded JSON-LD form |
| `html.source_code` | The page HTML. For a page larger than `max_file_size`, a placeholder message instead of the HTML |
| `html.source_code_hash` | SHA-256 hash of `html.source_code` |
| `html.content` | The visible text of the page |
| `html.content_hash` | SHA-256 hash of `html.content` |
| `html.content_keywords` | The most frequent words of `html.content` |
| `html.internal_links_fqdns` | Host names on the site's own domain that the page links to |
| `html.external_links` | Links to other sites |
| `html.external_links_fqdns` | Host names of those links |
| `html.external_links_domains` | Registered domains of those links |
| `html.script_links` | Scripts the page loads |
| `html.iframe_links` | Iframes the page loads |
| `html.favicon_links` | Icons the page links to |
| `html.trackers[]` | Tracking IDs found in the page, one entry per tracker (e.g. Google Analytics) |
| `html.trackers[].name` | Tracker name |
| `html.trackers[].values` | IDs found for that tracker |
| `html.emails` | E-mail addresses found in the page |
| `html.emails_internal` | E-mail addresses found in the page that are on the site's own domain |
| `html.inspect_disabled` | Whether the page tries to block inspection of its content |
| `favicon[]` | One entry per icon |
| `favicon[].url` | Icon URL |
| `favicon[].hash` | Hash of the icon |
| `robots_txt.content` | Content of the site's `robots.txt` |
| `robots_txt.hash` | Hash of `robots_txt.content` |
| `robots_txt.disallowed_links` | The `Disallow` paths |
| `http.headers[]` | HTTP response headers, one entry per header |
| `http.headers[].name` | Header name |
| `http.headers[].value` | Header value |
| `http.cookies[]` | Cookies the page set, one entry per cookie |
| `http.cookies[].name` | Cookie name |
| `http.cookies[].value` | Cookie value |
| `http.cookies[].domain` | Domain the cookie applies to |
| `http.cookies[].path` | Path the cookie applies to |
| `http.cookies[].expires` | Expiry time |
| `http.cookies[].secure` | Whether the cookie is sent over HTTPS only |
| `http.cookies[].http_only` | Whether the cookie is hidden from JavaScript |
| `http.cookies[].same_site` | The cookie's `SameSite` attribute |
| `http.cookies[].same_party` | The cookie's `SameParty` attribute |
| `http.cookies[].priority` | The cookie's `Priority` attribute |
| `http.cookies[].size` | Size of the cookie, in bytes |
| `http.cookies[].session` | Whether it is a session cookie |
| `http.redirection_history[]` | Each URL on the way to the final page |
| `http.redirection_history[].url` | The URL |
| `http.redirection_history[].status_code` | The HTTP status it returned |
| `screenshot` | Only with `screenshot=true`: the capture, with the `screenshot.*` fields below |
| `screenshot.url` | The requested URL |
| `screenshot.redirected_url` | The URL the browser ended up on after redirects |
| `screenshot.screenshot_url` | Link to the JPEG image: a signed link that expires after 7 days. `null` if the capture was skipped |
| `screenshot.connection_status` | `success` when the page was loaded |
| `screenshot.check_date` | When the screenshot was taken (UTC) |
| `connection_status` | `success` when the page was loaded |
| `version` | Version of the result format (currently `1`) |
| `check_date` | When the lookup was performed (UTC) |

## Response Schema

_Inferred from examples._ Built from the 2 saved 2xx example responses: the fields they contain, with the types seen there. It is not a contract.

| Field | Type |
|---|---|
| `html` | object |
| `html.meta` | object |
| `html.meta.name` | string |
| `html.meta.description` | string |
| `html.meta.language` | string |
| `html.meta.language_alternatives` | array |
| `html.meta.keywords` | array |
| `html.meta.noindex_status` | boolean |
| `html.meta.encoding` | string |
| `html.meta.canonical_url` | string |
| `html.meta.title` | string |
| `html.meta.json_ld` | array<object> |
| `html.meta.json_ld[].@id` | string |
| `html.meta.json_ld[].@type` | array<string> |
| `html.meta.json_ld[].*` | array<object> |
| `html.meta.json_ld[].*[].@value` | string |
| `html.meta.og` | array<object> |
| `html.meta.og[].name` | string |
| `html.meta.og[].value` | string |
| `html.internal_links_fqdns` | array<string> |
| `html.external_links_fqdns` | array<string> |
| `html.external_links_domains` | array<string> |
| `html.external_links` | array<string> |
| `html.script_links` | array<string> |
| `html.iframe_links` | array |
| `html.trackers` | array |
| `html.emails` | array |
| `html.emails_internal` | array |
| `html.favicon_links` | array<string> |
| `html.inspect_disabled` | boolean |
| `html.source_code` | string |
| `html.source_code_hash` | string |
| `html.content` | string |
| `html.content_keywords` | array<string> |
| `html.content_hash` | string |
| `http` | object |
| `http.redirection_history` | array<object> |
| `http.redirection_history[].url` | string |
| `http.redirection_history[].status_code` | number |
| `http.cookies` | array<object> |
| `http.cookies[].name` | string |
| `http.cookies[].value` | string |
| `http.cookies[].domain` | string |
| `http.cookies[].path` | string |
| `http.cookies[].same_party` | string |
| `http.cookies[].priority` | null |
| `http.cookies[].size` | number |
| `http.cookies[].expires` | string |
| `http.cookies[].secure` | boolean |
| `http.cookies[].http_only` | boolean |
| `http.cookies[].same_site` | string |
| `http.cookies[].session` | boolean |
| `http.headers` | array<object> |
| `http.headers[].name` | string |
| `http.headers[].value` | string |
| `url` | string |
| `favicon` | array<object> |
| `favicon[].url` | string |
| `favicon[].hash` | string |
| `robots_txt` | object |
| `robots_txt.content` | string |
| `robots_txt.hash` | string |
| `robots_txt.disallowed_links` | array<string> |
| `version` | number |
| `check_date` | string |
| `connection_status` | string |
| `technology` | object |
| `technology.stacks` | array<object> |
| `technology.stacks[].slug` | string |
| `technology.stacks[].name` | string |
| `technology.stacks[].confidence` | number |
| `technology.stacks[].icon` | string |
| `technology.stacks[].website` | string |
| `technology.stacks[].cpe` | string |
| `technology.stacks[].version` | string |
| `technology.stacks[].categories` | array<string> |
| `technology.stacks[].description` | string |
| `technology.stacks[].alternative_cpe_names` | array |
| `technology.stacks[].clean_version` | null |
| `screenshot` | object (in 1 of 2) |
| `screenshot.screenshot_url` | string (in 1 of 2) |
| `screenshot.check_date` | string (in 1 of 2) |
| `screenshot.connection_status` | string (in 1 of 2) |
| `screenshot.redirected_url` | string (in 1 of 2) |
| `screenshot.url` | string (in 1 of 2) |

## Errors

`400` (`10400`) if `url` is missing or invalid, or a parameter is out of range. The validation error currently names the parameter `domain`, although the request parameter is `url` (see the example). A `400` without `parameters` (only `details`, e.g. *"Target is not allowed."*) means the target cannot be analyzed. `500` for an unexpected error. `503` means the service or its browser is busy: retry after the number of seconds in the `Retry-After` header. See [Getting Started → Errors](/getting-started/errors/).

## Examples

### 200 · www.deepinfo.com

```bash
curl 'https://api.deepinfo.com/v1/lookup/webdata?url=https%3A%2F%2Fwww.deepinfo.com' \
  -H 'apikey: YOUR_API_KEY' \
  -H 'Accept: application/json'
```

`Content-Type: application/json` · `ratelimit-limit: 1` · `ratelimit-remaining: 0` · `ratelimit-reset: 1` · `deepinfo-request-id: 5f0c6a8e-1b2d-4c3e-9f4a-7b8c9d0e1f2a`

```json
{
  "html": {
    "meta": {
      "name": "Deepinfo",
      "description": "Deepinfo is a Continuous Threat Exposure Management platform. Discover every asset, monitor every exposure, and act before attackers do, all from one place.",
      "language": "en",
      "language_alternatives": [],
      "keywords": [],
      "noindex_status": false,
      "encoding": "utf-8",
      "canonical_url": "https://www.deepinfo.com/",
      "title": "Deepinfo | Continuous Threat Exposure Management Platform",
      "json_ld": [
        {
          "@id": "https://www.deepinfo.com/#organization",
          "@type": [
            "http://schema.org/Organization"
          ],
          "http://schema.org/name": [
            {
              "@value": "Deepinfo"
            }
          ],
          "http://schema.org/url": [
            {
              "@value": "https://www.deepinfo.com/"
            }
          ]
        }
      ],
      "og": [
        {
          "name": "og:site_name",
          "value": "Deepinfo"
        },
        {
          "name": "og:type",
          "value": "website"
        },
        {
          "name": "og:title",
          "value": "Deepinfo | Continuous Threat Exposure Management Platform"
        }
      ]
    },
    "internal_links_fqdns": [
      "www.deepinfo.com",
      "docs.deepinfo.com",
      "platform.deepinfo.com"
    ],
    "external_links_fqdns": [
      "linkedin.com"
    ],
    "external_links_domains": [
      "linkedin.com"
    ],
    "external_links": [
      "https://linkedin.com/company/deepinfo"
    ],
    "script_links": [
      "https://www.deepinfo.com//static/js/site.js?v=60464885",
      "https://www.deepinfo.com//static/js/blur-text.js?v=100"
    ],
    "iframe_links": [],
    "trackers": [],
    "emails": [],
    "emails_internal": [],
    "favicon_links": [
      "https://www.deepinfo.com//static/img/favicon.svg"
    ],
    "inspect_disabled": false,
    "source_code": "<!DOCTYPE html><html lang=\"en\"><head>\n<meta charset=\"utf-8\">\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n    <meta name=\"robots\" conten…",
    "source_code_hash": "ad93f88fec2753e10d83cad96f866788c00f869001f290bc1952828c44e92e1c",
    "content": "Skip to main content\nPlatform \nPLATFORM\nContinuous Threat Exposure Management\nPlatform Overview How the Platform Works Methodology Integrations Mobile App\nEASM\n…",
    "content_keywords": [
      "data",
      "management",
      "platform",
      "see",
      "threat"
    ],
    "content_hash": "14ab6575f7a485db17770369046e7516de0d1b6bb810911a4554930412106cc0"
  },
  "http": {
    "redirection_history": [
      {
        "url": "https://www.deepinfo.com",
        "status_code": 200
      }
    ],
    "cookies": [
      {
        "name": "intercom-device-id-xki6sc8r",
        "value": "<cookie value>",
        "domain": ".www.deepinfo.com",
        "path": "/",
        "same_party": "Lax",
        "priority": null,
        "size": 63,
        "expires": "2027-06-20T15:18:37.000Z",
        "secure": false,
        "http_only": false,
        "same_site": "Lax",
        "session": false
      }
    ],
    "headers": [
      {
        "name": "content-type",
        "value": "text/html; charset=utf-8"
      },
      {
        "name": "server",
        "value": "cloudflare"
      },
      {
        "name": "strict-transport-security",
        "value": "max-age=31536000; includeSubDomains; preload"
      },
      {
        "name": "x-frame-options",
        "value": "DENY"
      }
    ]
  },
  "url": "https://www.deepinfo.com",
  "favicon": [
    {
      "url": "https://www.deepinfo.com//static/img/favicon.svg",
      "hash": "b742b1a1b669ce7b299449865e7cf766abd9a9e205ff3e88af9f54b9dfdc3b8b"
    }
  ],
  "robots_txt": {
    "content": "# AI training crawlers — explicit allow\nUser-agent: GPTBot\nAllow: /\n…",
    "hash": "8a12283023f4ae29941dca8de0633eb44d95175ce3123f7b89573bdbeeb1eabb",
    "disallowed_links": [
      "/404.html",
      "/api/"
    ]
  },
  "version": 1,
  "check_date": "2026-09-23T14:45:19.751Z",
  "connection_status": "success",
  "technology": {
    "stacks": [
      {
        "slug": "cloudflare",
        "name": "Cloudflare",
        "confidence": 100,
        "icon": "CloudFlare.svg",
        "website": "http://www.cloudflare.com",
        "cpe": "cpe:/a:deepvendor:cloudflare",
        "version": "",
        "categories": [
          "CDN"
        ],
        "description": "Cloudflare is a web-infrastructure and website-security company, providing content-delivery-network services, DDoS mitigation, Internet security, and distributed domain-name-server services.",
        "alternative_cpe_names": [],
        "clean_version": null
      },
      {
        "slug": "google-font-api",
        "name": "Google Font API",
        "confidence": 100,
        "icon": "Google Font API.png",
        "website": "http://google.com/fonts",
        "cpe": "cpe:/a:deepvendor:google_font_api",
        "version": "",
        "categories": [
          "Font scripts"
        ],
        "description": "Google Font API is a web service that supports open-source font files that can be used on your web designs.",
        "alternative_cpe_names": [],
        "clean_version": null
      }
    ]
  }
}
```

### 200 · With Screenshot

```bash
curl 'https://api.deepinfo.com/v1/lookup/webdata?url=https%3A%2F%2Fwww.deepinfo.com&screenshot=true' \
  -H 'apikey: YOUR_API_KEY' \
  -H 'Accept: application/json'
```

`Content-Type: application/json` · `ratelimit-limit: 1` · `ratelimit-remaining: 0` · `ratelimit-reset: 1` · `deepinfo-request-id: 5f0c6a8e-1b2d-4c3e-9f4a-7b8c9d0e1f2a`

```json
{
  "html": {
    "meta": {
      "name": "Deepinfo",
      "description": "Deepinfo is a Continuous Threat Exposure Management platform. Discover every asset, monitor every exposure, and act before attackers do, all from one place.",
      "language": "en",
      "language_alternatives": [],
      "keywords": [],
      "noindex_status": false,
      "encoding": "utf-8",
      "canonical_url": "https://www.deepinfo.com/",
      "title": "Deepinfo | Continuous Threat Exposure Management Platform",
      "json_ld": [
        {
          "@id": "https://www.deepinfo.com/#organization",
          "@type": [
            "http://schema.org/Organization"
          ],
          "http://schema.org/name": [
            {
              "@value": "Deepinfo"
            }
          ],
          "http://schema.org/url": [
            {
              "@value": "https://www.deepinfo.com/"
            }
          ]
        }
      ],
      "og": [
        {
          "name": "og:site_name",
          "value": "Deepinfo"
        },
        {
          "name": "og:type",
          "value": "website"
        },
        {
          "name": "og:title",
          "value": "Deepinfo | Continuous Threat Exposure Management Platform"
        }
      ]
    },
    "internal_links_fqdns": [
      "www.deepinfo.com",
      "docs.deepinfo.com",
      "platform.deepinfo.com"
    ],
    "external_links_fqdns": [
      "linkedin.com"
    ],
    "external_links_domains": [
      "linkedin.com"
    ],
    "external_links": [
      "https://linkedin.com/company/deepinfo"
    ],
    "script_links": [
      "https://www.deepinfo.com//static/js/site.js?v=60464885",
      "https://www.deepinfo.com//static/js/blur-text.js?v=100"
    ],
    "iframe_links": [],
    "trackers": [],
    "emails": [],
    "emails_internal": [],
    "favicon_links": [
      "https://www.deepinfo.com//static/img/favicon.svg"
    ],
    "inspect_disabled": false,
    "source_code": "<!DOCTYPE html><html lang=\"en\"><head>\n<meta charset=\"utf-8\">\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n    <meta name=\"robots\" conten…",
    "source_code_hash": "412dba8743c4b3646af69f214d722e4e60d764e77d0f6643b08281e2b778a83a",
    "content": "Skip to main content\nPLATFORM\nPlatform Overview\nHow the Platform Works\nMethodology\nIntegrations\nMobile App\nEASM\nCTI\nBRP\nTPRM\nDSI\nDATA & API\nBrowse API docs\nData…",
    "content_keywords": [
      "data",
      "platform",
      "management",
      "deepinfo",
      "see"
    ],
    "content_hash": "b7099e35c209b835c57e4ed4377f2f1e5ec36b98bc579b3b1c3d4b65ab22cc85"
  },
  "http": {
    "redirection_history": [
      {
        "url": "https://www.deepinfo.com",
        "status_code": 200
      }
    ],
    "cookies": [
      {
        "name": "intercom-device-id-xki6sc8r",
        "value": "<cookie value>",
        "domain": ".www.deepinfo.com",
        "path": "/",
        "same_party": "Lax",
        "priority": null,
        "size": 63,
        "expires": "2027-06-20T15:18:49.000Z",
        "secure": false,
        "http_only": false,
        "same_site": "Lax",
        "session": false
      }
    ],
    "headers": [
      {
        "name": "content-type",
        "value": "text/html; charset=utf-8"
      },
      {
        "name": "server",
        "value": "cloudflare"
      },
      {
        "name": "strict-transport-security",
        "value": "max-age=31536000; includeSubDomains; preload"
      },
      {
        "name": "x-frame-options",
        "value": "DENY"
      }
    ]
  },
  "url": "https://www.deepinfo.com",
  "favicon": [
    {
      "url": "https://www.deepinfo.com//static/img/favicon.svg",
      "hash": "b742b1a1b669ce7b299449865e7cf766abd9a9e205ff3e88af9f54b9dfdc3b8b"
    }
  ],
  "robots_txt": {
    "content": "# AI training crawlers — explicit allow\nUser-agent: GPTBot\nAllow: /\n…",
    "hash": "8a12283023f4ae29941dca8de0633eb44d95175ce3123f7b89573bdbeeb1eabb",
    "disallowed_links": [
      "/404.html",
      "/api/"
    ]
  },
  "version": 1,
  "check_date": "2026-09-23T14:45:32.606Z",
  "connection_status": "success",
  "technology": {
    "stacks": [
      {
        "slug": "cloudflare",
        "name": "Cloudflare",
        "confidence": 100,
        "icon": "CloudFlare.svg",
        "website": "http://www.cloudflare.com",
        "cpe": "cpe:/a:deepvendor:cloudflare",
        "version": "",
        "categories": [
          "CDN"
        ],
        "description": "Cloudflare is a web-infrastructure and website-security company, providing content-delivery-network services, DDoS mitigation, Internet security, and distributed domain-name-server services.",
        "alternative_cpe_names": [],
        "clean_version": null
      },
      {
        "slug": "google-font-api",
        "name": "Google Font API",
        "confidence": 100,
        "icon": "Google Font API.png",
        "website": "http://google.com/fonts",
        "cpe": "cpe:/a:deepvendor:google_font_api",
        "version": "",
        "categories": [
          "Font scripts"
        ],
        "description": "Google Font API is a web service that supports open-source font files that can be used on your web designs.",
        "alternative_cpe_names": [],
        "clean_version": null
      }
    ]
  },
  "screenshot": {
    "screenshot_url": "https://diss-999.storage.googleapis.com/202609/23/GJjwF7PjcQ4xjkk2nXGMKyW9.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=<credential>&X-Amz-Date=20260923T144536Z&X-Amz-Expires=604800&X-Amz-Signature=<signature>&X-Amz-SignedHeaders=host&x-id=GetObject",
    "check_date": "2026-09-23T14:45:36.767Z",
    "connection_status": "success",
    "redirected_url": "https://www.deepinfo.com/",
    "url": "https://www.deepinfo.com"
  }
}
```

### 400 · Invalid URL

```bash
curl 'https://api.deepinfo.com/v1/lookup/webdata?url=not_a_url' \
  -H 'apikey: YOUR_API_KEY' \
  -H 'Accept: application/json'
```

`Content-Type: application/json` · `ratelimit-limit: 1` · `ratelimit-remaining: 0` · `ratelimit-reset: 1` · `deepinfo-request-id: 5f0c6a8e-1b2d-4c3e-9f4a-7b8c9d0e1f2a`

```json
{
  "code": 10400,
  "parameters": [
    {
      "param": "domain",
      "details": [
        "The given input is not a valid domain name."
      ],
      "subcode": 10000
    }
  ],
  "solution": "https://docs.deepinfo.com/reference/"
}
```

## Worked Examples

Worked examples of this endpoint, each on its own page with the exact request and its response: [Web Data Examples](/reference/lookup/web-data/examples/).

- [Everything About a Web Page](/reference/lookup/web-data/examples/default/): Everything about developer.mozilla.org in one call: technologies, meta tags, HTML, visible text, links, scripts, favicons, robots.txt and headers.
- [With a Screenshot](/reference/lookup/web-data/examples/screenshot/): Web data with screenshot=true: the same data plus a screenshot object with a signed link to a JPEG of the page.
- [Tracking IDs on a Page](/reference/lookup/web-data/examples/trackers/): Web data of www.mozilla.org: html.trackers finds a Google Tag Manager container and two Google Analytics IDs.