Skip to main content

Technical Overview

AdRetriever operates automated web crawlers (“Site Walk”) on behalf of advertisers who use the AdRetriever platform. A Site Walk collects public inventory listings from a customer’s website—properties, vehicles, products, or similar—and keeps that catalog in sync so dynamic ads can stay accurate.

Site Walk is not a general-purpose search indexer. Each customer site is crawled by a dedicated spider that is scoped to that site’s domains and listing URLs. Crawls run on a schedule the customer configures in AdRetriever, or on demand when someone triggers a manual import.

Categories of AdRetriever HTTP requests​

CategoryPurpose
Site Walk (inventory spiders)Scheduled or manual import of listing pages and related assets for a configured website
Image URL validationHEAD/GET checks on image URLs supplied during import

What Site Walk does​

For a configured website, the spider typically:

  1. Starts from known entry URLs for that site (inventory list pages, sitemap-like paths, or a partner API).
  2. Follows links only within allowed domains for that spider.
  3. Fetches HTML
  4. Extracts structured listing data, such as product name, price, detail page and image URLs.

Site Walk is intended to read the same public pages a visitor would use to browse inventory.

How often Site Walk runs​

Frequency is per website, set by the AdRetriever customer. The default update frequency is once per day.

Load during a Site Walk: By default, Site Walk is throttled to about 1 concurrent request per remote server or domain, and adds a 2-second delay between requests.

Identifying AdRetriever Site Walk​

User-Agent header​

User-Agent: AdRetriever (https://adretriever.com)

Reverse DNS​

All Site Walk requests are made from IP addresses that support reverse lookup resolving to *.walk.adretriever.com.

Technical behavior​

Retries and throttling​

Failed requests may be retried with increasing delay on that host. HTTP 429 and 5xx responses contribute to backoff. The goal is to complete the import with minimal server load.

Extra HTML/CSS/JS requests​

In order to accurately render listing details on websites, Site Walk may request additional assets (CSS, JS, fonts, images).

Controlling Site Walk with robots.txt​

Site Walk obeys robots.txt directives.

Allow Site Walk (explicit)​

User-agent: AdRetriever
Allow: /

Block Site Walk (entire site)​

User-agent: AdRetriever
Disallow: /

Block or allow specific paths​

User-agent: AdRetriever
Disallow: /admin/
Disallow: /checkout/
Allow: /inventory/

Site Walk evaluates the most specific applicable rule for each URL. If AdRetriever is not mentioned, the User-agent: * group applies.

Verify that traffic is really AdRetriever​

  1. Confirm the User-Agent contains AdRetriever.
  2. Confirm that a reverse lookup on the request IP resolves to a *.walk.adretriever.com DNS name.

At a glance​

OperatorAdRetriever
CategoryAdvertising / catalog sync (customer-authorized)
Default robots.txt tokenAdRetriever
Primary User-AgentAdRetriever (https://adretriever.com)
Respects robots.txt (Site Walk)Yes
Typical scheduleOnce daily