AdFenix Segmentation Crawler

The web crawler Realforce (formerly Adfenix) uses to read property listings from the websites of the real estate agencies we work for.

The AdFenix Segmentation Crawler is operated by Realforce (formerly Adfenix), a marketing platform for real estate agencies. It visits the public websites of agencies that are Realforce customers and reads their property listings and agent profiles, such as address, price, rooms, images, status and the responsible agent, so that we can create and update advertising campaigns for those listings on the agency's behalf. It only crawls sites belonging to our customers, discovers new listings by checking the agency's listing pages a few times a day, limits how often it requests pages from each site, and honours robots.txt. It does not log in, submit forms, collect information about site visitors, or use the content it fetches to train AI models.

How to recognise it

User-Agent: AdFenix Segmentation Crawler
robots.txt name: AdFenix
Signature-Agent: "https://api.realforce.com"

Every request is signed with Web Bot Auth (HTTP Message Signatures). See Verifying a request below. Requests that claim to be the AdFenix Segmentation Crawler but are not signed with one of our keys do not come from us.

If you allow the crawler through a firewall by its User-Agent, that rule keeps working: the User-Agent does not change.

What it fetches

Listing pages on the agency's website, to read the details of a property.
Listing overview and search result pages that the agency has asked us to watch, to find new listings.
Agent profile pages, to link listings to the agent responsible for them.
Listing data endpoints that the agency has explicitly configured for us, as an alternative to reading HTML.

The crawler only requests pages on websites that belong to Realforce customers, plus data endpoints those customers have set up for us. It does not crawl beyond the pages listed above, and it does not run JavaScript, so it does not load your scripts, analytics or ads.

When it visits

When a listing is published or first seen on the agency's site, the crawler fetches that listing page.
It returns to a listing occasionally to keep the price and status (for example "sold") up to date.
It checks the configured overview pages for new listings on a schedule, by default every six hours.

Requests to each site are queued and rate limited, and we can slow the crawler down further for any site on request. Requests time out after 60 seconds, accept compressed responses, and follow at most five redirects.

What the data is used for

The listing information is used to create, update and stop advertising campaigns for the agency that owns the listing, and is available only to that agency within Realforce. We do not sell it, publish it elsewhere, or use it to train AI models.

Controlling access

The crawler reads /robots.txt before crawling a site and follows the rules for the AdFenix user agent, falling back to the rules for *. robots.txt names cannot contain spaces, so use AdFenix; matching is case-insensitive, and a line written as User-agent: AdFenix Segmentation Crawler is honoured too. To stop the crawler from visiting your whole site:

User-agent: AdFenix
Disallow: /

To keep it out of specific sections only:

User-agent: AdFenix
Disallow: /internal/
Disallow: /search

Changes to robots.txt are picked up within 24 hours. If /robots.txt returns a 4xx status other than 429, the crawler treats the site as having no restrictions. A server error or a dropped connection is retried twice within a few seconds. If it returns 429, or still can't be fetched because of a server error or a timeout, the crawler keeps following the last copy it read, or, if it has none, stops requesting pages from the site until it can read one, trying again after a minute.

Data endpoints that an agency has set up for us are called with that agency's permission, so robots.txt does not apply to them. Every page the crawler fetches follows robots.txt.

If you are a Realforce customer, blocking the crawler will stop new and updated listings from reaching your campaigns, so please contact your Realforce representative first.

Verifying a request

Each request carries Signature-Agent, Signature-Input and Signature headers as described in the Web Bot Auth drafts. The signature covers the @authority and signature-agent components, uses Ed25519, is tagged web-bot-auth, and is valid for one minute.

Our public keys are published at:

https://api.realforce.com/.well-known/http-message-signatures-directory

Contact

If the crawler is causing problems on your site, or you have questions about how it works, email admin@realforce.com. Include your domain and, if possible, the time and URL of the requests in question.