comparano

Our feed fetcher

What ComparanoFeedBot is, and how to let it through a firewall.

In short

  • You are probably here because this string turned up in a server log or a firewall report.
  • We only download a file a merchant configured in their own Comparano account. We never crawl a site.
  • If your firewall is blocking us, allow the user agent — the rules for the common ones are below.
  • We publish no list of IP addresses, and section 4 explains why we think that would be worse.

1.What this is

Comparano is a price comparison service. A shop that lists its products with us gives us the address of its product feed — a file the shop itself publishes — and we download that file on a schedule the shop chooses, so the prices and stock we show stay current.

We only ever fetch a URL a merchant configured in their own Comparano account. We do not crawl a site, follow links, submit forms, or look for pages nobody gave us.

2.The user agents

Two, and they are separate on purpose: a host may reasonably want to allow the small occasional check while rate-limiting the large download differently, and a single name would make that impossible to express.

User agentWhat it doesHow often
ComparanoFeedBot/1.1 (+https://comparano.com/bot)Downloads the product feed the merchant configured.On the merchant's chosen interval — typically every 3 to 24 hours, plus a small headers-only request in between to see whether the file is still reachable.
ComparanoBot/1.1 (+https://comparano.com/bot)Checks that the shop is reachable, and reads the file or tag that proves the domain belongs to the merchant who claimed it.Rarely. Once during sign-up, then occasionally.

Both also carry a From header (info@comparano.com), so there is always somebody to write to about them.

3.How to allow it

Any one of these is enough on its own; you do not need all of them.

Match on the token, not the whole string. ComparanoFeedBot and ComparanoBot never change. The version after the slash does, whenever we ship a release — a rule that matches the full user agent will start failing the next time it moves.

Cloudflare — WAF › Custom rules › Skip

(http.user_agent contains "ComparanoFeedBot") or (http.user_agent contains "ComparanoBot")

ModSecurity / OWASP CRS

SecRule REQUEST_HEADERS:User-Agent "@contains ComparanoFeedBot" \
    "id:1900001,phase:1,pass,nolog,ctl:ruleEngine=Off"

nginx

map $http_user_agent $comparano {
    default              0;
    "~*ComparanoFeedBot"  1;
    "~*ComparanoBot"      1;
}

Apache

SetEnvIfNoCase User-Agent "ComparanoFeedBot" comparano
Allow from env=comparano

WordPress security plugins — Wordfence, Sucuri, iThemes — each have an allow-list that takes a user agent as well as an IP; add ComparanoFeedBot there. The setting most often responsible for blocking us is the rate limit for crawlers that are not on the list, which treats a single large file download as an attack.

Hosting panels such as cPanel, Plesk or DirectAdmin usually front the site with mod_security and a rate limiter. Ask your host to allow the user agent for the feed URL; that is normally a one-line change on their side, and quoting this page tends to be enough.

4.Why we publish no IP list

We are asked for a fixed list of addresses often enough to explain the answer here. Our fetchers run on infrastructure whose addresses change, and a list that goes stale fails silently — the shop’s feed starts breaking and nothing says why. Matching the user agent keeps working.

If your policy requires addresses, write to us and we will work something out for your setup rather than publish a range that will be wrong in a month.

5.What a block looks like from our side

A blocked feed does not fail loudly for the merchant. We record the status, mark the feed as failing after a few consecutive attempts and email them — but the message says their feed is unreachable, which is why they end up asking their host rather than us. These are the answers we see when something in front of a shop is stopping us.

StatusUsually means
403A WAF rule, or an allow-list that does not include us.
429A rate limiter counting our one download as abuse.
503A bot-protection challenge page. We cannot solve one, so we treat it as unavailable.
—No answer at all — a firewall dropping the connection rather than refusing it.

6.Asking us to stop

We fetch a feed because a merchant asked us to, so the fastest way to stop is for the shop to pause or remove it in their Comparano account. If you believe we are requesting something nobody configured, write to us with a log line and we will look into it the same day.