Cloudflare & Elfsight AI Chatbot: Whitelist & Crawler Details

Ask the Community General Questions
1

Hi Elfsight Team,

I’m using Cloudflare and want to make sure its anti-bot rules (like Block AI Bots or WAF challenges) don’t block the Elfsight AI Chatbot from crawling and reading my website.

Could the dev team clarify:

  1. Crawler Identifiers: What is the official User-Agent string and/or IP address ranges used by the AI Chatbot crawler so we can add them to Cloudflare’s allowlist?

  2. Cloudflare WAF Setup: What is the recommended Custom WAF rule or exception to keep indexing smooth without compromising site security?

  3. Crawl Failures: How does the bot handle Cloudflare challenge pages (403/429 errors), and are crawl failures flagged in the dashboard?

Thanks!

3

Update:

Following my above inquiry, I ran into this post.

Based on the guidance provided, I have created a Cloudflare WAF rule that whitelists the user-agent Elfsight-AI-Chatbot-Knowledge-Crawler.

However, I still think it’s important to receive feedback from your dev team to ensure I haven’t missed any other crawlers (i.e., user-agents) that may impact the performance of the Elfsight AI Chatbot. For example, since the chatbot relies on models like gpt-5.4-mini and gpt-5.6-luna, should we also whitelist the user-agent GPTBot or any other third-party crawlers?

Finally, is this whitelisting guidance and full list of User-Agents documented anywhere in Elfsight’s Knowledge Base? Having an official Help Center article covering firewall setup and Cloudflare (i.e., CDN) compatibility for the AI Chatbot would be extremely valuable for all users.

Thank you!

4

Hi @Petar_Dietrich :waving_hand:

Thanks for your questios and a detailed follow-up. Yes, the User-Agent you found is the correct one. To make things clearer, here’s a full breakdown from our devs:

Crawler Identifiers

Our AI Chatbot uses the following official crawler User-Agent:

Elfsight-AI-Chatbot-Knowledge-Crawler/1.0 (+https://elfsight.com)

The full User-Agent string is:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Elfsight-AI-Chatbot-Knowledge-Crawler/1.0 (+https://elfsight.com)

We don’t have a fixed list of crawler IP addresses, so for Cloudflare we recommend creating an exception based on the User-Agent rather than IP addresses.


Cloudflare WAF Setup

  • Under Security → WAF → Custom rules, you can create a rule using the following expression:
(http.user_agent contains "Elfsight-AI-Chatbot-Knowledge-Crawler")

  • Set the action to Skip.

  • Skip only the security layers that may interfere with crawling, such as Super bot Fight Mode / Block AI Bots, and, if necessary, relevant Managed Rules, rather than disabling protection for the site as a whole.

This allows the Elfsight crawler through while keeping the rest of your Cloudflare protections active.

You can also verify the setup from a terminal with:

curl -I -A "Elfsight-AI-Chatbot-Knowledge-Crawler/1.0 (+https://elfsight.com)" https://your-domain.com/

Replace your-domain.com with your actual domain. Ideally, the request should return HTTP 200, rather than a 403, 429, or a Cloudflare challenge page.


Crawl Failures

Our crawler cannot bypass Cloudflare JavaScript challenges, CAPTCHA pages, or similar protections.

If a request is blocked with a 403, 429, or another challenge response, the affected page won’t be indexed.

In the configurator, such pages will appear with an error status (red icon) under Knowledge → Web Pages. The exact HTTP status code isn’t currently shown in the interface; instead, you’ll see a general page-processing error along with Retry / Retrain options.

Once you’ve added the Cloudflare exception, you can retrain the affected pages.

We don’t have a dedicated article at the moment, but this is a nice idea and we’ll definitely think about it.

If you have any questions left, let us know. We’ll be happy to help :slightly_smiling_face:

5

Hi @Max,

Thank you for the information and your quick reply.

I performed the terminal cURL test. Our website passed with flying colors (HTTP/1.1 200 OK).

Based on what you provided, please advise on the following:

  1. Does the configurator automatically detect and report AI chatbot crawling issues (e.g., 403 or 429 errors)?

  2. How often does Elfsight’s AI chatbot crawl our web pages? If it only crawls once or during initial training, I highly recommend automating recurring crawls. As an option, your developers can add a simple button command labeled “Retrain All” somewhere in the Web Pages section of the configurator. Web pages are frequently updated, and many websites have hundreds of pages!

  3. Do I need to whitelist GPTBot? Please refer to my previous inquiry/update.

Please note: Updating your documentation to reflect the above is crucial to ensure optimal performance for Elfsight’s AI chatbot.

Cheers!

6

Hi there, @Petar_Dietrich :waving_hand:

Thanks for your follow-up!

Regarding the crawling issue detection — the configurator doesn’t continuously monitor pages or automatically detect crawling issues in the background, as pages aren’t recrawled automatically at the moment.

Crawling issues can be identified when the crawler is actively running, for example:

  • during the initial setup;
  • when adding new web pages to the Knowledge Base;
  • after manually clicking Retrain.

Regarding the crawling frequency, the chatbot currently crawls pages during the initial setup and whenever you manually trigger a Retrain. Automatic recurring crawls are not supported at the moment.

We already have this request on our Wishlist for automatically updating the Knowledge Base after page changes or at set intervals. We’ll share any updates there if this functionality becomes available.

As for GPTBot — you don’t need to whitelist it separately. The crawler required for the AI Chatbot Knowledge Base is the Elfsight-AI-Chatbot-Knowledge-Crawler, so allowing this User-Agent in Cloudflare should be sufficient.

Hope this helps :slightly_smiling_face: