# Cloudflare & Elfsight AI Chatbot: Whitelist & Crawler Details **URL:** \ **Category:** General Questions\ **Tags:** ai-chatbot\ **Created:** [September 21, 2026, 1:39pm UTC](https://community.elfsight.com/t/cloudflare-elfsight-ai-chatbot-whitelist-crawler-details/174766 "2026-09-21T13:39:28Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Petar\_Dietrich](https://sea2.discourse-cdn.com/flex020/user_avatar/community.elfsight.com/petar_dietrich/32/92003_2.png) [@Petar\_Dietrich](https://community.elfsight.com/u/Petar_Dietrich)\ **Post date:** [September 21, 2026, 1:39pm UTC](https://community.elfsight.com/t/cloudflare-elfsight-ai-chatbot-whitelist-crawler-details/174766/1 "2026-09-21T13:39:28Z")
Hi Elfsight Team, I’m using Cloudflare and want to make sure its anti-bot rules (like _Block AI Bots_ or WAF challenges) don’t block the Elfsight AI Chatbot from crawling and reading my website. Could the dev team clarify: 1. **Crawler Identifiers:** What is the official **User-Agent string** and/or **IP address ranges** used by the AI Chatbot crawler so we can add them to Cloudflare’s allowlist? 2. **Cloudflare WAF Setup:** What is the recommended Custom WAF rule or exception to keep indexing smooth without compromising site security? 3. **Crawl Failures:** How does the bot handle Cloudflare challenge pages (403/429 errors), and are crawl failures flagged in the dashboard? Thanks! ---
**Author:** ![Petar\_Dietrich](https://sea2.discourse-cdn.com/flex020/user_avatar/community.elfsight.com/petar_dietrich/32/92003_2.png) [@Petar\_Dietrich](https://community.elfsight.com/u/Petar_Dietrich)\ **Post date:** [September 21, 2026, 2:36pm UTC](https://community.elfsight.com/t/cloudflare-elfsight-ai-chatbot-whitelist-crawler-details/174766/3 "2026-09-21T14:36:51Z")
**Update:** Following my above inquiry, I ran into [this post](https://community.elfsight.com/t/issue-with-web-pages-training/173461?utm_source=gemini). Based on the guidance provided, I have created a Cloudflare WAF rule that whitelists the user-agent `Elfsight-AI-Chatbot-Knowledge-Crawler`. However, **I still think it’s important to receive feedback from your dev team to ensure I haven’t missed any other crawlers (i.e., user-agents) that may impact the performance of the Elfsight AI Chatbot**. For example, since the chatbot relies on models like `gpt-5.4-mini` and `gpt-5.6-luna`, should we also whitelist the user-agent `GPTBot` or any other third-party crawlers? Finally, **is this whitelisting guidance and full list of User-Agents documented anywhere in Elfsight’s Knowledge Base**? Having an official Help Center article covering firewall setup and Cloudflare (i.e., CDN) compatibility for the AI Chatbot would be extremely valuable for all users. Thank you! ---
**Author:** ![Max](https://sea2.discourse-cdn.com/flex020/user_avatar/community.elfsight.com/max/32/710_2.png) [@Max](https://community.elfsight.com/u/Max)\ **Post date:** [September 21, 2026, 2:53pm UTC](https://community.elfsight.com/t/cloudflare-elfsight-ai-chatbot-whitelist-crawler-details/174766/4 "2026-09-21T14:53:08Z")
Hi @Petar_Dietrich 👋 Thanks for your questios and a detailed follow-up. Yes, the User-Agent you found is the correct one. To make things clearer, here’s a full breakdown from our devs: ### **Crawler Identifiers** * * * Our AI Chatbot uses the following official crawler User-Agent: ```auto Elfsight-AI-Chatbot-Knowledge-Crawler/1.0 (+https://elfsight.com) ``` The full User-Agent string is: ```auto Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Elfsight-AI-Chatbot-Knowledge-Crawler/1.0 (+https://elfsight.com) ``` We don’t have a fixed list of crawler IP addresses, so for Cloudflare we recommend creating an exception based on the User-Agent rather than IP addresses. ### **Cloudflare WAF Setup** * * * - Under **Security → WAF → Custom rules** , you can create a rule using the following expression: ```auto (http.user_agent contains "Elfsight-AI-Chatbot-Knowledge-Crawler") ``` - Set the action to **Skip**. - Skip only the security layers that may interfere with crawling, such as **Super bot Fight Mode / Block AI Bots** , and, if necessary, relevant Managed Rules, rather than disabling protection for the site as a whole. This allows the Elfsight crawler through while keeping the rest of your Cloudflare protections active. You can also verify the setup from a terminal with: ```auto curl -I -A "Elfsight-AI-Chatbot-Knowledge-Crawler/1.0 (+https://elfsight.com)" https://your-domain.com/ ``` Replace `your-domain.com` with your actual domain. Ideally, the request should return `HTTP 200`, rather than a `403`, `429`, or a Cloudflare challenge page. ### **Crawl Failures** * * * Our crawler cannot bypass Cloudflare JavaScript challenges, CAPTCHA pages, or similar protections. If a request is blocked with a `403`, `429`, or another challenge response, the affected page won’t be indexed. In the configurator, such pages will appear with an error status (red icon) under **Knowledge → Web Pages**. The exact HTTP status code isn’t currently shown in the interface; instead, you’ll see a general page-processing error along with **Retry / Retrain** options. Once you’ve added the Cloudflare exception, you can retrain the affected pages. * * * We don’t have a dedicated article at the moment, but this is a nice idea and we’ll definitely think about it. If you have any questions left, let us know. We’ll be happy to help 🙂 ---
**Author:** ![Petar\_Dietrich](https://sea2.discourse-cdn.com/flex020/user_avatar/community.elfsight.com/petar_dietrich/32/92003_2.png) [@Petar\_Dietrich](https://community.elfsight.com/u/Petar_Dietrich)\ **Post date:** [September 21, 2026, 3:10pm UTC](https://community.elfsight.com/t/cloudflare-elfsight-ai-chatbot-whitelist-crawler-details/174766/5 "2026-09-21T15:10:05Z")
Hi @Max, Thank you for the information and your quick reply. I performed the terminal cURL test. Our website passed with flying colors (`HTTP/1.1 200 OK`). Based on what you provided, please advise on the following: 1. Does the configurator automatically detect and report AI chatbot crawling issues (e.g., 403 or 429 errors)? 2. How often does Elfsight’s AI chatbot crawl our web pages? If it only crawls once or during initial training, I highly recommend automating recurring crawls. As an option, your developers can add a simple button command labeled “Retrain All” somewhere in the Web Pages section of the configurator. Web pages are frequently updated, and many websites have hundreds of pages! 3. Do I need to whitelist GPTBot? Please refer to my previous inquiry/update. **Please note:** Updating your documentation to reflect the above is crucial to ensure optimal performance for Elfsight’s AI chatbot. Cheers! ---
**Author:** ![Max](https://sea2.discourse-cdn.com/flex020/user_avatar/community.elfsight.com/max/32/710_2.png) [@Max](https://community.elfsight.com/u/Max)\ **Post date:** [September 22, 2026, 6:13am UTC](https://community.elfsight.com/t/cloudflare-elfsight-ai-chatbot-whitelist-crawler-details/174766/6 "2026-09-22T06:13:09Z")
Hi there, @Petar_Dietrich 👋 Thanks for your follow-up! Regarding the crawling issue detection — the configurator doesn’t continuously monitor pages or automatically detect crawling issues in the background, as pages aren’t recrawled automatically at the moment. Crawling issues can be identified when the crawler is actively running, for example: - during the initial setup; - when adding new web pages to the Knowledge Base; - after manually clicking **Retrain**. Regarding the crawling frequency, the chatbot currently crawls pages during the initial setup and whenever you manually trigger a **Retrain**. Automatic recurring crawls are not supported at the moment. We already have [this request](https://community.elfsight.com/t/automatically-update-the-knowledge-base-after-any-changes-made-on-the-page-or-after-set-interval-training-with-urls-option/139965) on our Wishlist for automatically updating the Knowledge Base after page changes or at set intervals. We’ll share any updates there if this functionality becomes available. As for **GPTBot** — you don’t need to whitelist it separately. The crawler required for the AI Chatbot Knowledge Base is the **Elfsight-AI-Chatbot-Knowledge-Crawler** , so allowing this User-Agent in Cloudflare should be sufficient. Hope this helps 🙂