Issue with web pages training

Ask the Community General Questions
1

Hello,

I think I am facing the exact same issue. Could you please take a look at my widget as well?

Details:

  • Widget ID: 662cf189-0c9c-4800-a48e-d3f3d494d5b6

  • Domain: apogee.gr

  • Affected URLs:

    • [https://apogee.gr/en/news/](https://apogee.gr/en/news/)

    • [https://apogee.gr/en/news/Presenting-Apogee-MailOps-operations-and-troubleshooting-for-self-hosted-mail-servers.559/](https://apogee.gr/en/news/Presenting-Apogee-MailOps-operations-and-troubleshooting-for-self-hosted-mail-servers.559/)

  • Firewall / CDN status: We have verified in Cloudflare logs that requests from the Elfsight-AI-Chatbot-Knowledge-Crawler User-Agent are passing through with a 200 OK / Skip status, so Cloudflare is not blocking the crawler.

Could you check the backend logs for my widget to see why the scraping fails with the red alert?

Thanks!

image
image399×344 10.3 KB

3

Hi there, @Markos :waving_hand:

I am so sorry about this issue!

Please let me check it with the dev team. I’ll report back once I have their response :slightly_smiling_face:

4

Hi @Markos :waving_hand:

Thank you for waiting!

Our devs looked into the issue, and it appears that your website’s bot protection (most likely Cloudflare or another WAF) is occasionally blocking requests from our crawler and returning a 403 Forbidden error.

To allow the AI Chatbot to access and process the page, please add an exception/allow rule in your Cloudflare or WAF settings for our crawler’s User-Agent:

Elfsight-AI-Chatbot-Knowledge-Crawler/1.0 (+https://elfsight.com)

Full User-Agent string:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Elfsight-AI-Chatbot-Knowledge-Crawler/1.0 (+https://elfsight.com)

Once the crawler is allowed, the link should be added and processed correctly. Check it out and let me know if it helped :slightly_smiling_face: