We’d like to provide a website URL, and the assistant will scrape the content to create knowledge base.

I tried the AI chatbot and thought this is already a base feature. Was surprised it is not! If it is there I go for it.

Greetings, @Tom_Fuchs and welcome to the Community :waving_hand: Many thanks for the feedback! I completely understand your point. Let’s hope the devs will release it as soon as possible and we’ll keep you posted here on any progress :slightly_smiling_face:

Attached is a python file I developed with GITHUB AI to scrape a website sitemap (best use: website content pages sitemap). Load the file into Visual Code or another python editor. Running the code will prompt you for the website URL address and generate both a JSON file and a text file of the sitem…

Hola Salvatore, no veo el archivo adjunto, me lo puedes enviar, es una pasada lo que has creado y tengo ganas de probarlo Mucha gracias El El mié, 11 jun 2025 a las 21:48, Salvatore Salvia via Elfsight Community < notifications@elfsight.discoursemail.com > escribió:

Robofi.ai looks interesting. But my next step is to build my own chat bot and own it. Sal “Tory” Salvia [image] Home Explore our collection of sailing documentaries and how-to sail and maintain your sailboat videos. Sailflix.com 301-641-3589 (m) s/v Sparkle…

Max, We previously tested a competing AI chatbot that used both website crawl data and uploaded training files. Unfortunately, we found it impossible to train that system to give valid responses. The competitor’s AI consistently confused our products and mishandled overlapping information between …

Hi there, @Paul_D :waving_hand: Thank you for your comment! Glad to say that it will possible to crawl a specific page for the knowledge base creation :slightly_smiling_face:

I believe Elfsight has web crawling as a future upgrade for the chat-bot. We use a python script to scrape a product sitemap on our website then clean up the output and convert it from JSON to TXT. We then uploaded the TXT file to our Elfsight chat-bot training port.

In our experience, scraping our web pages for chatbot training wasn’t very effective. We started by scraping a single, comprehensive FAQ page from our site. While the content is SEO-optimized and works well for human visitors, it repeatedly confused the AI. We had to heavily edit and shorten the t…

We had success scraping our product sitemap then scrubbing it for unneeded lines and characters. We did include a training instruction to add a URL when available. To do the scrape we used Git-Hub Co-pilot chat to develop a script in python. Sal “Tory” Salvia [image] Home Explore our co…

URL-based data scraping to create a knowledge base

Wishlist AI Chatbot

Max September 24, 2025, 1:25pm 58

Hey everyone!

I am coming with the great news - now you can train your chatbot using page URLs

Check this post for more details - AI Chatbot: Train your assistant using webpage URLs 🚀

6 Likes

Topic		Replies	Views
AI Chatbot: Train your assistant using webpage URLs 🚀 New Feature ai-chatbot	5	585	September 24, 2025
Automatically crawl all pages within a website to a knowledge base AI Chatbot Gathering-Feedback	0	49	September 24, 2025
Training AI Chat Bot with sitemap AI Chatbot Released	7	1079	December 22, 2025
Crawl specific page for the answer General Questions ai-chatbot	5	27	February 9, 2026
AI Chatbot: [MAJOR UPDATE] Sitemap training, new language model and more! 🚀 New Feature ai-chatbot	11	577	January 19, 2026

URL-based data scraping to create a knowledge base

Related topics