I believe Elfsight has web crawling as a future upgrade for the chat-bot. We use a python script to scrape a product sitemap on our website then clean up the output and convert it from JSON to TXT. We then uploaded the TXT file to our Elfsight chat-bot training port.
2 Likes
In our experience, scraping our web pages for chatbot training wasn’t very effective.
We started by scraping a single, comprehensive FAQ page from our site. While the content is SEO-optimized and works well for human visitors, it repeatedly confused the AI.
We had to heavily edit and shorten the training document created from the scraped page, removing any entries that overlapped with other training materials — even subtle or partial overlaps — before the chatbot gave satisfactory answers again.
Our takeaway: content that works for people doesn’t always work for the AI.
What works better for us is monitoring chatbot history emails. When we spot an unsatisfactory answer, we upload that chat to ChatGPT along with our training documents, explain the correct response, and ask ChatGPT to suggest specific edits to the training files.
Since the chatbot is newly installed, we’re currently making a few edits per day.
At this stage, we’re impressed by the quality of nearly all responses.
3 Likes
We had success scraping our product sitemap then scrubbing it for unneeded lines and characters. We did include a training instruction to add a URL when available. To do the scrape we used Git-Hub Co-pilot chat to develop a script in python.
Sal “Tory” Salvia
Sailflix.com
301-641-3589 (m)
s/v Sparkle Plenty
Fort Pierce, Florida
1 Like
3 posts were merged into an existing topic: AI Chatbot: Training Best Practices Thread
Update: the feature moved to development! 
6 Likes
There should be a way to add the website address to train the chat box versus uploading documents.
1 Like
Hi there, @user25363 
Thank you for the feedback!
Happy to say that this feature is already in works. I’ve moved your comment to the thread, where we’ll keep you updated 
1 Like
Hey everyone!
I am coming with the great news - now you can train your chatbot using page URLs
Check this post for more details - AI Chatbot: Train your assistant using webpage URLs 🚀
5 Likes
Great work!
2 Likes
Glad to hear that! Thank you so much for the feedback 
Does this read the web page(s) live in real time?
1 Like
No, it doesn’t. But check out more information at AI Chatbot: Train your assistant using webpage URLs 🚀
1 Like
Hi there, @John_Carson
Yep, the pages edits are not pulled to the widget automatically now, but this request is already on the Wishlist. Feel free to upvote it - Automatically update the knowledge base after any changes made on the page (training with URLs option)
The good news is that you can manually do this using the “Refrain” button
