Website Data Source

Introduction

Enhancing your Expert's knowledge using public website data on EveryAnswer is a powerful way to ensure comprehensive and up-to-date content. This feature enables you to extract and integrate information from publicly accessible websites, making your Expert more knowledgeable and versatile. This guide provides detailed instructions on how to add public website data to your Expert.

Accessing the Feature

1. In the left navigation, under Settings, click Experts.
2. From the list of Experts, click the pencil icon on the Expert you'd like to add data to.
3. Click on the Data tab.
4. Click on the Web Data sub-tab.
5. Click on the Add Website button.

Process to Add Public Website Data

1. In the popup, enter a URL in the Website URL field.
2. Optionally, under Excluded Paths, enter paths to exclude (e.g., /path or https://www.example.com/path).
3. Click the Scan Now button.
4. Review the automatically detected pages under Discovered Pages. Uncheck pages you don't want to include.
5. Optionally go back and add more Excluded Paths.
6. Once satisfied, click Add to Expert.
7. EveryAnswer will process the pages, converting each into an AI-optimized format and training your Expert.
8. Once complete, your Expert will know that information until you remove it.

Specific URLs and Sitemaps

When entering a non-root URL (e.g., www.domain.com/blog), EveryAnswer will attempt to determine whether there are other pages under it or if it's a stand-alone page. This adjustment helps optimize the import process automatically.

EveryAnswer uses sitemaps whenever available to speed up the process of page discovery. If a sitemap is found, it's used to create the list of pages. If not, EveryAnswer will attempt to crawl the site. Sitemaps allow for importing an unlimited number of pages (limited only by plan storage), whereas without a sitemap, EveryAnswer can crawl up to 300 pages.

Page Updates and Rescanning

Keeping your Expert's knowledge current is critical. EveryAnswer offers features to detect new pages and rescan existing pages for updates. These capabilities vary based on your subscription plan:

Detecting New Pages

• EveryAnswer can periodically check for new pages on the websites you've already imported. This helps keep your Expert up-to-date with the latest information.

Rescanning for Updated Content

• Existing pages can be rescanned to detect and import any changes or updates, ensuring your Expert's knowledge remains current.

Plan Limitations

• The Starter (free) plan does not include automatic detection of new pages or rescanning for updates.
• Higher-tier plans offer these features, with the frequency of checks determined by the specific plan level. More advanced plans typically offer more frequent checks for new pages and content updates.

Manual Updates

• Regardless of the plan, you can manually trigger a rescan of a website to update your Expert's knowledge. Note that there is a 24-hour cooldown period between manual rescans of the same page to prevent excessive server load.

Limitations and Considerations

• EveryAnswer cannot extract from private or password-protected websites.
• The number of websites you can extract from depends on your subscription plan's character limit.
• Ensure compliance with legal and ethical standards when importing website data.
• Advanced techniques and a distributed server network reduce the chance of being blocked or facing rate limits.
• EveryAnswer does not honor URL arguments (components after '#', '?', or '&') when importing pages to prevent duplicate content.

For further information on managing your Experts or other features, refer to the relevant sections in your documentation, such as General Knowledge Data Source, Forms, and Lead Capture.

Related Documentation

• Experts
• Plans & Pricing
• General Knowledge Data Source
• Forms
• Lead Capture