AI Agent Knowledge Base Management

Overview

The AI Agents Knowledge Base is where you manage the information your AI Agent can use to answer customer questions. It gives teams more control over what the agent knows, which website pages it can reference, and how crawl issues are reviewed or resolved.

With this update, users can view crawl status, identify failed pages, manually re-crawl pages, exclude specific pages or sections, and add knowledge sources in more flexible ways.

Why this matters

AI Agents rely on accurate and current business information to provide helpful responses. If outdated pages, old blogs, incomplete content, or failed crawls are included in the agent’s knowledge, the quality of the agent’s answers may be affected.

The Knowledge Base helps teams keep the agent’s source information clean, relevant, and reliable.

What you can do in the Knowledge Base

Users can:

Add website content as a knowledge source
Add a sitemap for structured website crawling
Add multiple individual URLs at once
View crawl status for pages
Identify pages that are still in progress, failed, or unavailable to the agent
Manually re-crawl failed or outdated pages
Delete pages that should no longer be used
Disable specific pages or paths from being referenced by the AI Agent
Keep knowledge centrally managed while controlling visibility at the AI Agent level

Accessing the AI Agents Knowledge Base

Step 1: Navigate to AI Agents.

Step 2: Open the agent you want to manage.

Step 3: Go to the Knowledge section.

The Knowledge Base displays the sources connected to the agent, such as website links, files, Q&A, and other configured knowledge inputs.

Adding website content to the Knowledge Base

You can add website content so the AI Agent can use the information from your website when answering questions.

Step 1: Click on ‘Add Links’.

Step 2: Choose the option to add a website or website source.

Step 3: Enter the website URL.

Step 4: Start the crawl.

Once the crawl begins, review the crawl status from the Knowledge Base.

The system crawls the website and makes the discovered pages available for review.

Adding a sitemap

A sitemap can be used to help the system discover and crawl website pages in a structured way.

Step 1: Select the sitemap option.

Step 2: Enter the sitemap URL.

Step 3: Start the crawl.

Review the crawled pages once processing begins.

Using a sitemap is helpful when you want the system to follow the website’s existing page structure.

Adding multiple URLs

You can add multiple URLs at once instead of entering each URL separately.

Step 1: Select the option to add individual URLs.

Step 2: Copy and paste the URLs into the URL field. Separate URLs by line breaks or commas.

Step 3: Click on ‘Add URLs’.

This is useful when you want the agent to reference a specific set of pages, such as service pages, policy pages, location pages, or FAQs.

Reviewing crawl status

After a crawl is triggered, the Knowledge Base provides visibility into the status of crawled pages.

You can review:

Pages that were crawled successfully
Pages that are still in progress
Pages where the crawl failed
Pages that are not currently visible to the AI Agent
When the crawl was last triggered
Who triggered the crawl, where available

This helps users understand whether the agent has access to the expected website content.

Re-crawling a page

Use the re-crawl option when a page failed to crawl, was updated recently, or needs to be refreshed in the Knowledge Base.

Step 1: Find the page you want to refresh.

Step 2: Review the current crawl status.

Step 3: Select the re-crawl option.

Step 4: Wait for the crawl status to update.

Manual re-crawling helps teams refresh knowledge without needing engineering support.

Excluding pages from AI Agent responses

You can disable specific pages or paths so the AI Agent does not use them when generating responses.

This is useful when:

Blog content is outdated
A page contains old promotional information
A page should be available on the website but not used by the agent
Certain sections are not relevant to customer-facing answers
The agent should only reference approved business information

To exclude pages:

Step 1: Search for the page or path you want to exclude.

Step 2: Select the page or group of pages.

Step 3: Turn off visibility for the AI Agent.

Once disabled, the AI Agent will not use those pages as sources for responses.

Bulk disabling pages or paths

If you need to exclude a larger set of pages, such as all blog pages, you can search for a path or keyword and select multiple pages at once.

Step 1: Open the crawled page list.

Step 2: Search for the relevant path or keyword.

Step 3: Select the pages you want to exclude.

Step 4: Disable visibility for the AI Agent.

This helps teams manage large websites more efficiently.

Deleting pages from the Knowledge Base

If a page should no longer be available as part of the knowledge source, you can delete it from the Knowledge Base.

Step 1: Find the page you want to remove.

Step 2: Select the delete option.

Step 3: Confirm the action.

Deleting a page removes it from the Knowledge Base. Disabling visibility only prevents the AI Agent from using the page.

Understanding module-specific visibility

The Knowledge Base is centrally managed and may be used by multiple platform modules. A page can remain available in the central Knowledge Base while being hidden from the AI Agent.

For example, a blog page may still be useful for Content Studio, but the business may not want the AI Agent to reference that blog when answering customer questions. In this case, the page can be disabled for AI Agent usage without removing it from the broader knowledge system.

Best practices

Review failed crawl pages regularly to make sure important content is not missing.
Disable outdated blogs, old promotions, expired offers, or pages that should not influence agent responses.
Use sitemap crawling for larger websites with many pages.
Use bulk URL entry when adding a specific set of approved pages.
Re-crawl key pages after making major website updates.
Keep the AI Agent’s knowledge focused on trusted, current, and customer-ready content.

Frequently Asked Questions

What happens if a website crawl fails?

The failed pages will appear in the Knowledge Base with their crawl status. You can review the failed pages and manually re-crawl them.

Can I re-crawl pages without contacting support or engineering?

Yes. The Knowledge Base includes manual re-crawl controls so users can refresh pages directly.

Can I exclude only some pages from the AI Agent?

Yes. You can disable specific pages or paths so the AI Agent does not reference them in responses.

Can I exclude blog pages from the AI Agent?

Yes. You can search for blog pages or blog paths, select them, and disable visibility for the AI Agent.

Does disabling a page remove it from the entire platform?

Not necessarily. Visibility can be controlled at the AI Agent level. This means a page can be hidden from the AI Agent while still remaining available in the centralized Knowledge Base for other modules.

Can I add multiple URLs at once?

Yes. You can copy and paste multiple URLs using line breaks or commas.

Can I upload an Excel file of URLs?

No. The current workflow supports bulk URL entry through copy-paste, not Excel file upload.

Summary

The AI Agents Knowledge Base gives businesses greater control over the information their AI Agent uses. Teams can now manage crawls, refresh pages, exclude outdated content, and keep agent responses grounded in accurate, approved knowledge.

This helps improve response quality, reduce manual support dependency, and support more reliable hyper-personalized customer experiences at scale.

Was this article helpful?