Resource Types

​

Resource Types Overview

• Datasource:
  • Files
    • PDF Documents
    • Word Documents
    • Text Files
    • Spreadsheets
  • URLs & Sitemaps
    • Single Pages
    • Website Crawls
    • Sitemap Indexing
  • Manual Content
    • Direct Text Entry

​

Files

​

Supported File Formats

​

File Upload

​

Single File Upload

​

Bulk Upload

Bulk Upload:
1. Select **Add Multiple Files**
2. Choose up to 10 files simultaneously
3. System processes in parallel
4. Monitor progress bar
5. All files indexed and ready

​

File Processing

1. Upload File
2. Extract Text
3. Split into Chunks
4. Generate Embeddings
5. Index for Search
6. Ready for Queries

• Text extraction: Depends on file size (typically 1-5 seconds)
• Chunking: Split into retrievable segments
• Embedding: Convert to vector representations
• Indexing: Make searchable

​

File Management

• See all uploaded files in resource list
• File size and upload date
• Processing status
• Number of indexed chunks

• Delete individual files
• Update datasource immediately
• Changes available to agents

• Add metadata/tags to files
• Group by topic or category
• Document file contents
• Track versions

​

Best Practices for Files

• Use Clear Names: Descriptive filenames help agents understand content
• Well-Structured: Use headings, sections, and clear formatting
• Compression: Large PDFs may take longer to process
• Clean Text: Remove unnecessary formatting or corrupted text
• Complete Content: Include all relevant information
• Versioning: Archive old versions, keep current ones

​

URLs & Sitemaps

​

Adding URLs

​

Single URL

​

Sitemaps

​

Web Crawling Options

​

URL Processing

1. URL Request
2. Fetch Web Content
3. Extract Main Text
4. Remove HTML/Styling
5. Split into Chunks
6. Generate Embeddings
7. Index for Search

• Dynamic content: May need JavaScript rendering
• Authentication: Public pages only
• Rate limiting: Respectful crawling speed
• Robots.txt: Honors site restrictions

​

URL Re-scraping (WEBPAGE Only)

​

Automatic Re-scraping

{
  "name": "Blog Homepage",
  "type": "WEBPAGE",
  "url": "https://myblog.com",
  "config": {
    "scheduled": true,
    "schedule_interval_hours": 24
  }
}

​

Manual Re-scraping

POST /resources/{resource_id}/scheduler/run?force=true

• New URLs found
• Jobs created for new URLs
• Processing status

​

URL Management

• See all indexed URLs
• Crawl date and re-scrape status
• Link extraction results
• Last indexed date

• ❌ Not available for URL (type: URL) resources
• ❌ Sitemaps not auto-discovered
• ❌ Only one initial URL per WEBPAGE
• ✅ Extracts links from scraped page
• ✅ Creates resources for new links

• Specific Pages: Target exact content
• Section Crawling: Crawl product docs section
• Sitemap: Complete website indexing
• Scheduled: Regular updates for dynamic sites

​

Best Practices for URLs

• Public Accessibility: Ensure pages are publicly accessible
• Clean URLs: Avoid login-required pages
• Stable URLs: Use permanent URLs, not temporary redirects
• Site Structure: Organize content logically for crawling
• Freshness: Recrawl periodically for updates
• Respect robots.txt: Honor site crawling rules

​

Manual Content

​

Direct Text Entry

​

Content Format

• Plain text
• Markdown with formatting
• Structured lists and tables
• Code snippets (syntax highlighting)
• Multi-paragraph content

# Company Return Policy

## Overview
Our 30-day return policy allows customers to return...

## Eligibility
- Items must be unused and in original condition
- Purchase receipt required
- Applies to purchases within 30 days

## How to Return
1. Visit our returns portal
2. Enter your order number
3. Select items to return
4. Print shipping label
5. Ship items back to us

## Timeline
- Processing: 5-7 business days
- Refund: 3-5 business days after processing

​

Text Entry Workflows

• Policy Statement
• Eligibility Rules
• Process Steps
• Contact Information

• Overview
• Prerequisites
• Step-by-step Instructions
• Troubleshooting
• Escalation Path

• Question
• Answer
• Related Topics
• See Also Links

​

Content Management

• Update text directly
• Changes indexed immediately
• Available to agents right away
• No additional upload needed

• Group related entries
• Add meaningful titles
• Use consistent formatting
• Tag for categorization

• Keep change history
• Document updates
• Track who edited
• Revert if needed

​

Best Practices for Manual Content

• Clear Structure: Use headings and sections
• Completeness: Include all necessary details
• Formatting: Use markdown for readability
• Specificity: Be detailed, not vague
• Examples: Include concrete examples
• Updates: Keep current and accurate

​

Resource Metadata

​

Adding Metadata

​

Resource Search & Retrieval

​

How Agents Use Resources

1. Agent query — e.g. “How do I reset my password?”
2. Search all resources, such as:
   • PDF: User Guide
   • URL: FAQ Page
   • Text: Reset Procedure
3. Find similar content
4. Return top results
5. Agent uses for response

​

Search Configuration

• Top K: Number of resources to return (default: 10)
• Similarity Threshold: Minimum relevance score
• Reranking: AI-powered result refinement
• Multi-Query: Generate query variations

​

Quality Metrics

• Retrieval Rate: How often resources are used
• Relevance: Quality of returned results
• Coverage: Percentage of queries with results
• Performance: Speed of retrieval

​

Resource Organization Strategy

​

By Content Type

• Files/
  • Product Guides
  • API Documentation
  • Policies
• URLs/
  • Blog Articles
  • Knowledge Base
  • Help Center
• Manual/
  • Quick Answers
  • FAQs
  • Common Issues

​

By Topic

• Getting Started/
• Product Features/
• Troubleshooting/
• Billing/
• Policies/

​

Next Steps

• Resource Metadata - Configure resource properties
• Datasource Metadata - Organize and manage datasources
• Datasource Types - Learn about Manual vs Google Drive
• Agent Configuration - Connect datasources to agents