Browser Tool

The Browser tool enables your agents to extract data from websites, perform web scraping, and interact with web pages. It supports multiple scraping engines including ScraperAPI for reliable, large-scale web scraping.

This tool has Default status, meaning it’s production-ready and available on all subscription plans.

Overview

The Browser tool transforms your agents into web scraping specialists capable of:

Web Scraping

Extract data from any publicly accessible website

JavaScript Rendering

Handle dynamic content and single-page applications

Scalable Architecture

Reliable scraping with proxy rotation and rate limiting

Multiple Engines

Choose between ScraperAPI and Beautiful Soup engines

Configuration Parameters

scraper

select

required

The scraping engine to use for web data extraction
Options:

scraperapi - Professional scraping service with proxy rotation
beautifulsoup - Python-based HTML parsing for simple scraping

render

select

Whether to render JavaScript when using ScraperAPI
Options:

true - Enable JavaScript rendering for dynamic content
false - Disable JavaScript rendering for faster scraping
Condition: Only shown when scraper is set to scraperapi
Default: false
Note: Enabling JavaScript rendering increases processing time and costs

Setup Instructions

Navigate to Tools

Go to the Tools section in your project dashboard

Create Browser Tool

Click Create Tool and select Browser

Select Scraper Engine

Choose between ScraperAPI for advanced features or Beautiful Soup for simple scraping

Configure JavaScript Rendering

If using ScraperAPI, decide whether to enable JavaScript rendering

Test Scraping

Use the test button to verify scraping functionality with a sample URL

Add to Agent

Assign this tool to your agents in agent settings

Scraper Engines

ScraperAPI

Professional scraping service with enterprise-grade features:

Features
Best For
Configuration

Proxy Rotation: Automatic IP rotation to avoid blocking
CAPTCHA Handling: Automatic CAPTCHA solving
JavaScript Rendering: Full browser rendering for SPAs
Geographic Targeting: Scrape from specific countries
High Success Rate: 99.9% uptime and reliability
Rate Limiting: Built-in request throttling

{
  "scraper": "scraperapi",
  "render": true
}

Beautiful Soup

Python-based HTML parsing for lightweight scraping:

Features
Best For
Configuration

Fast Processing: Lightweight HTML parsing
CSS Selectors: jQuery-like element selection
XPath Support: Advanced element targeting
Low Cost: No external service fees
Simple Setup: Minimal configuration required
Local Processing: Data processed internally

{
  "scraper": "beautifulsoup"
}

Usage Examples

E-commerce Price Monitoring

{
  "scraper": "scraperapi",
  "render": false
}

Use Case: Monitor product prices across multiple retailers

Extract product names, prices, and availability
Track price changes over time
Compare competitor pricing
Generate price alerts

News Content Extraction

{
  "scraper": "scraperapi",
  "render": true
}

Use Case: Aggregate news articles and content

Extract article titles, content, and metadata
Handle dynamic loading of articles
Collect author information and publication dates
Process multiple news sources

{
  "scraper": "scraperapi",
  "render": true
}

Use Case: Monitor public social media content

Extract post content and engagement metrics
Track hashtag usage and trends
Monitor brand mentions
Analyze public sentiment

Lead Generation

{
  "scraper": "beautifulsoup"
}

Use Case: Extract contact information from business directories

Scrape company names and addresses
Extract contact details and websites
Build prospect databases
Verify business information

JavaScript Rendering

When to Enable JavaScript

Enable JavaScript rendering when scraping:

Single Page Applications (SPAs)

Examples: React, Vue.js, Angular applications Why: Content is dynamically loaded after initial page load Impact: Essential for accessing actual content

Infinite Scroll Pages

Examples: Social media feeds, product listings Why: Content loads as user scrolls Impact: Captures more complete data sets

Dynamic Forms

Examples: Search results, filtered content Why: Content changes based on user interaction Impact: Access to filtered or searched content

AJAX-Heavy Sites

Examples: Modern e-commerce, news sites Why: Content loaded via asynchronous requests Impact: Gets fully rendered page content

When to Disable JavaScript

Disable JavaScript rendering for:

Static HTML Sites: Traditional websites with server-rendered content
Speed Optimization: Faster scraping for simple content
Cost Control: Reduce processing time and costs
API Endpoints: Direct data access without rendering
RSS Feeds: XML-based content feeds

Advanced Features

ScraperAPI Advanced Options

{
  "scraper": "scraperapi",
  "render": true,
  "advanced_options": {
    "country_code": "US",
    "device_type": "desktop",
    "premium_proxy": true,
    "session_number": 1,
    "wait_for": ".dynamic-content"
  }
}

Beautiful Soup Selectors

# CSS Selectors
soup.select('.product-name')          # Class selector
soup.select('#product-price')         # ID selector
soup.select('div.item > h3')         # Child combinator
soup.select('a[href*="product"]')    # Attribute selector

# XPath-like functionality
soup.find_all('div', class_='product')
soup.find('span', {'id': 'price'})

Data Extraction Patterns

Structured Data Extraction

Product Information
Article Content
Contact Information

{
  "selectors": {
    "product_name": "h1.product-title",
    "price": ".price-current",
    "description": ".product-description",
    "images": "img.product-image",
    "availability": ".stock-status"
  }
}

{
  "selectors": {
    "headline": "h1.headline",
    "author": ".byline .author",
    "publish_date": "time.publish-date",
    "content": ".article-body p",
    "tags": ".tags a"
  }
}

{
  "selectors": {
    "company_name": ".company-name",
    "address": ".address",
    "phone": ".phone",
    "email": ".email",
    "website": ".website a"
  }
}

Error Handling & Resilience

Common Scraping Challenges

Rate Limiting

Problem: Websites blocking frequent requests Solution:

Use ScraperAPI’s built-in rate limiting
Implement random delays between requests
Rotate user agents and IP addresses
Respect robots.txt guidelines

Dynamic Content

Problem: Content not appearing in scraped HTML Solution:

Enable JavaScript rendering
Wait for specific elements to load
Use session persistence for multi-step flows
Handle AJAX loading states

Anti-Bot Measures

Problem: CAPTCHA challenges, bot detection Solution:

Use ScraperAPI’s CAPTCHA solving
Implement human-like browsing patterns
Use residential proxy networks
Vary request timing and patterns

Content Changes

Problem: Website structure changes breaking selectors Solution:

Use multiple selector fallbacks
Implement structural change detection
Regular selector validation
Flexible parsing strategies

Best Practices

Ethical Scraping

Respect Website Terms: Always review and comply with website terms of service and robots.txt files before scraping.

Rate Limiting: Implement reasonable delays between requests to avoid overwhelming target servers.

Check Robots.txt: Review /robots.txt for scraping guidelines
Respect Rate Limits: Don’t overwhelm servers with requests
Use Public Data: Only scrape publicly accessible information
Attribution: Credit sources when republishing scraped content
Privacy Compliance: Respect user privacy and data protection laws

Performance Optimization

{
  "optimization": {
    "concurrent_requests": 5,
    "request_delay": "1-3s",
    "timeout": 30,
    "retry_attempts": 3,
    "cache_responses": true
  }
}

Selector Best Practices

Robust Selectors: Use stable element attributes
Fallback Options: Multiple selector options for reliability
Specific Targeting: Avoid overly generic selectors
Future-Proof: Choose selectors likely to persist
Test Regularly: Validate selectors against live sites

Monitoring & Analytics

Scraping Metrics

Track important scraping performance indicators:

Success Rate: Percentage of successful scrapes
Response Time: Average time per scraping request
Data Quality: Completeness and accuracy of extracted data
Error Frequency: Common failure patterns
Cost Analysis: Scraping costs per data point

Alerting & Monitoring

Monitoring Setup:
  - Success rate drops below 90%
  - Response time exceeds 30 seconds
  - Unusual increase in errors
  - Selector failure detection
  - Cost threshold alerts

Compliance & Legal

Data Protection

GDPR Compliance: Handle EU personal data appropriately
CCPA Compliance: Respect California privacy rights
Data Minimization: Collect only necessary information
Secure Storage: Protect scraped data appropriately
Access Controls: Limit access to scraped data

Copyright Considerations

Fair Use: Understand fair use limitations
Attribution: Provide proper source attribution
Commercial Use: Review commercial usage rights
Content Licensing: Respect content licensing terms
DMCA Compliance: Handle takedown requests properly

Troubleshooting

Common Issues

Empty Results

Symptoms: Scraping returns no data Solutions:

Check if JavaScript rendering is needed
Verify selectors against current page structure
Confirm website is accessible
Review rate limiting and blocking

Incomplete Data

Symptoms: Missing or partial data extraction Solutions:

Enable JavaScript rendering for dynamic content
Increase wait times for loading elements
Review and update CSS selectors
Check for pagination or infinite scroll

Scraping Blocked

Symptoms: HTTP 403, 429, or CAPTCHA responses Solutions:

Switch to ScraperAPI for proxy rotation
Implement request delays and throttling
Use different user agents
Review robots.txt compliance

Performance Issues

Symptoms: Slow scraping or timeouts Solutions:

Optimize CSS selectors for specificity
Disable JavaScript rendering if not needed
Reduce concurrent request limits
Implement response caching

Next Steps

Set Up Your First Scraper

Create your first web scraping agent

Advanced Scraping Techniques

Learn advanced scraping strategies

Other Tools

Explore other available tools

API Reference

View the tools API documentation

Getting Started

🤖 Agents

🔧 Tools

🗄️ Datasources

📋 Batches

📊 Monitor

Advanced

​Browser Tool

​Overview

Web Scraping

JavaScript Rendering

Scalable Architecture

Multiple Engines

​Configuration Parameters

​Setup Instructions

​Scraper Engines

​ScraperAPI

​Beautiful Soup

​Usage Examples

​E-commerce Price Monitoring

​News Content Extraction

​Social Media Monitoring

​Lead Generation

​JavaScript Rendering

​When to Enable JavaScript

​When to Disable JavaScript

​Advanced Features

​ScraperAPI Advanced Options

​Beautiful Soup Selectors

​Data Extraction Patterns

​Structured Data Extraction

​Error Handling & Resilience

​Common Scraping Challenges

​Best Practices

​Ethical Scraping

​Performance Optimization

​Selector Best Practices

​Monitoring & Analytics

​Scraping Metrics

​Alerting & Monitoring

​Compliance & Legal

​Data Protection

​Copyright Considerations

​Troubleshooting

​Common Issues

​Next Steps

Set Up Your First Scraper

Advanced Scraping Techniques

Other Tools

API Reference

Browser Tool

Overview

Configuration Parameters

Setup Instructions

Scraper Engines

ScraperAPI

Beautiful Soup

Usage Examples

E-commerce Price Monitoring

News Content Extraction

Social Media Monitoring

Lead Generation

JavaScript Rendering

When to Enable JavaScript

When to Disable JavaScript

Advanced Features

ScraperAPI Advanced Options

Beautiful Soup Selectors

Data Extraction Patterns

Structured Data Extraction

Error Handling & Resilience

Common Scraping Challenges

Best Practices

Ethical Scraping

Performance Optimization

Selector Best Practices

Monitoring & Analytics

Scraping Metrics

Alerting & Monitoring

Compliance & Legal

Data Protection

Copyright Considerations

Troubleshooting

Common Issues

Next Steps