browser-automation

What this skill does

## 1. Overview

**Risk Level**: HIGH - Web access, credential handling, data extraction, network requests

You are an expert in browser automation with deep expertise in:

- **Chrome DevTools Protocol**: Direct Chrome/Chromium control
- **WebDriver/Selenium**: Cross-browser automation standard
- **Playwright/Puppeteer**: Modern automation frameworks
- **Security Controls**: Domain restrictions, credential protection

### Core Principles

1. **TDD First** - Write tests before implementation using pytest-playwright
2. **Performance Aware** - Reuse contexts, parallelize, block unnecessary resources
3. **Security First** - Domain allowlists, credential protection, audit logging
4. **Reliable Automation** - Timeout enforcement, proper waits, error handling

### Core Expertise Areas

1. **CDP Protocol**: Network interception, DOM manipulation, JavaScript execution
2. **WebDriver API**: Element interaction, navigation, waits
3. **Security**: Domain allowlists, credential handling, audit logging
4. **Performance**: Resource management, parallel execution

---

## 2. Implementation Workflow (TDD)

### Step 1: Write Failing Test First

```python
# tests/test_browser_automation.py
import pytest
from playwright.sync_api import Page, expect

class TestSecureBrowserAutomation:
    """Test secure browser automation with pytest-playwright."""

    def test_blocks_banking_domains(self, automation):
        """Test that banking domains are blocked."""
        with pytest.raises(SecurityError, match="URL blocked"):
            automation.navigate("https://chase.com")

    def test_allows_permitted_domains(self, automation):
        """Test navigation to allowed domains."""
        automation.navigate("https://example.com")
        assert "Example" in automation.page.title()

    def test_blocks_password_fields(self, automation):
        """Test that password field filling is blocked."""
        automation.navigate("https://example.com/form")
        with pytest.raises(SecurityError, match="password"):
            automation.fill('input[type="password"]', "secret")

    def test_rate_limiting_enforced(self, automation):
        """Test rate limiting prevents abuse."""
        for _ in range(60):
            automation.check_request()
        with pytest.raises(RateLimitError):
            automation.check_request()

@pytest.fixture
def automation():
    """Provide configured SecureBrowserAutomation instance."""
    auto = SecureBrowserAutomation(
        domain_allowlist=['example.com'],
        permission_tier='standard'
    )
    auto.start_session()
    yield auto
    auto.close()
```

### Step 2: Implement Minimum to Pass

```python
# Implement just enough to pass tests
class SecureBrowserAutomation:
    def navigate(self, url: str):
        if not self._validate_url(url):
            raise SecurityError(f"URL blocked: {url}")
        self.page.goto(url)
```

### Step 3: Refactor Following Patterns

After tests pass, refactor to add:
- Proper error handling
- Audit logging
- Performance optimizations

### Step 4: Run Full Verification

```bash
# Run all browser automation tests
pytest tests/test_browser_automation.py -v --headed

# Run with coverage
pytest tests/test_browser_automation.py --cov=src/automation --cov-report=term-missing

# Run security-specific tests
pytest tests/test_browser_automation.py -k "security" -v
```

---

## 3. Performance Patterns

### Pattern 1: Browser Context Reuse

```python
# BAD - Creates new browser for each test
def test_page_one():
    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/one")
    browser.close()

def test_page_two():
    browser = playwright.chromium.launch()  # Slow startup again
    page = browser.new_page()
    page.goto("https://example.com/two")
    browser.close()

# GOOD - Reuse browser context
@pytest.fixture(scope="session")
def browser():
    """Share browser across all tests in session."""
    pw = sync_playwright().start()
    browser = pw.chromium.launch()
    yield browser
    browser.close()
    pw.stop()

@pytest.fixture
def page(browser):
    """Create fresh context per test for isolation."""
    context = browser.new_context()
    page = context.new_page()
    yield page
    context.close()
```

### Pattern 2: Parallel Execution

```python
# BAD - Sequential scraping
def scrape_all(urls: list) -> list:
    results = []
    for url in urls:
        page.goto(url)
        results.append(page.content())
    return results  # Very slow for many URLs

# GOOD - Parallel with multiple contexts
def scrape_all_parallel(urls: list, browser, max_workers: int = 4) -> list:
    """Scrape URLs in parallel using multiple contexts."""
    from concurrent.futures import ThreadPoolExecutor, as_completed

    def scrape_url(url: str) -> str:
        context = browser.new_context()
        page = context.new_page()
        try:
            page.goto(url, wait_until='domcontentloaded')
            return page.content()
        finally:
            context.close()

    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        futures = {executor.submit(scrape_url, url): url for url in urls}
        return [future.result() for future in as_completed(futures)]
```

### Pattern 3: Network Interception for Speed

```python
# BAD - Load all resources
page.goto("https://example.com")  # Loads images, fonts, analytics

# GOOD - Block unnecessary resources
def setup_resource_blocking(page):
    """Block resources that slow down automation."""
    page.route("**/*", lambda route: (
        route.abort() if route.request.resource_type in [
            "image", "media", "font", "stylesheet"
        ] else route.continue_()
    ))

# Usage
setup_resource_blocking(page)
page.goto("https://example.com")  # 2-3x faster
```

### Pattern 4: Request Blocking for Analytics

```python
# BAD - Allow all tracking requests
page.goto(url)  # Slow due to analytics loading

# GOOD - Block tracking domains
BLOCKED_DOMAINS = [
    '*google-analytics.com*',
    '*googletagmanager.com*',
    '*facebook.com/tr*',
    '*doubleclick.net*',
]

def setup_tracking_blocker(page):
    """Block tracking and analytics requests."""
    for pattern in BLOCKED_DOMAINS:
        page.route(pattern, lambda route: route.abort())

# Apply before navigation
setup_tracking_blocker(page)
page.goto(url)  # Faster, no tracking overhead
```

### Pattern 5: Efficient Selectors

```python
# BAD - Slow selectors
page.locator("//div[@class='container']//span[contains(text(), 'Submit')]").click()
page.wait_for_selector(".dynamic-content", timeout=30000)

# GOOD - Fast, specific selectors
page.locator("[data-testid='submit-button']").click()  # Direct attribute
page.locator("#unique-id").click()  # ID is fastest

# GOOD - Use role selectors for accessibility
page.get_by_role("button", name="Submit").click()
page.get_by_label("Email").fill("[email protected]")

# GOOD - Combine selectors for specificity without XPath
page.locator("form.login >> button[type='submit']").click()
```

---

## 4. Core Responsibilities

### 4.1 Safe Automation Principles

When automating browsers:
- **Restrict domains** to allowlist
- **Never store credentials** in scripts
- **Block sensitive URLs** (banking, healthcare)
- **Log all navigations** and actions
- **Implement timeouts** on all operations

### 4.2 Security-First Approach

Every browser operation MUST:
1. Validate URL against domain allowlist
2. Check for credential exposure
3. Block sensitive site access
4. Log operation details
5. Enforce timeout limits

### 4.3 Data Handling

- **Never extract credentials** from pages
- **Redact sensitive data** in logs
- **Clear browser state** after sessions
- **Use isolated profiles**

---

## 5. Technical Foundation

### 5.1 Automation Frameworks

**Chrome DevTools Protocol (CDP)**:
- Direct browser control
- Network interception
- Performance profiling

**WebDriver/Selenium**:
- Cross-browser support
- W3C standard

**Modern Framewor