> ## Documentation Index
> Fetch the complete documentation index at: https://docs.statproxies.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrapy Integration

> Configure Scrapy middleware to route requests through Stat Proxies.

Scrapy is a Python web scraping framework. This guide shows how to configure it to use Stat Proxies via a custom downloader middleware.

## Middleware setup

Create a proxy middleware in your Scrapy project:

```python theme={null}
# myproject/middlewares.py

import base64

class StatProxiesMiddleware:
    def __init__(self, proxy_url, proxy_user, proxy_pass):
        self.proxy_url = proxy_url
        self.proxy_auth = base64.b64encode(
            f"{proxy_user}:{proxy_pass}".encode()
        ).decode()

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            proxy_url=crawler.settings.get('STAT_PROXY_URL'),
            proxy_user=crawler.settings.get('STAT_PROXY_USER'),
            proxy_pass=crawler.settings.get('STAT_PROXY_PASS'),
        )

    def process_request(self, request, spider):
        request.meta['proxy'] = self.proxy_url
        request.headers['Proxy-Authorization'] = f'Basic {self.proxy_auth}'
```

## Configure settings

Add your proxy credentials and enable the middleware in `settings.py`:

```python theme={null}
# settings.py

STAT_PROXY_URL = 'http://192.168.1.1:3128'
STAT_PROXY_USER = 'myuser'
STAT_PROXY_PASS = 'mypass'

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.StatProxiesMiddleware': 350,
}
```

## Rotating through multiple proxies

To distribute requests across multiple proxies, modify the middleware:

```python theme={null}
import random
import base64

class RotatingProxiesMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        proxy_list = crawler.settings.getlist('STAT_PROXY_LIST')
        return cls(proxy_list)

    def process_request(self, request, spider):
        proxy = random.choice(self.proxies)
        # Format: host:port:user:pass
        host, port, user, password = proxy.split(':')

        request.meta['proxy'] = f'http://{host}:{port}'
        auth = base64.b64encode(f"{user}:{password}".encode()).decode()
        request.headers['Proxy-Authorization'] = f'Basic {auth}'
```

```python theme={null}
# settings.py

STAT_PROXY_LIST = [
    '192.168.1.1:3128:user1:pass1',
    '192.168.1.2:3128:user2:pass2',
    '192.168.1.3:3128:user3:pass3',
]

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.RotatingProxiesMiddleware': 350,
}
```

## Verifying it works

Add a quick spider to confirm your proxy is active:

```python theme={null}
import scrapy

class IPCheckSpider(scrapy.Spider):
    name = 'ipcheck'
    start_urls = ['https://httpbin.org/ip']

    def parse(self, response):
        self.logger.info(f'Proxy IP: {response.json()["origin"]}')
```

Run it:

```bash theme={null}
scrapy crawl ipcheck
```

The logged IP should be your proxy IP, not your real one.

<Warning>
  Make sure to replace the example credentials with your actual proxy details from the Stat Proxies dashboard.
</Warning>

## Related

<CardGroup cols={2}>
  <Card title="Python Requests Library" icon="python" href="/articles/requests-python">
    Simpler Python proxy usage with requests
  </Card>

  <Card title="Blocked Requests" icon="shield" href="/articles/blocked-requests">
    How to handle blocks and improve success rates
  </Card>
</CardGroup>