Best User Agents for Web Scraping & How to Use Them

When performing web scraping, using the wrong user agent can lead to instant blocks. Websites often rely on user agents to differentiate between real users and bots. To avoid detection, it’s crucial to use well-formed and frequently updated user agents in your web scraping projects.

In this guide, you’ll discover:

• What a user agent is and why it matters for web scraping
• A list of the best user agents for scraping
• How to set and rotate user agents in Python
• Additional best practices to prevent getting blocked

Let’s dive in! 🚀

What Is a User Agent?

A User Agent (UA) is a string sent in the HTTP request headers that identifies the browser, operating system, and other details. Web servers use this information to render the appropriate content for the user’s device.

Example of a User Agent String:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36

Breaking It Down:

• Mozilla/5.0 – Browser family
• (Windows NT 10.0; Win64; x64) – Operating system details
• AppleWebKit/537.36 (KHTML, like Gecko) – Rendering engine
• Chrome/123.0.0.0 – Browser version
• Safari/537.36 – Compatibility framework

By modifying the user agent, you can make your web scraper appear as a real browser, reducing the risk of detection.

Why User Agents Are Important for Web Scraping

Most websites analyze user agents to filter out bot traffic. If your scraper sends an invalid or outdated user agent, it may get blocked instantly.

By using an appropriate user agent, you can:

• Mimic a real browser and blend in with normal traffic.
• Bypass anti-bot protections that check for default scraping libraries.
• Improve request success rates and avoid CAPTCHAs or IP bans.

However, using only one user agent repeatedly can still trigger anti-bot systems. This is why rotating user agents is crucial.

Best User Agents for Web Scraping (Updated List)

Below is a curated list of effective user agents for web scraping:

Google Chrome User Agents:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36

Mozilla Firefox User Agents:

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 14.4; rv:124.0) Gecko/20100101 Firefox/124.0
Mozilla/5.0 (X11; Linux i686; rv:124.0) Gecko/20100101 Firefox/124.0

Other Browsers:

Mozilla/5.0 (Macintosh; Intel Mac OS X 14_4_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.2420.81

💡 Tip: You can check your own user agent by visiting WhatIsMyUserAgent.

How to Set a Custom User Agent in Python

Many websites implement bot-detection mechanisms that block requests with missing or incorrect user-agent headers. In this section, Let's go with different ways to set and rotate user agents efficiently.

1. Using the requests Library

The simplest way to define a user agent is by modifying the headers of a request using the popular requests library.

Example: Setting a Static User Agent

import requests

# Define headers with a custom User-Agent
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
}

# Send a request with the custom User-Agent
response = requests.get("https://httpbin.org/headers", headers=headers)

# Print the response headers
print(response.text)

Output:

{
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Host": "httpbin.org",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
  }
}

This confirms that the server correctly receives and recognizes the user agent string.

2. Rotating User Agents for Better Anonymity

Using a single user agent repeatedly can lead to blocks. To avoid this, rotate user agents using a predefined list.

Example: Rotating User Agents with random

import requests
import random

# List of different user agents
user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0"
]

# Select a random user agent
headers = {"User-Agent": random.choice(user_agents)}

# Send a request with the randomly chosen user agent
response = requests.get("https://httpbin.org/headers", headers=headers)

print(response.text)

By rotating user agents, your scraper appears more human-like and reduces the chances of detection.

3. Using fake_useragent for Dynamic User Agent Generation

Instead of maintaining a static list, you can dynamically generate user agents using the fake_useragent library.

Installation:

pip install fake-useragent

Example: Generating Random User Agents

from fake_useragent import UserAgent
import requests

# Create a UserAgent object
ua = UserAgent()

# Generate a random user agent
headers = {"User-Agent": ua.random}

# Send a request with a dynamically generated user agent
response = requests.get("https://httpbin.org/headers", headers=headers)

print(response.text)

This method provides a broader variety of user agents while keeping them updated.

4. Setting a Custom User Agent in Selenium

When using Selenium for web scraping, setting a user agent requires modifying browser options.

Example: Setting a User Agent in Chrome

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# Configure Chrome options
chrome_options = Options()
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36")

# Start browser with custom user agent
driver = webdriver.Chrome(options=chrome_options)

# Open a test page to verify user agent
driver.get("https://httpbin.org/headers")

# Extract and print page content
print(driver.page_source)

driver.quit()

By using browser automation tools like Selenium, you can simulate real user behavior and bypass advanced anti-bot measures.

5. Verifying Your User Agent

To ensure your user agent is correctly set, use the following methods:

1. Check the response headers from https://httpbin.org/headers
2. Use browser developer tools (F12 > Network > Headers) to inspect requests
3. Use logging to confirm user agent rotation in scrapers

Example: Logging User Agents in a Loop

import requests
import random
import time

# User agent list
user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0"
]

# Loop through requests
for i in range(5):
    user_agent = random.choice(user_agents)
    headers = {"User-Agent": user_agent}
    
    response = requests.get("https://httpbin.org/headers", headers=headers)
    print(f"Request {i+1} - User-Agent: {user_agent}")
    
    time.sleep(2)  # Add delay to avoid rate limiting

This script logs different user agents over multiple requests, helping you debug rotation strategies.

How to Rotate User Agents at Scale

Instead of using a single static user agent, it's better to rotate them dynamically to avoid detection. Here’s how you can rotate user agents in Python:

import requests
import random

user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
]

headers = {"User-Agent": random.choice(user_agents)}
response = requests.get("https://httpbin.org/headers", headers=headers)
print(response.text)

This script randomly selects a user agent from the list, making your scraper more difficult to detect.

Additional Best Practices to Avoid Getting Blocked

Even with the best user agents, web scraping requires additional techniques to stay undetected:

• Use proxies to avoid IP bans.
• Implement delays & random intervals between requests.
• Rotate headers and request patterns to mimic human behavior.
• Avoid excessive scraping to prevent triggering rate limits.
• Monitor response codes to detect blocks and adapt accordingly.

Even with user-agent and proxy rotation and all those tips, websites may still implement advanced detection techniques, such as fingerprinting, JavaScript challenges, and Captcha verification. This is where CapSolver comes in.

CapSolver specializes in Solvinf Capttcha challenges, ensuring uninterrupted web scraping. By integrating CapSolver, you can automatically solve CAPTCHAs and keep your scraper running smoothly

Claim Your Bonus Code for top captcha solutions -CapSolver: CAPTCHA. After redeeming it, you will get an extra 5% bonus after each recharge, Unlimited

Conclusion

Using the right user agents is a critical step in web scraping. In this guide, we covered:
✅ What a user agent is and how it works
✅ A list of effective user agents for scraping
✅ How to set and rotate user agents in Python
✅ Additional best practices to stay undetected

By combining user agent rotation with other anti-detection techniques, you can successfully scrape data without getting blocked.

FAQ

1. What is a user agent in web scraping?
A user agent is a string that identifies the browser or client software to a web server. In web scraping, it's used to mimic real user activity and avoid detection.

2. Is web scraping for personal use illegal?
Web scraping is generally legal for personal use, but you must respect a website’s terms of service and avoid scraping sensitive or copyrighted data.

3. What is the purpose of user agent rotation in web scraping?
User agent rotation helps avoid detection and blocking by making requests appear to come from different browsers or devices.

4. How can I prevent being blocked while web scraping?
To avoid blocking, use IP rotation, CAPTCHA-solving, delays between requests, and ensure compliance with the site’s robots.txt.

5. Can web scraping impact a website’s performance?
Yes, scraping too frequently can overload a website’s server. It’s important to scrape responsibly with limited requests.