CAPSOLVER
Blog
How to Change User Agent to Solve Cloudflare

How to Change User Agent to Solve Cloudflare

Logo of CapSolver

Anh Tuan

Data Science Expert

14-Jan-2025

Why User Agent Matters for passing Cloudflare

What Is a User Agent?

A User Agent (UA) is a string sent by your browser or client to identify itself to the server. It typically includes details like the browser name, version, operating system, and device type. For example:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36.

Cloudflare analyzes this string to detect bots. If your UA matches known automation tools (e.g., Python-Requests or HeadlessChrome), you’ll trigger Cloudflare’s anti-bot systems.


Step-by-Step: Changing Your User Agent

1. Identify the Target’s Expected User Agent

Before customizing, check what User Agents real users of the site are using. Tools like WhatIsMyBrowser or browser developer tools (Network tab > Headers) can help. For example, if the site is popular with Chrome users, mimic their UA.

2. Set a Custom User Agent in Your Code

Here’s how to modify the UA in popular programming languages:

Python (Requests Library)

python Copy
import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
}

response = requests.get('https://example.com', headers=headers)

JavaScript (Node.js with Axios)

javascript Copy
const axios = require('axios');

axios.get('https://example.com', {
  headers: {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
  }
});

Using Headless Browsers (Puppeteer)

javascript Copy
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36');
  await page.goto('https://example.com');
})();

3. Rotate User Agents

Using the same UA repeatedly can still flag you. Rotate UAs from a predefined list to mimic diverse users. Tools like Fake UserAgent simplify this:

python Copy
from fake_useragent import UserAgent
import requests

ua = UserAgent()
headers = {'User-Agent': ua.random}
response = requests.get('https://example.com', headers=headers)

Common Mistakes to Avoid

  1. Using Outdated User Agents
    Cloudflare maintains lists of suspicious UAs. Avoid strings tied to old browser versions (e.g., Chrome/58.0.3029.110 from 2017).

  2. Ignoring Headless Browser Fingerprints
    Even with a valid UA, headless browsers leak automation signals (e.g., missing plugins like navigator.plugins). Use stealth plugins like puppeteer-extra-plugin-stealth.

  3. Forgetting IP Rotation
    Pair UA rotation with residential proxies to avoid IP-based blocks. Static or sticky proxies work best for maintaining session consistency.

  4. Combine with TLS Fingerprinting
    Cloudflare checks TLS handshake patterns. Libraries like curl_cffi (Python) or tls-client (JavaScript) mimic real browser TLS fingerprints, reducing detection risk.


Final Thoughts

Changing your User Agent is a simple yet powerful way to bypass Cloudflare, but it’s not foolproof. Combine it with IP rotation, TLS fingerprinting, and anti-detection tools for robust results.

Happy scraping! 🤖

Compliance Disclaimer: The information provided on this blog is for informational purposes only. CapSolver is committed to compliance with all applicable laws and regulations. The use of the CapSolver network for illegal, fraudulent, or abusive activities is strictly prohibited and will be investigated. Our captcha-solving solutions enhance user experience while ensuring 100% compliance in helping solve captcha difficulties during public data crawling. We encourage responsible use of our services. For more information, please visit our Terms of Service and Privacy Policy.

More