Web Scraping in C#: Step-by-Step Tutorial in 2025

Lucas Mitchell
Automation Engineer
23-Jan-2025

Your friendly guide to scraping websites, handling CAPTCHAs, and even taking screenshots!
Why Web Scraping in C#?

C# isn’t just for building Windows apps or games—it’s a powerhouse for web scraping too! With libraries like HtmlAgilityPack, Selenium, and Puppeteer Sharp, you can extract data, automate interactions, and even solve CAPTCHAs (yes, really). In this tutorial, we’ll use https://www.scrapethissite.com/pages/ as our playground. Let’s dive in!
Step 1: Setting Up Your Project

First, create a new C# console app. Then, install these NuGet packages:
bash
Install-Package HtmlAgilityPack # For HTML parsing
Install-Package Selenium.WebDriver # For browser automation
Install-Package PuppeteerSharp # For screenshots & advanced scraping
Install-Package Capsolver.SDK # For CAPTCHA solving
Example 1: Basic Page Scraping
Let’s scrape country data from https://www.scrapethissite.com/pages/simple/.
csharp
using HtmlAgilityPack;
using System.Net;
var url = "https://www.scrapethissite.com/pages/simple/";
var client = new WebClient();
client.Headers.Add("User-Agent", "Mozilla/5.0"); // Be polite!
var html = client.DownloadString(url);
var doc = new HtmlDocument();
doc.LoadHtml(html);
var countries = doc.DocumentNode.SelectNodes("//div[@class='country']");
foreach (var country in countries)
{
var name = country.SelectSingleNode(".//h3").InnerText.Trim();
var capital = country.SelectSingleNode(".//span[@class='country-capital']").InnerText.Trim();
Console.WriteLine($"Country: {name}, Capital: {capital}");
}
This prints all countries and their capitals. Simple, right?
Example 2: Handling JavaScript with Selenium
Some pages need a real browser. Let’s scrape the AJAX example page using Selenium:
csharp
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
var options = new ChromeOptions();
options.AddArgument("--headless"); // Run in background
using var driver = new ChromeDriver(options);
driver.Navigate().GoToUrl("https://www.scrapethissite.com/pages/ajax-javascript/");
var dynamicContent = driver.FindElement(By.CssSelector(".ajax-content")).Text;
Console.WriteLine($"AJAX Content: {dynamicContent}");
Example 3: Solving CAPTCHAs with Capsolver
Got a CAPTCHA blocking your way? Use Capsolver to bypass ReCaptchaV2. Here’s how:
- Sign up for Capsolver, grab your API key.
- Use the API to solve a CAPTCHA:
csharp
using System;
using System.Net.Http;
using System.Threading.Tasks;
using System.Text;
using System.Text.Json;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using System.Threading;
namespace CapSolverSeleniumExample
{
class Program
{
static async Task Main(string[] args)
{
string token = await GetCaptchaToken();
using var driver = new ChromeDriver();
driver.Navigate().GoToUrl("https://www.example.com");
Thread.Sleep(5000);
driver.ExecuteScript("document.getElementById('g-recaptcha-response').innerHTML = arguments[0];", token);
var submitButton = driver.FindElement(By.Id("submit-button"));
submitButton.Click();
Thread.Sleep(5000);
driver.Quit();
}
static async Task<string> GetCaptchaToken()
{
string apiKey = "YOUR_API_KEY";
string siteKey = "6Le-wvkSAAAAAPBMRTvw0Q4Muexq9bi0DJwx_kl-";
string siteUrl = "https://www.example.com";
using var client = new HttpClient();
var payload = new
{
clientKey = apiKey,
task = new
{
type = "ReCaptchaV3TaskProxyLess",
websiteKey = siteKey,
websiteURL = siteUrl,
pageAction = "login"
}
};
var requestContent = new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");
var createTaskResponse = await client.PostAsync("https://api.capsolver.com/createTask", requestContent);
var createTaskResponseString = await createTaskResponse.Content.ReadAsStringAsync();
using var createTaskJsonDoc = JsonDocument.Parse(createTaskResponseString);
var root = createTaskJsonDoc.RootElement;
if (!root.TryGetProperty("taskId", out var taskIdElement))
{
Console.WriteLine("Failed to create task: " + createTaskResponseString);
return null;
}
int taskId = taskIdElement.GetInt32();
while (true)
{
await Task.Delay(1000);
var resultPayload = new { clientKey = apiKey, taskId = taskId };
var resultContent = new StringContent(JsonSerializer.Serialize(resultPayload), Encoding.UTF8, "application/json");
var getTaskResponse = await client.PostAsync("https://api.capsolver.com/getTaskResult", resultContent);
var getTaskResponseString = await getTaskResponse.Content.ReadAsStringAsync();
using var getTaskJsonDoc = JsonDocument.Parse(getTaskResponseString);
var resultRoot = getTaskJsonDoc.RootElement;
if (resultRoot.TryGetProperty("status", out var statusElement))
{
string status = statusElement.GetString();
if (status == "ready")
{
if (resultRoot.TryGetProperty("solution", out var solutionElement) && solutionElement.TryGetProperty("gRecaptchaResponse", out var tokenElement))
{
return tokenElement.GetString();
}
return null;
}
if (status == "failed" || resultRoot.TryGetProperty("errorId", out _))
{
Console.WriteLine("Solve failed! response: " + getTaskResponseString);
return null;
}
}
}
}
}
}
Works seamlessly with Selenium to automate CAPTCHA-heavy forms!
Example 4: Taking Screenshots with Puppeteer Sharp
Want visual proof of your scraping? Capture a screenshot:
csharp
using PuppeteerSharp;
await new BrowserFetcher().DownloadAsync();
using var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = true });
using var page = await browser.NewPageAsync();
await page.GoToAsync("https://www.scrapethissite.com/pages/javascript/");
await page.ScreenshotAsync("screenshot.png");
Perfect for debugging or archiving pages.
Pro Tips for 2025 Web Scraping
- Respect robots.txt: Check https://www.scrapethissite.com/robots.txt first.
- Rate limiting: Add delays with
Thread.Sleep(2000)
to avoid overwhelming servers. - User-Agent rotation: Mimic real browsers to avoid detection.
- Error handling: Wrap code in
try-catch
blocks for network issues.
Compliance Disclaimer: The information provided on this blog is for informational purposes only. CapSolver is committed to compliance with all applicable laws and regulations. The use of the CapSolver network for illegal, fraudulent, or abusive activities is strictly prohibited and will be investigated. Our captcha-solving solutions enhance user experience while ensuring 100% compliance in helping solve captcha difficulties during public data crawling. We encourage responsible use of our services. For more information, please visit our Terms of Service and Privacy Policy.
More

Is Web Scraping Legal? the Comprehensive Guide for 2025
2025 Web Scraping Legal Challenges: Key Regulations, Compliance, and Case Studies

Lucas Mitchell
24-Jan-2025

Top 5 Captcha Solvers for reCAPTCHA Recognition in 2025
Explore 2025's top 5 CAPTCHA solvers, including AI-driven CapSolver for fast reCAPTCHA recognition. Compare speed, pricing, and accuracy here

Lucas Mitchell
23-Jan-2025

What Is reCAPTCHA Recognition? A Beginner's Guide
Struggling with reCAPTCHA image grids? Discover how Capsolver's AI-powered recognition solves 'Select all ' challenges instantly. Learn API integration, browser extensions, and pro tips to automate CAPTCHA solving with 95%+ accuracy

Ethan Collins
23-Jan-2025

What is a reCAPTCHA Site Key and How to Find It?
Learn how to find a reCAPTCHA Site Key manually or with tools like Capsolver. Fix common issues and automate CAPTCHA solving for developers and web scraping.

Rajinder Singh
23-Jan-2025

How to Bypass Cloudflare Challenge While Web Scraping in 2025
Learn how to bypass Cloudflare Challenge and Turnstile in 2025 for seamless web scraping. Discover Capsolver integration, TLS fingerprinting tips, and fixes for common errors to avoid CAPTCHA hell. Save time and scale your data extraction.

AloĂsio VĂtor
23-Jan-2025

How to Solve Image CAPTCHAs in Web Scraping: A Complete Guide for 2025
Learn how to solve image CAPTCHAs effectively with CapSolver in 2025

Ethan Collins
23-Jan-2025