Using Playwright with Ruby: Step-by-Step Guide for 2024

Web scraping has become an essential skill for developers who need to gather data from websites. Playwright, a powerful browser automation tool, is often used for this purpose. In this guide, we will explore how to use Playwright with Ruby to scrape data from a website. We will walk through a practical example using the website Quotes to Scrape.

Prerequisites

Before we begin, make sure you have the following installed on your machine:

• Ruby (Version 2.7 or later)
• Node.js (Playwright requires Node.js to run)
• Playwright Gem (Ruby wrapper for Playwright)

You can install the necessary dependencies by running:

gem install playwright-ruby-client

Setting Up Playwright

After installing the playwright-ruby-client gem, you need to set up Playwright in your Ruby script. Here’s how you can do it:

require 'playwright'

Playwright.create(playwright_cli_executable_path: '/path/to/node_modules/.bin/playwright') do |playwright|
  browser = playwright.chromium.launch(headless: false)
  page = browser.new_page
  page.goto('http://quotes.toscrape.com/')
  
  # Example scraping code will go here
  
  browser.close
end

Replace '/path/to/node_modules/.bin/playwright' with the actual path to the Playwright CLI on your system.

Scraping Quotes from the Website

Now, let's write the code to scrape quotes from the website. We will extract the text of each quote and the corresponding author.

require 'playwright'

Playwright.create(playwright_cli_executable_path: '/path/to/node_modules/.bin/playwright') do |playwright|
  browser = playwright.chromium.launch(headless: false)
  page = browser.new_page
  page.goto('http://quotes.toscrape.com/')
  
  quotes = page.query_selector_all('.quote')

  quotes.each do |quote|
    quote_text = quote.query_selector('.text').text_content.strip
    author = quote.query_selector('.author').text_content.strip
    puts "#{quote_text} - #{author}"
  end

  browser.close
end

Explanation

1. Navigating to the Website: The script navigates to the http://quotes.toscrape.com/ URL using the page.goto method.
2. Selecting Quotes: We use page.query_selector_all('.quote') to select all elements that have the class quote.
3. Extracting Text and Author: For each quote, we extract the text content and the author using the respective selectors.
4. Output: Finally, we print each quote followed by its author to the console.

Running the Script

You can run this Ruby script from your terminal:

ruby playwright_scraper.rb

Make sure to replace playwright_scraper.rb with the filename of your script.

Handling CAPTCHA Challenges with Playwright and Ruby

CAPTCHA challenges are a common obstacle when scraping websites, designed to differentiate between human users and bots. For developers using Playwright with Ruby, overcoming these challenges is essential to successfully automate data extraction. In this guide, we'll explore how to integrate CAPTCHA solving services using CapSolver with Playwright. Depending on the type of CAPTCHA implemented by the website, you can either configure CapSolver via an extension (the simplest method) or through their API for more advanced use cases.

For detailed instructions on setting up the extension, visit CapSolver Extension Documentation. For API integration, refer to the CapSolver API Documentation.

Conclusion

This guide has shown you how to set up Playwright with Ruby and scrape data from a website. The example used here is simple but can be expanded for more complex tasks. Playwright’s ability to automate browser tasks makes it a powerful tool for web scraping and testing.

Happy scraping!