Web Scraping With VBA Web Scraping Multiple Pages from Websites Then I remembered Selenium which is software that automates browsers. You can write code that instructs Selenium to do things like open a web page, fill in a form, or click a button, and it's really easy to use. Aug 13, 2018 Even though web scraping is relatively easy to learn and execute, it’s a powerful tool that you can use to collect existing data from websites, then easily manipulate, analyze and store it to your liking; or to automate workflows in web-based platforms. In this tutorial I’ll show you step-by-step: How to set up Selenium in IntelliJ platform.

1. Using Selenium To Web Scrape
2. Web Scraping Java Selenium Examples
3. Scraping With Selenium
4. Selenium Web Scraping Python
5. Web Scraping Using Selenium Python
6. Web Scraping Java Selenium Programming
7. Web Scraping Java Selenium Code

Playwright is a browser automation library for Node.js (similar to Selenium or Puppeteer) that allows reliable, fast, and efficient browser automation with a few lines of code. Its simplicity and powerful automation capabilities make it an ideal tool for web scraping and data mining. It also comes with headless browser support (more on headless browsers later on in the article). The biggest difference compared to Puppeteer is its cross-browser support. In this article, we will discuss:

• Various features of Playwright
• Common web scraping use cases of Playwright
• How playwright compares to others (Cypress vs Playwright vs Puppeteer vs Selenium) ?
• Playwright Pros and Cons

What are headless browsers and why they are used for web scraping?

Before we even get into Playwright let’s take a step back and explore what is a headless browser. Well, a headless is a browser without a user interface. The obvious benefits of not having a user interface is less resource requirement and the ability to easily run it on a server. Since headless browsers require fewer resources we can spawn many instances of it simultaneously. This comes in handy when scraping data from several web pages at once.

The main reason why headless browsers are used for web scraping is that more and more websites are built using Single Page Application frameworks (SPA) like React.js, Vue.js, Angular… If you scrape one of those websites with a regular HTTP client like Axios, you would get an empty HTML page since it's built by the front-end Javascript code. Headless browsers solve this problem by executing the Javascript code, just like your regular desktop browser.

Getting Started with Playwright

The best way to learn something is by building something useful. We will write a web scraper that scrapes financial data using Playwright.

The first step is to create a new Node.js project and installing the Playwright library.

Let’s create a index.js file and write our first playwright code.

In the example above we are creating a new chromium instance of the headless browser. Notice I set headless to false for now (line 4), this will pop up a UI when we run the code. Setting this to true will run Playwright in headless mode. We create a new page in the browser and then we visit the yahoo finance website. We're waiting for 5 seconds and then close the browser.

Let’s hop into the yahoo finance website in our browser. Inspect the home page for yahoo finance. Let’s say we are building a financial application and we would like to scrape all the stock market data for our application. On the yahoo home page, you will see that the top composite market data shows in the header. We can inspect the header element and its DOM node in the browser inspector shown below.

Observe that this header has an id=YDC-Lead-Stack-Composite. We can target this id and extract the information within. We can add the following lines to our code.

page.$eval function requires two parameters. The first one is a selector identifier. In this scenario, we passed in the id of the node we wanted to grab. The second parameter is an anonymous function. Any kind of client-side code that you can think of running inside a browser can be run in this function.

🔎 Let’s take a close look at the*$eval*code block again.

The highlighted portion is simple client-side JS code that is grabbing all the li elements within the header node. Then we are doing some data manipulation and returning it.

You can learn more about this $eval function in the official doc here.

Executing this code prints the following in the terminal.

We have successfully scraped our first piece of information.

Next, let’s scrape a list of elements from a table. We are going to scrape the most actively traded stocks from https://finance.yahoo.com/most-active. Below I have provided a screenshot of the page and the information we are interested in scraping.

As you can see that the id we are interested in is fin-scr-res-table. We can drill down our search to targeting the table element in that DOM node.

Here’s the script that will do the trick,

page.$eval sort of acts like querySelector property of client side JavaScript (Learn more about querySelector).

What if I want to scrape all the tags of a certain type (i.e.a, li) in a webpage?

In such cases, we can simple use the page.$$(selector) function for this. This will return all the elements matching the specific selector in the given page.

Next, let’s scrape some images from a webpage. For this example we will be using our home page scrapingbee.com. Let’s head over there. Take a look at the image below. We will be scraping the image of our friendly robot ScrapingBeeBot here.

The fundamental idea is the same. First we target the DOM node and them grab the image we are interested in. In order to download the image however, we need the image src. Once we have the source we have to make a HTTP GET request to the source and download the image. Let’s dive into the example below.

As you can see above, first we target the DOM node we are interested in. Then on line 11 we are acquiring the src attribute from the image tag. Finally we make a GET request with axios and save the image in our file system.

How about taking screenshots?

We can take a screenshot of the page with Playwright as well. Playwright includes a page.screenshot method. Using this method we can take one or multiple screenshots of the webpage.

Using Selenium To Web Scrape

We can also limit our screenshot to a specific portion of the screen. We have to specify the coordinates of our viewport. Here’s an example of how to do this.

The x and y coordinates starts from the top left corner of the screen.

Querying with XPath expression selectors

Another simple yet powerful feature of Playwright is its ability to target and query DOM elements with XPath expressions. What is an XPath Expression? XPath Expression is a defined pattern that is used to select a set of nodes in the DOM.

The best way to explain this is to demonstrate this with a comprehensive example. Let’s say we are trying to grab all the navigation links from StackOverflow blog. Observe that we want to scrape the nav element in the DOM. We can see that the nav element we are interested in is suspended in the tree in the following hierarchy html > body > div > header > nav

Using this information we can create our xpath expression. Our expression in this case will be xpath=//html/body/div/header/nav.

Here’s the script that will use the xpath expression to target the nav element in the DOM.

The XPath engine inside Playwright is equivalent to native Document.evaluate() expression. You can learn more about it here.

ℹ️ For additional information on XPath read the official Playwright documentationhere.

There will be times when we would want to scrape a webpage that is authentication protected. Now, one of the benefit of Playwright is that it makes it really simple to submit forms. Let’s dive into an example of this scenario.

As you can see in the example above we can easily simulate clicks and form fill events. Running the above script will result in something like below.

ℹ️ Checkout the official docs to learn more about authentication with playwright

How does Playwright compare to some of the other known solutions such as Puppeteer and Selenium? The main selling point of Playwright is the ease of usage. It is very developer-friendly compared to Selenium. Puppeteer on the other hand is also developer-friendly and easy to set up; therefore, Playwright doesn’t have a significant upper hand against Puppeteer.

Let’s take a look at the npm trends and popularity for all three of these libraries.

Web Scraping Java Selenium Examples

You can see that Puppeteer is clearly the most popular choice among three. However, looking at the GitHub activity of these libraries, we can conclude both Playwright and Puppeteer has a strong community of open source developers behind it.ℹ️ source https://www.npmtrends.com/playwright-vs-puppeteer-vs-selenium

How about documentation?
Both Puppeteer and Playwright has excellent documentation. Selenium on the other hand has a fairly good documentation, but it could have been better.

Performance comparison

Doing a fined grained comparison of these three frameworks is beyond the scope of this article. Luckily for us, other people have already done this before. You can take a look at this detailed article for a performance comparison of these tools. When we ran the same scraping script in all these three environments we experience a longer executing time in Selenium compared to Playwright and Puppeteer. Puppeteer and Playwright performance was almost identical to most of the scraping jobs we ran. However, looking at various performance benchmarks (more fined tuned ones like the link above) it seems like Playwright does perform better in few scenarios than Puppeteer.

Finally, here’s a summary of our comparison of these libraries.

Category	Playwright	Puppeteer	Selenium Web Driver
Execution time	Fast and Reliable	Fast and Reliable	Slow startup time
Documentation	Excellent	Excellent	Overall fairly well documented with some exception.
Developer Experience	Very Good	Very Good	Fair
Community	Small but active community.	Has a large community with lots of active projects.	Has a large and active community

To summarize, Playwright is a powerful headless browser, with excellent documentation and a growing community behind it. Playwright is ideal for your web scraping solution if you already have Node.js experience, want to get up and running quickly, care about developer happiness and performance. I hope this article gave you a good first gleam of Playwright. That’s all for today and see you next time.

By Rob Gravelle

In the Web Page Scraping with jsoup article I described how to extract data from a web page using the open-source jsoup Java library. As an HTML parser, jsoup only sees the raw page source and is completely unaware of any content that is added to the DOM via JavaScript after the initial page load. For that, you need to employ some kind of embedded browser engine, such as Oracle WebView.

Another approach is to use a headless browser, that is, a web browser without a graphical user interface. There are several good ones to choose from, including Chrome, Firefox, PhantomJS, Zombie JS, HtmlUnit, and Splash. I today's article, we'll be automating the Chrome headless browser from a Python script to fetch a web page and read the dynamically generated contents of an element.

Scraping With Selenium

Project Setup

Python is an ideal language for web page scraping because it's more light-weight that full-fledged languages like Java. There is also a Selenium WebDriver for python. (Actually, there is one for Java as well!) The driver is basically the engine that runs the browser, much like a database driver.

I'll be developing the script within MS Visual Studio Code. It just makes everything much easier. Here's a great tutorial on getting started with python in Visual Studio Code.

Once you've got the python extension set up, you're ready to create the project.

1. Select File > Add Folder to Workspace… from the main menu.
2. Browse the root folder of your project. Mine is simply called 'demo'.
3. Now we'll need to download the ChromeDriver - WebDriver for Chrome. Place the chromedriver executable in your project root.
4. Next, we'll install Selenium. In the Integrated Terminal (select View < Integrated Terminal from the main menu if it is not already open), type 'pip install selenium' at the prompt and hit the Enter key.

The Demo Page

We'll be testing our script on a very simple web page. It has one paragraph element whose text which will be updated via JavaScript. You can either host the page on a server or, to keep things really simple, just save it locally to your project folder!

The Script

Our script will test the above page by loading it into the headless chrome browser, fetching the '#dynamic-text' element, and printing its innerHTML to the console. If it says, 'JavaScript rendered content,' then we've got the JS-rendered text. Otherwise, it might be time to revisit this whole solution!

1. Create a new file named 'page_scraping_demo.py' in your project root. Visual Studio Code will immediately recognize it as a python script.
2. Add the following code to the file and save your changes.

Running the Script

It's time to put our code to work.

Right-click anywhere in the script editor and select 'Run Python File in Terminal' from the popup menu. A browser window will open and load the page:

Selenium Web Scraping Python

The script's output will be displayed in the terminal:

Selenium Registry Issue in Windows

There is a known selenium issue in Windows that causes the following error to appear in the terminal:

To fix it, you'll have to go into regedit.exe and add required keys as per the instructions provided by Jari Mäkeläinen (aka jarmake):

Web Scraping Using Selenium Python

1. Open the registry with regedit (just click on Windows start menu and start typing regedit, it should come up)
2. From the registry explorer, expand HKEY_LOCAL_MACHINE, and from there expand SOFTWARE
3. Expand Policies. (I was missing everything from this point.)
4. So what I had to do was to select the Policies by left clicking on it, and then right click and from the context menu select New > Key and name it Google.
5. Once that is created, select that and right click and again select New > Key and name that Chrome.
6. Select that folder, and right click. Choose New > String. Name it 'MachineLevelUserCloudPolicyEnrollmentToken' and leave the value empty. (I set mine to '2')
   If you already have Google and Chrome under Policies, I think only adding the key as in step 6 should work. I've attached a picture to show how it should look like when it's done.
   And just to clarify, this is under SOFTWARE/Policies, not directly under SOFTWARE.

You can download the files that we worked on today from GitHub.

Going Forward…

Now that we've gotten our feet wet with python, selenium, and the chrome headless browser, we'll tackle a more complex example next time that illustrates how to gather data from a dynamically generated page.

Web Scraping Java Selenium Programming

Rob Gravelle resides in Ottawa, Canada. His design company has built web applications for numerous businesses and government agencies. Email him.

Rob's alter-ego, 'Blackjacques', is an accomplished guitar player, who has released several CDs and cover songs. His band, Ivory Knight, was rated as one of Canada's top hard rock and metal groups by Brave Words magazine (issue #92).

Web Scraping Java Selenium Code