1. Cheerio

Who is this for: NodeJS developers who want a straightforward way to parse HTML.

Why you should use it: Cheerio offers an API similar to jQuery, so developers familiar with jQuery will immediately feel at home using Cheerio to parse HTML. It is blazing fast, and offers many helpful methods to extract text, html, classes, ids, and more. It is by far the most popular HTML parsing library written in NodeJS.

2. Beautiful Soup

Who is this for: Python developers who just want an easy interface to parse HTML, and don't necessarily need the power and complexity that comes with Scrapy.

Why you should use it: Like Cheerio for NodeJS developers, Beautiful Soup is by far the most popular HTML parser for Python developers. It's been around for over a decade now and is extremely well documented, with many tutorials on using it to scrape various website in both Python 2 and Python 3.

3. Puppeteer

Who is this for: Puppeteer is a headless Chrome API for NodeJS developers who want very granular control over their scraping activity.

Why you should use it: As an open source tool, Puppeteer is completely free. It is well supported and actively being developed and backed by the Google Chrome team itself. It is quickly replacing Selenium and PhantomJS as the default headless browser automation tool. It has a well thought out API, and automatically installs a compatible Chromium binary as part of its setup process, meaning you don't have to keep track of browser versions yourself.

4. Content Grabber

Who is this for: Enterprises looking to build robust, large scale web data extraction agents.

Why you should use it: Content Grabber provides end to end service, offering knowledgeable experts to develop custom web data extraction solutions for experts. In addition, they offer training and implementation services, as well as the ability to host their solutions on premise.

5. Mozenda

Who is this for: Enterprises looking for a cloud based self serve web scraping platform need look no further. With over 7 billion pages scraped, Mozenda has experience in serving enterprise customers from all around the world.

Why you should use it: Mozenda allows enterprise customers to run web scrapers on their robust cloud platform. They set themselves apart with the customer service (providing both phone and email support to all paying customers). Its platform is highly scalable and will allow for on premise hosting as well.

1. Scraper API

Who is this for: Scraper API is a tool for developers building web scrapers, it handles proxies, browsers, and CAPTCHAs so developers can get the raw HTML from any website with a simple API call.

Why you should use it: Scraper API is a tool for developers building web scrapers, it handles proxies, browsers, and CAPTCHAs so developers can get the raw HTML from any website with a simple API call. It doesn't burden you with managing your own proxies, it manages its own internal pool of over a hundreds of thousands of proxies from a dozen different proxy providers, and has smart routing logic that routes requests through different subnets and automatically throttles requests in order to avoid IP bans and CAPTCHAs. With special pools of proxies for scraping Amazon and other ecommerce listings, Google and other search engine results, Yelp and other review sites, and Twitter and Facebook and other social media sites, web scraping has never been this easy!

2. CSV Scraper

Who is this for: CSV Scraper is a fantastic tool for people who want to extract data from websites without having to code.

Why you should use it: CSV Scraper is the perfect tool for people who want to scrape websites without learning to code. It includes a point and click interface, allowing users to scrape behind login forms, fill in forms, input search terms, scroll through infinite scroll, render javascript, and more. It's fully hosted, allowing users to run their scrapers in the cloud. Best of all, it comes with a generous free tier allowing users to test the service on up to 200 pages. While it's currently in private beta, you can sign up below to be invited to the public beta, which will begin near the end of March 2019.

3. ParseHub

Who is this for: Parsehub is an incredibly powerful tool for building web scrapers without coding. It is used by analysts, journalists, data scientists, and everyone in between.

Why you should use it: Parsehub is dead simple to use, you can build web scrapers simply by clicking on the data that you want. It then exports the data in JSON or Excel format. It has many handy features such as automatic IP rotation, allowing scraping behind login walls, going through dropdowns and tabs, getting data from tables and maps, and much much more. In addition, it has a generous free tier, allowing users to scrape up to 200 pages of data in just 40 minutes!

4. Scrapy

Who is this for: Scrapy is an open source tool for Python developers looking to build scalable web crawlers. It handles all of the plumbing (queueing requests, proxy middleware, etc.) that makes building web crawlers difficult.

Why you should use it: As an open source tool, Scrapy is completely free. It is battle tested, and has been one of the most popular Python libraries for years. It is well documented and there are many tutorials on how to get started. In addition, deploying the crawlers is very simple and reliable, the processes can run themselves once they are set up.]

5. Diffbot

Who is this for: NodeJS developers who want a straightforward way to parse HTML.

Why you should use it: Cheerio offers an API similar to jQuery, so developers familiar with jQuery will immediately feel at home using Cheerio to parse HTML. It is blazing fast, and offers many helpful methods to extract text, html, classes, ids, and more. It is by far the most popular HTML parsing library written in NodeJS.

To scrape a search engine successfully the two major factors are time and amount.

The more keywords a user needs to scrape and the smaller the time for the job the more difficult scraping will be and the more developed a scraping script or tool needs to be.

Scraping scripts need to overcome a few technical challenges:

IP rotation using Proxies (proxies should be unshared and not listed in blacklists)

Proper time management, time between keyword changes, pagination as well as correctly placed delays Effective longterm scraping rates can vary from only 3–5 requests (keywords or pages) per hour up to 100 and more per hour for each IP address / Proxy in use. The quality of IPs, methods of scraping, keywords requested and language/country requested can greatly affect the possible maximum rate.

Correct handling of URL parameters, cookies as well as HTTP headers to emulate a user with a typical browser

HTML DOM parsing (extracting URLs, descriptions, ranking position, sitelinks and other relevant data from the HTML code)

Error handling, automated reaction on captcha or block pages and other unusual responses.

An example of a open source scraping software which makes use of the above mentioned techniques is GoogleScraper.This framework controls browsers over the DevTools Protocol and makes it hard for Google to detect that the browser is automated.

Programming languages

When developing a scraper for a search engine almost any programming language can be used but depending on performance requirements some languages will be favorable.

PHP is a commonly used language to write scraping scripts for websites or backend services, it has powerful capabilities built in (DOM parsers, libcURL) but its memory usage is typical 10 times the factor of a similar C/C++ code. Ruby on Rails as well as Python are also frequently used to automated scraping jobs. For highest performance C++ DOM parsers should be considered.

Even bash scripting can be used together with cURL as command line tool to scrape a search engine.

Search engine scraping is the process of harvesting URLs, descriptions, or other information from search engines such as Google, Bing or Yahoo. This is a specific form of screen scraping or web scraping dedicated to search engines only.

Most commonly larger search engine optimization (SEO) providers depend on regularly scraping keywords from search engines, especially Google, to monitor the competitive position of their customers' websites for relevant keywords or their indexing status.

Search engines like Google do not allow any sort of automated access to their service[1] but from a legal point of view there is no known case or broken law.

The process of entering a website and extracting data in an automated fashion is also often called "crawling". Search engines like Google, Bing or Yahoo are getting almost all their data from automated crawling bots.

Difficulties

Google is the by far largest search engine with most users in numbers as well as most revenue in creative advertisements, this makes Google the most important search engine to scrape for SEO related companies.

Google does not take legal action against scraping, likely for self-protective reasons. However Google is using a range of defensive methods that makes scraping their results a challenging task.

Google is testing the User-Agent (Browser type) of HTTP requests and serves a different page depending on the User-Agent. Google is automatically rejecting User-Agents that seem to originate from a possible automated bot. A typical example would be using the command line browser cURL, Google will directly reject to serve any pages to it while Bing is a bit more forgiving, Bing does not seem to care about User-Agents.

Google is using a complex system of request rate limitation which is different for each Language, Country, User-Agent as well as depending on the keyword and keyword search parameters. The rate limitation can make it unpredictable when accessing a search engine automated as the behaviour patterns are not known to the outside developer or user.

Network and IP limitations are as well part of the scraping defense systems. Search engines can not easily be tricked by changing to another IP, while using proxies is a very important part in successful scraping. The diversity and abusive history of an IP is important as well.

Offending IPs and offending IP networks can easily be stored in a blacklist database to detect offenders much faster. The fact that most ISPs give dynamic IP addresses to customers requires that such automated bans be only temporary, to not block innocent users.

Behaviour based detection is the most difficult defense system. Search engines serve their pages to millions of users every day, this provides a large amount of behaviour information. A scraping script or bot is not behaving like a real user, aside from having non-typical access times, delays and session times the keywords being harvested might be related to each other or include unusual parameters. Google for example has a very sophisticated behaviour analyzation system, possibly using deep learning software to detect unusual patterns of access. It can detect unusual activity much faster than other search engines.

HTML markup changes, depending on the methods used to harvest the content of a website even a small change in HTML data can render a scraping tool broken until it was updated.

General changes in detection systems. In the past years search engines have tightened their detection systems nearly month by month making it more and more difficult to reliable scrape as the developers need to experiment and adapt their code regularly.