#xbyte

Amazon is a well-known e-commerce platform with a large amount of data available in various formats on the web. This data can be invaluable for gaining business insights, particularly by analyzing product reviews to understand the quality of products provided by different vendors.

In this guide we will look into web scraping steps to extract amazon reviews of a particular product and save it in excel or csv format. Since manually copying information online can be tedious, in this guide we’ll focus on scraping reviews from Amazon. This hands-on experience will enhance our practical understanding of web scraping techniques.

Before we start, make sure you have Python installed in your system, you can do that from this link: python.org. The process is very simple, just install it like you would install any other application.

Now that everything is set let’s proceed:

Amazon provides many services to the Prime members. There’s presently no way of just exporting product data from Amazon to the spreadsheet for business requirements that you may have. Either to do comparison shopping, competitor research, or building an API for the app projects.

It’s evident that web scraping can easily solve this problem.

Amazon product data scraping will permit you to choose particular data that you’d wish from Amazon site into a JSON or spreadsheet file. You may even make an automated procedure, which runs on the daily, weekly, monthly basis for constantly updating your data.

The room sharing and rentals business are expected to stand out in the hotel business by 2020. Having superior profit margins with progressively growing markets, the vacation rentals might stay for a very long time. This race has become harder and it is the main reason why vacation rentals companies are looking for newer price monitoring services techniques. Scrape Google Hotels by region is among the maximum return activities that offer businesses benefits in the competition.

We have to convert all these elements into easy JavaScript objects that stores the XY position value, font size and text content that looks anything like {'text':'Tennis Ball', 'fontSize':'14px', 'x':100,'y':200}. Therefore, we need to write some functions for that like given below:

Writing new scrapers for different e-commerce websites can be very tedious and expensive. This tutorial will demonstrate a few advanced methods to create a simple scraper, which could recognize patterns as well as bring data from an e-commerce page.

Let’s take a quick look at some product pages as well as identify some design patterns about how the product prices get displayed on different sites.

Sephora.com

Amazon.com

Patterns and Observations

Certain patterns, which we recognized by searching at the product pages include:

Price generally comes above further currency figures

Price is a currency figure having the biggest font sizes

Prices look like currency figures (never like words)

Prices comes within initial 600 pixels of height

Certainly, there might be exemptions to these comments, we’ll chat how to cope with these exemptions later in the blog. We can use all the observations to make an effective and general scraper.

Execution of General E-commerce Scrapers

1st Step: Installation

Here, the tutorial utilizes Google Chrome as a web browser. In case, you are not using it, you can just install it and follow the instruction.

Rather than Google Chrome, the developers use programmable versions of the Google Chrome named Puppeteer. It will eliminate the requirement of running GUI apps to run a scraper. Though, it is outside the range of the tutorial.

2nd Step: Chrome Developer Tool

Different codes presented here are designed in as easy as possible manner so it can’t fetch the prices from all product pages available there.

For the meantime, we’ll visit any Sephora or Amazon product pages in the Google Chrome browser.

Visit that product pages in the Google Chrome

Then right-click anyplace on a page to choose ‘Inspect’ option and open Chrome DevTools

Then click on a DevTools’ Console tab

Within a Console tab, enter some JavaScript codes and browser will accomplish the codes in context of a web page, which have been loaded. Also, you can study more about the DevTools through the official documentation.

3rd Step: Running a Javascript snippet

You need to copy this JavaScript snippet given below and paste that in a console.

let elements = [ ...document.querySelectorAll(' body *') ]

function createRecordFromElement(element) {

const text = element.textContent.trim()

var record = {}

const bBox = element.getBoundingClientRect()

if(text.length <= 30 && !(bBox.x == 0 && bBox.y == 0)) {

record['fontSize'] = parseInt(getComputedStyle(element)['fontSize']) }

record['y'] = bBox.y

record['x'] = bBox.x

record['text'] = text

return record

}

let records = elements.map(createRecordFromElement)

function canBePrice(record) {

if( record['y'] > 600 ||

record['fontSize'] == undefined || !record['text'].match(/(^(US ){0,1}(rs\.|Rs\.|RS\.|\$|₹|INR|USD|CAD|C\$){0,1}(\s){0,1}[\d,]+(\.\d+){0,1}(\s){0,1}(AED){0,1}$)/) )

return false

else return true

}

-

let possiblePriceRecords = records.filter(canBePrice)

let priceRecordsSortedByFontSize = possiblePriceRecords.sort(function(a, b) {

if (a['fontSize'] == b['fontSize']) return a['y'] > b['y']

return a['fontSize'] < b['fontSize']

})

console.log(priceRecordsSortedByFontSize[0]['text']);

Press the ‘Enter’ key and you will see the product price displayed on a console.

If you don’t do that, you have perhaps visited the product page that is an exemption to our explanations. It is completely common, we’ll chat how we can increase our script for covering more product pages about these types. You can try any sample pages given in the step 2.

This animated GIF given below indicates how we extract the prices from Amazon.com

How Does It Work?

First, we need to draw all the HTML DOM elements in a page

let elements = [ ...document.querySelectorAll(' body *') ]

We have to convert all these elements into easy JavaScript objects that stores the XY position value, font size and text content that looks anything like {'text':'Tennis Ball', 'fontSize':'14px', 'x':100,'y':200}. Therefore, we need to write some functions for that like given below:

function createRecordFromElement(element) {

const text = element.textContent.trim() // Brings content of an element

var record = {} // Starts an easy JavaScript object

const bBox = element.getBoundingClientRect()

// getBoundingClientRect is the function given by Google Chrome, this returns

// an object that comprises x,y values, width and height

if(text.length <= 30 && !(bBox.x == 0 && bBox.y == 0)) {

record['fontSize'] = parseInt(getComputedStyle(element)['fontSize'])

}

// getComputedStyle is the function given by Google Chrome, this returns an

// object having all the style data. As this function is fairly

// time-consuming, we only collect the font sizes of elements those

// length of text content is nearly 30 and whose x as well as y coordinates are not 0

record['y'] = bBox.y

record['x'] = bBox.x

record['text'] = text

return record

}

Now, transform all the collected elements to JavaScript objects through applying the functions on all the elements through JavaScript map functions.

let records = elements.map(createRecordFromElement)

Think about the explanations we made about how the price gets displayed. Now, we can filter those records that match with the design observations. Therefore we require a function, which says whether the given records match with the design observations.

function canBePrice(record) {

if(record['y'] > 600 ||

record['fontSize'] == undefined || !record['text'].match(/(^(US ){0,1}(rs\.|Rs\.|RS\.|\$|₹|INR|USD|CAD|C\$){0,1}(\s){0,1}[\d,]+(\.\d+){0,1}(\s){0,1}(AED){0,1}$)/) )

return false

else return true

}

We use Regular Expression option for checking if the provided text is the currency figures or not. Also, you may modify that regular expression if it doesn’t include any pages, which you’re testing with.

Currently, we may filter only the records, which are perhaps pricing records

let possiblePriceRecords = records.filter(canBePrice)

To conclude, as we’ve witnessed, prices come as a currency figure getting the maximum font size. In case, there are several currency figures having equally higher font sizes, then price perhaps corresponds to one residing with the higher positions. We sort out all our records depending on the conditions, through JavaScript’s sort functions.

let priceRecordsSortedByFontSize = possiblePriceRecords.sort(function(a, b) {

if (a['fontSize'] == b['fontSize']) return a['y'] > b['y']

return a['fontSize'] < b['fontSize']

})

Currently, we just have to show that on a console

console.log(priceRecordsSortedByFontSize[0]['text'])

Take that Further

Affecting to the GUI-less-dependent Scalable Programs

You may replace the Google Chrome having the headless variety of that named Puppeteer. It is perhaps the quickest option for web rendering. This works completely depending on the similar ecosystem given in the Google Chrome. When the Puppeteer is all set, you can programmatically insert our script into a headless browser as well as have the pricing returned to the function in a program.

Improve and Enhance the Scripts

You will immediately notice that a few product pages won’t work with a script as they don’t trail the expectations we have fulfilled about how product prices are displayed as well as the patterns that we have recognized.

Unfortunately, there are no “holy grails” or perfect solutions for that problem. This is quite possible to produce more pages and recognize more patterns as well as improve this scraper.

Another important step, which you would utilize to deal with other pages include employing Artificial Intelligence or Machine Learning dependent methods to recognize and categorize patterns as well as automate the procedure to a bigger amount. This sector is a growing field we at X-Byte are using these methods already with variable degrees of attainment.

If you want any help in Amazon price scraping, you can investigate our tutorial specially intended for Amazon:

We Can Assist With Data and Automation Requirements

Convert the Internet to structured, meaningful, and practical data

Your Name

[email protected]

Please enter data sources, details, requests - everything relevant

You SHOULD NOT contact X-Byte for all help with the Tutorials as well as Codes using a form or through calling us, in its place please add the comments to the end of this tutorial page to get help.

Disclaimer

X-Byte offers Brand and Reputation Monitoring Services For eCommerce, Retail Sector, Social Media and Online forums. We manage the reputation of your brand.

Nowadays, every business needs numerous data to derive its value. Be it structured or unstructured data. Business data plays a vital role in driving growth, development, and innovation for your dream venture. But the data that enterprise possesses come handy only if they are structured and the company can derive value from it. The increasing importance of big data gives birth to the concept of web scraping. With the help of web scraping services, you can extract unrecognized and unmanaged data and convert them into organized and structured formats. This strategy will help you out if your brand is being talked about on multiple platforms like social media, expert forums, comments, etc. Whereas, with the help of scraping tool algorithm, you can fetch only data that contains reference about the brand.

In an online world, “Brand Monitoring” is essential for organization. And the organizations face serious negative consequences as a result, if they do NOT invest in Brand Monitoring services.