#scrapy framework

Oleg Volodin

2015-03-05 20:07

           Comments                                    

This is the second post about extracting movie information from the site Justdial. The first one was devoted to scraping the same data using Selenium. This post describes the abilities of the Scrapy framework to extract data including dynamically loaded Javascript objects.

Scrapy is the great framework for crawling both the whole site and the only part of it. Along with crawling, Scrapy can be used for extracting, processing and saving data. Moreover, Scrapy uses Twisted asynchronous networking Python library.

Unfortunately, Scrapy cannot download the part of a webpage that is loaded by means of JSON. In this post I'll show how to circumvent this flaw.

In the beginning, we should create a spider in the Scrapy standart spider folder. Read more

Oleg Volodin

2015-02-25 20:07

           Comments                                    

I have created this blog for making notes to myself and other persons who are interested in scraping information from the Internet sites, automating routine operations using different programmable tools.

The first task I decided to tackle was scraping some information from the site Justdial. I have to grab all movies, showtimes, information on Chennai cinema theaters like name, address and telephone number. To get this information, I need analyze the first page of the cinema theaters that justdial.com return us. The address of this page is http://www.justdial.com/Chennai/Cinema-Halls/ct-7451/page-1. Using Firebug tools of the Firefox browser I can get all necessary html tags that contain useful information.

Unfortunately, when I examined the page of a particular cinema theater thoroughly, I noted that movies titles and showtimes appeared on the page by means of JavaScript. Luckily, pythonists have Selenium, a great framework for getting, testing and analyzing web pages. So, before parsing a web page with information loaded by JavaScript, we have to use a webdriver Selenium suggests. For my task I chose the PhantomJS headless browser. BeautifulSoup is the next great python library that helps me to parse the DOM tree the Selenium webdriver returned.

Armed with this information, let's start coding.