Crawling, Parsing, Scraping Data.

    I have created a site http://www.safehavn.com for collecting information about products selling on Amazon in CSV files. The site is powered by Django using requests, unirest libraries. The hosting is a free OpenShift cloud.

Extraction Lawyers Data From South Carolina Bar Site Using Selenium.

Oleg Volodin

2015-03-11 19:09

           Comments                                    

This post is devoted to extracting lawyers information from the South Carolina Bar site. The url of the staring page is http://www.scbar.org/MemberResources/MemberDirectory.aspx. As you can see from the extension, the site was developed using ASP (Active Server Pages) technology with a generous amount of dynamic parts of the web application. The best and the quickest solution to grab data from this kind of sites is Selenium.

Analyzing DOM of the starting page, we can detect that the lookup form of a lawyer member appears in the page through the iframe url 'https://www.scbar.org/cvweb_scbar/cgi-bin/utilities.dll/OpenPage?wrp=lawyerSearch.htm'. It will be our base_url from which we start the search.

In the beginning we import all necessary packages.

from selenium import webdriver from selenium.webdriver.common.keys import Keys import csv

In the following code we create the context of a CSV file which the script will write the output in. Also we instantiate a Selenium webdriver. As opposed to the one of the previous post in which I used headless browser PhantomJS, now I chose a common Selenium browser, the Firefox webdriver, both for fun and visibility of webdriver actions.

base_url = 'https://www.scbar.org/cvweb_scbar/cgi-bin/utilities.dll/OpenPage?wrp=lawyerSearch.htm' with open('lawyers.csv', 'w') as f:    file = csv.writer(f)    file.writerow(["Name", "Company", "Address", "City", "Email", "Phone"])    browser = webdriver.Firefox()

Next we get a page from the base_url and find an element called 'STATE'. Then we send 'SC' to get information about South Carolina lawyers and click on Search button.

browser.get(base_url) elem = browser.find_element_by_name('STATE') elem.send_keys('SC') browser.find_element_by_xpath("//input[@type='button']").click()

Afterwards the browser follows all 15 links representing the list of lawyers and extracts information about them using selenium method find_element_by_xpath. The processed data about every lawyer dumped into CSV file.

while browser.find_elements_by_xpath("/html/body/table/tbody/tr/td/table[1]/tbody/tr/td[3]/a"):     for i in xrange(1, 16):          try:              info = browser.find_element_by_xpath("//tr[{0}]/td[1]/a".format(i))              info.click()              try:                  lawyers = str(browser.find_element_by_xpath("/html/body/table/tbody/tr/td/table/tbody/tr[1]/td[2]").text)                  ls = [lawyer.strip().split(',') for lawyer in lawyers.splitlines()]                  file.writerow([''.join(ls[0]), ''.join(ls[1]), ''.join(ls[2]), ''.join(ls[4]), ''.join(ls[5]), ''.join(ls[6])])              except:                  pass              browser.back()          except:              pass

Finally, we click on the next page and grab information about the following 15 lucky lawyers and so on.

try:   centers = browser.find_element_by_xpath("/html/body/table/tbody/tr/td/table[1]/tbody/tr/td[3]/a").click() except:   pass

In this post we will extract the list of products and prices from the National Instruments site,one of the leading manufacturer of industrial automation tools both hardware and software.

In the following code we use a CrawlSpider class as a base class for our spider because of the convenient method itsuggests to follow links and extract data.

class LabviewSpider(CrawlSpider):     name = 'national_instruments'     allowed_domains = ['sine.ni.com', 'ni.com']     start_urls = ['http://www.ni.com/help/map.htm']

Selenium and Python vs 1mln. lawyers.

Oleg Volodin

2015-03-05 20:07

           Comments                                    

This is the second post about extracting movie information from the site Justdial. The first one was devoted to scraping the same data using Selenium. This post describes the abilities of the Scrapy framework to extract data including dynamically loaded Javascript objects.

Scrapy is the great framework for crawling both the whole site and the only part of it. Along with crawling, Scrapy can be used for extracting, processing and saving data. Moreover, Scrapy uses Twisted asynchronous networking Python library.

Unfortunately, Scrapy cannot download the part of a webpage that is loaded by means of JSON. In this post I'll show how to circumvent this flaw.

In the beginning, we should create a spider in the Scrapy standart spider folder. Read more