#web crawling

Okay, so I'm the Growth Team product manager at my job - so my goal is getting more people to see the site. In this day and age, that means SEO: search engine optimization.

Even with the scales removed (sorry - confidential info!) it's clear things are going up. Well... until we hit a snag. See, Google doesn't like duplicate content, but Googlebot - the webcrawler that visits your site and reports back - is pretty dumb. It thinks example.com and example.com?session_id=123 are two different sites, even though a human could see this query parameter has nothing to do with the contents of the site. So... we gotta give Googlebot some rules.

Now, I've never bothered to do this for my own projects, but it's best practice to include a top-level robots.txt file so those non-human visitors go where they're supposed to. For example, we recently started disallowing query parameters to eliminate the duplicate page issue.

Uh... as it turns out, due to our site’s setup, one particular query parameter is exceptionally important, specifically for Googlebot. In fact, if the robot visits the page without that, it sees an empty shell. And our search traffic declines, week-over-week:

Luckily, the team figured it out and fixed it right away - in a single line of code! - and this past week had some of our highest-traffic days ever. But let this be a lesson: take extreme care when it comes to the bots!

tl;dr: duplicate content is bad, but no content is worse; use robots.txt, and give the bots a sitemap too; track it all in Google Analytics

Original Equipment Manufacturer is the producer of a vehicle’s original components. This means the parts manufactured by an OEM are identical to the initial parts and have the same quality and compatibility that came with the original product. Automobile owners also have the option of going with aftermarket parts, rebuilt, refurbished or even used parts. Whereas rebuilt or used parts are quite unreliable and hence, less trusted; this leads to a large number of customers going for the aftermarket parts. Aftermarket parts are the ones manufactured by vendors other than the original manufacturer, and are made to function and perform as well as the original. These are priced much lower and hence, pose as great

OEM Parts Monitoring competitors to OEM. Original manufacturers need to be constantly aware of the vendors offering similar products and their pricing. This is where ome to their rescue.

Our Price monitoring services combine latest software tools with data mining and personal insight to generate regular reports on price trends in the market. Our inputs help you make an accurate analysis and informed decisions to maintain your products at competitive rates. We provide data on a daily, weekly, monthly or quarterly basis, as per our client’s requirement.

Web Scraping is another service that is becoming indispensable for any competitive business. Web Scraping is the method of automatically mining data or collecting information from the internet and present it in a format that

is deciphered and easily understood by its consumer. These services help the businesses take informed decisions on their product features, pricing and business road map. These services collate data that is scattered all over the web and transform it into a structured, organised and interrelated information.  

There are a few reasons as to how an OEM industry would benefit from Data Scraping

Access to customer preferences, buying habits and trends can empower the manufacturer to make significantly improved decisions.

Data on trends based on geographical regions help them to build various strategies on marketing for different regions.

The automobile industry is experiencing very advanced technologies, very advanced materials and very advanced structural elements that are changing perpetually. According to Chris Tobie, the collision business specialist at

American Honda, “Because of the rapid body technology changes, two model years of the same vehicle may appear similar but employ very different body structures.” Thus, it would be in the best interest of a competitor to stay abreast of the latest technologies and trends in the industry.

Due to fierce competition in automobile parts industry, the aftermarket manufacturers often come up with unique features trying to outperform the original equipment. For an OEM it becomes crucial to keep an eye out for such trends and keep their products up-to-date with the latest developments.

In design and manufacturing divisions, errors can be more expensive with each successive production stage. A recall on the model spirals the warranty spend on the manufacturer. Here OEM needs data related to the vehicle’s age, performance, and failure scenarios to reduce warranty spend.