#web-crawler

Resolved: How can I tell how often Google crawls my site? #computers #fix #answer

How can I tell how often Google crawls my site?

I’ve started a relatively new website, and I submitted it to google and everything. I use google’s webmaster tools as well. I’m wondering how to figure out the frequency of google’s spider accessing my website. I always hear people talking in forums about “I’m getting 250 pages crawled an hour” and so on… Is there software to use? A javascript…

1: Normal politeness needed on the web

There is a line in the robots.txt that each domain tells you how often you can crawl that domain.

The problem is that multiple domains can be hosted with a hosted service, and they can be hosted even on one machine.

Therefore even though you are polite to one domain, you are hurting a hosting service, or you are hurting a machine.

So you have to make plans in order to not hit one machine or domain too hard.

2: How to get a bunch of machines involved in crawling

If you crawl on just one machine, and the state is very good, and it is really easy to keep the state of the crawl on one machine.

But then it is hard to make a very big search engine, because you get as much as  one machine will crawl. So of course in practice, you need to crawl on thousands of machines.

If you are going to obey politeness, that means that each of your thousand machines needs to tell the other ones, what it has crawled and what it is about to craw and what it it is going to crawl in 20 min. All of these communication overhead can slow down the crawler, which hurts your aim of having a big search engine.

3: How to consume a lot of bandwidth to keep the expensive resources busy

One thing that people do to lessen the communication is they pre-process the corpus that they already have. So when you have crawled a set of pages you can process them and extract out all the links that they point to. You can then normalize those links so that yahoo.com and xxxxx.yahoo.com wind up being the same string, and then you can farm those strings out to your thousand machines, and then they don't have to communicate because they know they are not going to hit the same domain.

However, they could accidentally hit the same hosting machine, but your can take care of that ahead of time as well.

How to: Does a company have implied right to crawl my website? #computers #fix #dev

Does a company have implied right to crawl my website?

I have found out that McAfee SiteAdvisor has reported my website as “may be having security issues”.

I care little about whatever McAfee thinks of my website (I can secure it myself and if not, McAfee definitely is not the company I’d be asking for help, thank you very much). What bothers me, though, is that they have, apparently, crawled my…

Fix: How are these 'bad bots' finding my closed webserver? #computers #it #programming

How are these ‘bad bots’ finding my closed webserver?

I’ve installed Apache a while ago, and a quick look at my access.log shows that all sorts of unknown IPs are connecting, mostly with a status code 403, 404, 400, 408. I have no idea how they’re finding my IP, because i only use it for personal use, and added a robots.txt hoping it’d keep search engines away. I block indexes and there’s nothing…

Solution: Does a company have implied right to crawl my website? #solution #fix #answer

Does a company have implied right to crawl my website?

I have found out that McAfee SiteAdvisor has reported my website as “may be having security issues”.

I care little about whatever McAfee thinks of my website (I can secure it myself and if not, McAfee definitely is not the company I’d be asking for help, thank you very much). What bothers me, though, is that they have, apparently, crawled my…

How-to: How are these 'bad bots' finding my closed webserver? #it #computers #development

How are these ‘bad bots’ finding my closed webserver?

I’ve installed Apache a while ago, and a quick look at my access.log shows that all sorts of unknown IPs are connecting, mostly with a status code 403, 404, 400, 408. I have no idea how they’re finding my IP, because i only use it for personal use, and added a robots.txt hoping it’d keep search engines away. I block indexes and there’s nothing…