#webcrawlers

A web page is created in an attempt to be searched, crawled, indexed, and appear in search engine results. Search engines crawl to detemines what is on a webpage. They crawl by downloading robots.txt files instructing search engines which ones to crawl and those  they should not crawl. The information about the sitemap is located in robots.txt. Search engine use algorithms to crawl and recrawl the site with the number of pages to be indexed. Crawling supports SEO by following links that enable them to unveil webpages on the internet that are interlinked from one page to another. Search engines crawl and index every URL that gets submitted on the sitemap. However, a URL with an image or a video makes it difficult for a search engine to discover what is in rather than comprehending metadata and filename. Crawlers through bots work like an interconnected spider web network finding information from one page to another and identifying by links. 

Search engine crawl and index  

Whenever a webpage is up on the web, it needs to be crawled and  indexed for its relevancy to appear on the search engine results page. If you have a travel website, it is relevant for a traveler to find useful information on your webpage. The higher your site is relevant with important information, the greater it ranks on search engines while addressing the pain point of a user.  

Factors that affect web page crawling  

∙A brand new site that has not been crawled yet 

∙No external websites linking to the site 

∙The site’s navigation makes it difficult for the robot to crawl 

∙small bits of code that are crawlers block search engines 

∙The site is penalized for malicious techniques 

∙A cluttered information a webpage 

Remove unnecessary redirects  

Crawling also improves link-building through backlinks. Refrain from using unnecessary redirects. Redirect usually occurs with 301 redirecting an old URL to a new URL. Unnecessary redirects enhance load time as they may prevent the crawling and indexing of the relevant URL and drive away the target user. 

Avoid broken links  

Limit broken links whether external or internal as this may increase the crawl budget. Crawl budget is Google or any other search engine crawling the number of pages on your site.  Factors that support SEO through crawling. 

Authority from others’ referrals  

If your site has referrals from authoritative and trustworthy web pages, your website authority increases. The quality of links to your website determines your authority. This signifies that your webpage is relevant, trustworthy, and necessary. 

Crawling ensures engagement based on  

· clicks and time an audience spends on a page.  

· Bounce rate 

by Ben Dean

_"The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning;"_

From _A Mathematical Theory of Communication_, Claude Shannon

Sometime in the first half of the first decade of the third millennium of the common era, I encountered a web page that, in retrospect, had a deep influence on my thinking about software bots, communication, and meaning. The page in question was a Google SERP, a search engine results page. The particular search term that led Google's PageRank algorithm to assemble those ten blue links in the order that it did is lost to me, though likely not to Google—but suffice to say, it must have been some high-entropy phrase. The results were all long pages of apparently procedurally-generated text. Any literate human would immediately discern the utter absence of meaning in the strings of uncommon words, liberally salted with links to other pages with the same high-entropy, low-content structure, but at the time, only a few autonomous software agents could make the distinction.

This was my first awareness of the bot ecosystem. Before the auto-following, friend-requesting, auction-bidding, market-making bots, I found this odd corner of the web. On one side, PageRank's spiderbots indexing web pages by search terms and following links indiscriminately. On the other, bots pushing the idea of "content provider" to the breaking point by reducing content to its information-theoretic definition. And, crucially, in the middle were the ad bots, bidding on search terms to get their master's sales pitch in front of as many potential customers as possible. This degenerate case of logical, though useless, software-mediated communication seems to have been a sort of perverse inspiration for a number of artistic experiments.

#####Wittgenbot

My first foray into the world of (pure software) bots was, truth be told, little more than a smarmy joke. My (autonomous software agent would)[https://twitter.com/LudWitgen], at quasi-random intervals, tweet a segment of Ludwig Wittgenstein's famed _Tractatus_. I perhaps flatter myself that the joke is clear enough that letting the thing die shortly after its creation was still sufficient for my purposes. The reality is that that this piece functions just fine as a description of the thing, and there is little call for the thing itself. There is no connection—that's the joke—but this means that there's no ecosystem, either.

#####I Am Unable

[I Am Unable to Tell You](http://turbulence.org/project/i-am-unable-to-tell-you/#), now out of service, was not so much a bot in and of itself in the strictest sense, but rather an attempt to explore ideas about web ecosystems with the runaway feedback loop I had found years earlier. Here was a site where the content was only accessible to users who were themselves creating the content by looking at the prior content. The site would create a video using the visitor's webcam while displaying a video created in the same way, earlier. There was no sound, and not much happened, yet people kept clicking next, and the tree of videos grew. The visitors were instrumentalized to become my content-creation bots, using our common desire for meaning and connection as bait.

#####The Perplex

[The Perplex](https://github.com/bd/perplex) is a bot that constructs languages by writing in them. These are, of course, not real languages in any but the most formal sense. Hearkening back to the Wittgenbot, the Perplex is an attempt to make _really good_ gibberish: neither random nor predictable, and certainly meaningful, at least on the level of sense. I sought to invert the logic of _I Am Unable to Tell You_, while maintaining my own authorial absence. Could I create the forms of human connection and communication without any meaning at all? Without any input from the world beyond the noise of random numbers? What would it look like? Using tools most commonly employed by bots to extract meaning (parsers, stochastic n-gram analysis, etc.), could I make a document that looked like it would have meaning, while remaining inherently empty?