#randomnews

Someone please save this series ;;

Happy Birthday robots.txt. I'll tell ya, it does NOT seem like it has been 26 years since 1994, but it has. 26 years ago the internet was super simple compared to today's internet. During the the infancy of any new idea many things happen such as discussion, learning and adaptation to name a few and the Internet is no different. Still today new ideas spawn every day just as this one did back in 1994. A robots-txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. It can help you instruct exactly what a search engine crawler can and can not do while visiting your website and indexing its files and pages. I was made aware of this birthday today when I took a look at Bannerlicious.com's Twitter account and saw a tweet by Web Design Museum @WebDesignMuseum announcing the 26th birthday of the robots.txt file creation by Martijn Koster. https://twitter.com/WebDesignMuseum/status/1279043535090716674 robots.txt Many times it's found that there are pages and files that you store or host that you only want those that you have told of the location to have access. If a search engine crawler visits your site and there are no rules for it to follow it will index everything it finds and make it a searchable result when a web user searches a search engine for something matching your files and gets directed to something that you wanted to keep private. Read the full article