Robots.txt is a text formatted file (.txt) that must be owned every internet site that is registered on search engines like google, yahoo, and bing. So robots.txt is very closely related to SEO. Robot.txt serves to control and manage which pages or directories may be displayed / indexed search engines.
When your site has domain name namadomain.com registered in search engine (eg google), then you automatically allow or instruct the search engine crawler robot to visit and crawl namadomain.com site. Then, When the search engine's crawler robot visits your site, the first file they crawl is the robots.txt located on. So, the robots.txt file must be placed in the main directory or in the public_html folder. The robots.txt file contains commands on the bot crawler about which pages may be indexed or displayed in search engines. This is of course to protect your system pages to be known by people. For example the page Administrator website which is a page that is abstinence to be known by the public or indexed by search engines. That's where robots.txt does its job. Similar to websites, Blogs also have robots.txt files. However, robots.txt on blogs has usually been set as default by its blog service provider.
