if you want to modify or manage your robots.txt, you must first be careful and understand well because the error settings can make your website is not indexed search engine. Therefore, I will now explain how to set robots.txt correctly.
Not only the code I described above can be used in robots.txt. For example, you want to block certain posts from being indexed by all search engines.
To block a folder or directory (including its content) on your site, you can do so as in the example below. I use a directory on the web with self hosted wordpress to be more familiar.
With robots.txt we can also block urls that have certain characters / words from being indexed in search engines. For example I use a question mark character (?) Which is usually used on urls for query strings or searches on certain webs that cause duplicates.
We can also set robots.txt to block files on your site in certain formats. For example, we will block files with the format php (php), css (css) and javascript (js).
