if you want to modify or manage your robots.txt, you must first be careful and understand well because the error settings can make your website is not indexed search engine. Therefore, I will now explain how to set robots.txt correctly.
Blocking Url
Not only the code I described above can be used in robots.txt. For example, you want to block certain posts from being indexed by all search engines.
Blocking Folders / Directories
To block a folder or directory (including its content) on your site, you can do so as in the example below. I use a directory on the web with self hosted wordpress to be more familiar.
Blocking URLs Containing Characters / Certain Words
With robots.txt we can also block urls that have certain characters / words from being indexed in search engines. For example I use a question mark character (?) Which is usually used on urls for query strings or searches on certain webs that cause duplicates.
Blocking Access On Specific File Extensions / Formats
We can also set robots.txt to block files on your site in certain formats. For example, we will block files with the format php (php), css (css) and javascript (js).