Menu

Create a robots.txt file for your WordPress

Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website
There are certain places where you should not allow Google crawlers to access them from crawling the whole website.

The robots.txt file should be added to the root of the site, where WordPress is installed. Here is an example of a robots.txt file, add this code to your robots.txt file.
User-agent:*
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Allow: /wp-content/uploads

Categories:   Security, WordPress

Comments