Home » Sitemap.xml and Robots.txt Files for SEO: Guide for Blogger and WordPress

Sitemap.xml and Robots.txt Files for SEO: Guide for Blogger and WordPress

  • by
robots.txt files for seo

robots.txt files for seo

After adding your website in Google Search Console, it’s time to create sitemap.xml and robots.txt files for SEO so that posts and website pages will be indexed by Google quickly.

Create an XML Sitemap

To make it easier for Google to understand the content of your website, you need to create an XML sitemap. An XML sitemap is a list of web pages that are addressed to search engines to make the website easy to find.

To create a sitemap for blogger, you can register your website sitemap in Google Search Console. Here’s how:

  • Go to the Google Search Console homepage.
  • Select the “Sitemaps” menu.
  • Add the “sitemap.xml” file and click the “Submit” button.
  • If successfully added, a green “Success” status will appear on the sitemaps list.
sitemap4

sitemap4

For WordPress you can use the Yoast plugin, select the “General” menu then click “Features” and make sure the position is ON for the red XML sitemaps.

sitemap5

sitemap5

 

Setting Robots.txt for SEO

One way to make your website SEO Friendly is by setting robots.txt file for SEO so that it will be easier for your website to be indexed by the Google search engine.

Robots.txt is a simple file with text format (.txt) whose function is to control the crawlbot or a robot that crawls the website.

When this file is installed on the website, it allows us to set which pages are allowed to be indexed by the Google search engine.

So when you register your website with Google Search Console, you will automatically ask the Google crawl robot to index your website. By activating robots.txt, it will make it easier for Google, Bing, Yandex, Baidu and their friends to crawl our website.

But, that’s not all…

Enabling robots.txt can also be applied if we don’t want the blog page to be indexed or not crawled by search engine bots.

However, the robots.txt file for SEO setting should not be careless because if there is an error when loading this robots.txt, it can be fatal to your website. Now, let’s do the process of setting the robots.txt file to Blogger in two ways:

  1. Enable Blogger’s default Robots.txt

Google recommends setting this robot.txt on the default mode so that it is safe for your website. Here are the settings for Blogger’s default robots.txt:

User-agent: Mediapartners-Google

Disallow:

 

User-agent: *

Disallow: /search

Allow: /

 

Sitemap: https://www.wordpresshostplans.com/sitemap.xml

Description:

  • User-agent: Mediapartners-Google: this code is in charge of helping the Google Adsense Bot to crawl ads on the website.
  • User-agent: * : These are all search results engine robots.
  • Disallow: /search : Block access and forbids pages such as labels from appearing in search engines.
  • Allow: / : This command allows bots to crawl your site and index this page.
  • Sitemap: https://www.wordpresshostplans.com/sitemap.xml : Sitemap or website sitemap.

 

       2. Create a custom robots.txt Blogger

The word “custom” above means that you can create this robot file as needed, for example setting up the Google bot to crawl certain pages so that they are indexed.

For example, you don’t want the Google bot to block the search / label page, because blocking that page can cause the coverage in the Google Search Console to have a warning notification.

This is a custom robots.txt file:

User-agent: *

Disallow: /p/contact-us.html

 

Sitemap: https://www.wordpresshostplans.com/sitemap.xml

The settings above mean that you want to stop search engines and block Google bots from crawl certain pages, for example for the Contact Us page only.

 

Setting the Robots.txt File in WordPress Site

The robots.txt file is actually already in the root directory folder on the file storage server called public_html. It’s just that robots.txt is a virtual file that cannot be accessed or replaced through any directory. So, when you open the public_html page it doesn’t find the robots.txt file there.

In order to be able to modify the robots.txt rule must add a new file there. Create your own robots.txt file which is placed in the public_html folder and enter the configuration script manually.

The new file that parts of your site is created serves as a replacement file which will later overwrite the existing configuration file. Configuring robots.txt is quite simple because there are only a few commands required. The following is an example of a robots.txt configuration file that you can apply:

User-agent: *

Disallow: /cgi-bin

Disallow: /wp-admin

Disallow: /wp-includes

Disallow: /wp-content

User-agent: Bingbot

Disallow: /

 

Or you can add some other configuration to allow some search engines to crawl. You just need to add a few lines of script as below:

User-agent: Mediapartners-Google

Allow: /

User-agent: Adsbot-Google

Allow: /

User-agent: Googlebot-Mobile

Allow: /

Sitemap: https://www.wordpresshostplans.com/sitemap.xml

Description:

  • User-agent: this section is used to describe the rule type/robot name.
  • Allow: to add which parts can be accessed by robots.
  • Disallow: to describe parts that are not allowed to be accessed by robots.

 

For WordPress you can also use the Yoast plugin, select the SEO menu – Tools – File Editor.

sitemap6

sitemap6

Then select “Create a New Robots.txt”. On this page, enter the rule you want to apply to the website.

In addition to the methods above, you can also manually upload the ftp client robots.txt file using FileZilla or access cPanel hosting.

 

Submit to Google Search Console

You can make sure that the robots.txt files for SEO that has been created is correct and ensure that no errors appear. Checking can be done using Google Search Console, the results are like the picture on the start page above.

Leave a Reply

Your email address will not be published.