Robots txt: Configuring it is essential for SEO in Magento 2 store

Robot-txt-Magento-2-banner

If you are reading the post, it means either you are a Magento 2 store owner or is responsible for the SEO of the store. You must be aware of the importance of robots txt file. It is a text file that instructs the web robots, i.e., search engines which pages to crawl and which not!

In the words of Google,

“A robots.txt file tells search engine crawlers which pages or files the crawler can or can’t request from your site.”

This tiny file is a part of EVERY website, and Magento 2 stores are no exception. Fortunately, the default Magento 2 supports to configure a robots txt file. And it will be shown in this article.

Why is Robots txt essential for SEO in Magento 2 Store?

  • Controls how the search engine spiders see and interact with the pages of Magento 2 store.
  • Its improper use can affect the rankings of your store negatively.
  • Robots.txt file is the foundation of how the search engine works.
  • Restrict duplicate content pages from appearing in SERPs.
  • It helps to avoid overloading your store with requests from Google’s crawler.
You may like:

Magento 2 hosting: Effective tips to choose the best one

SMTP (Simple Mail Transfer Protocol): The benefits and how to configure in Magento 2

Configure Robots txt File In Magento 2

1. First of all, login to your Magento 2 Admin Panel.

2. Click Content and under Design, choose Configuration.

robots txt

3. From the left side, Edit the Global Design Configuration.

robots txt

4. Expand the Search Engine Robots section.

Magento Search Engine Robots

5. Now you need to do the following:

a) Set Default Robots option to one of the following:

INDEX, FOLLOW: Instructs search engine crawlers to index the store and check back later for changes.

NOINDEX, FOLLOW: Instructs search engine crawlers to avoid indexing the store but check back later for changes.

INDEX, NOFOLLOW: Instructs search engine crawlers to index the store once but do not check back for changes.

NOINDEX, NOFOLLOW: Instructs search engine crawlers to avoid indexing the store and do not check back for changes.

b) In Edit Custom Instruction Of Robots.txt File option, you can enter the custom instructions if needed. For example, while developing your Magento 2 ecommerce store, you can disallow access to all folders by entering custom instructions.

c) Reset To Default button will remove your custom instructions and reset the Magento 2 robots.txt file to system’s default.

6. Once you’re done with the configuration, hit Save Configuration button to apply the changes.

Custom Instructions For Magento 2 Robots.txt

Like any other web application or e commerce platform, you can add custom instructions to the robots.txt file in Magento 2. Here are some examples of “Disallow” to consider for Magento 2 Robots txt file.

Allow Full Access

  1. User-agent:*
  2. Disallow:

Disallow Access to All Folders

  1. User-agent:*
  2. Disallow: /

Default Instructions

  1. Disallow: /lib/
  2. Disallow: /*.php$
  3. Disallow: /pkginfo/
  4. Disallow: /report/
  5. Disallow: /var/
  6. Disallow: /catalog/
  7. Disallow: /customer/
  8. Disallow: /sendfriend/
  9. Disallow: /review/
  10. Disallow: /*SID=

Restrict User Account & Checkout Pages

  1. Disallow: /checkout/
  2. Disallow: /onestepcheckout/
  3. Disallow: /customer/
  4. Disallow: /customer/account/
  5. Disallow: /customer/account/login/

Disallow Catalog Search Pages

  1. Disallow: /catalogsearch/
  2. Disallow: /catalog/product_compare/
  3. Disallow: /catalog/category/view/
  4. Disallow: /catalog/product/view/

Disallow URL Filter Searches

  1. Disallow: /*?dir*
  2. Disallow: /*?dir=desc
  3. Disallow: /*?dir=asc
  4. Disallow: /*?limit=all
  5. Disallow: /*?mode*

Restrict CMS Directories

  1. Disallow: /app/
  2. Disallow: /bin/
  3. Disallow: /dev/
  4. Disallow: /lib/
  5. Disallow: /phpserver/
  6. Disallow: /pub/

Disallow Duplicate Content

  1. Disallow: /tag/
  2. Disallow: /review/

Conclusion

How-to-Create-Magento-2-Robots.txt_Meetanshi-880x346

The robots txt file is essential for your Magento 2 ecommerce store when communicating with web crawlers. This standard defines how to inform the search engine bots about the pages of your Magento 2 store that should be excluded or opened for crawling. That’s why the robots txt file is significant for the correct Magento 2 store indexation and its overall search visibility.

As Magento 2 provides a mechanism for creating a robots txt file, there is no need to create one manually. All you need to do is add some configuration in Magento 2 itself, and a robots txt will be generated periodically.

I hope that you found this quick guide on how to configure Robots txt file in Magento 2 useful. If you have any question, feel free to share them in the comments below.

Level up your website with ArrowHiTech – official partner of Magento

Our own Magento development support services is plus the option for monthly support packages. It is to make sure your online business always remains in top form. And of course, there are countless other developer websites that can be easily found and perused as well.

Capture

ArrowHiTech Development Agency

Our Magento services:

Magento Consulting And E commerce Strategy

Magento Website Development

Magento Progressive Web Application Development

135 Total Views 4 Views Today