What is robots.txt?
A robots.txt file is a simple text file that tells search engine crawlers which parts of your site they can or cannot access. It uses a set of commands to control crawling behavior by section and by specific types of crawlers (e.g., mobile vs. desktop).
In practice, robots.txt helps you manage how search engines interact with your marketplace. These instructions are defined using Allow or Disallow rules for certain (or all) crawlers.
To learn more, refer to Google’s guide on robots.txt.
Do I need to configure my robots.txt file?
By default, your Arcadier marketplace pages and item listings are fully accessible to search engine crawlers. This is ideal for most marketplaces because it improves SEO by allowing search engines to index as many pages as possible.
However, you may want to configure robots.txt in these scenarios:
- Private marketplace: Block all content from search engines.
- Development stage: Prevent indexing until your marketplace goes live.
- Paid links or ads: Apply special instructions for compliance with Google guidelines.
How do I configure my robots.txt file?
- Go to Settings > Analytics/SEO > Robots.
- Enter your desired rules in the text area.
- Click Save.
Your changes will update the robots.txt file for your marketplace domain.
For detailed instructions and best practices, check Google’s robots.txt guide here.