How to Submit a Robots.txt File in the New Google Search Console | 6fc Live

Controlling how search engines like Google crawl and index your WordPress or WooCommerce site, such as “My Shop” (from your “Blog Page” guide), is crucial for SEO. A robots.txt file helps manage crawler access to pages like product listings (e.g., “Flying Ninja” from your “Creating Products” guide) or blog posts. This guide addresses how to create, upload, and submit an updated robots.txt file in the new Google Search Console (GSC), based on the 6fc Live Ask Google Webmasters video and enriched with current web sources. It also clarifies the transition from the old GSC, as raised by Div’s question about robots.txt submission.

Why Submit a Robots.txt File?

• Manage Crawling: Allows or blocks Googlebot from accessing specific pages (e.g., admin areas, staging sites) to optimize crawl budget.
• SEO Control: Prevents indexing of low-value pages (e.g., internal search results) while ensuring key content (e.g., product pages) is crawled.
• WooCommerce Fit: Enhances visibility for eCommerce pages or membership content (from your “Membership Plans” guide) for verified users (from your “Email Verification” guide).
• Limitations: Not all crawlers obey robots.txt; it doesn’t guarantee non-indexing (use noindex for that); changes may take 24-72 hours to reflect.

Context: Old vs. New Google Search Console

• Old GSC (Pre-2019): Had a dedicated Robots.txt Tester tool for editing, testing, and submitting robots.txt files directly in GSC.
• New GSC (Post-2019): The Robots.txt Tester was sunset in November 2023, replaced by the Robots.txt Report in Settings, which doesn’t allow direct editing or submission but supports recrawl requests.
• Div’s Question (Video): As of the 2019 video, the new GSC lacked a robots.txt submission feature, requiring users to switch to the old version. Google’s plan was to rethink features, not just copy them, to address modern website needs efficiently.
• Current Status (2025): The Robots.txt Report now allows viewing and requesting recrawls, but editing and submission are handled externally (e.g., via hosting or CMS).

Step 1: Create or Edit Your Robots.txt File

1. Understand Robots.txt:
   • A robots.txt file lives at your site’s root (e.g., https://yoursite.com/robots.txt) and uses simple rules to allow/disallow crawler access.
   • Example: txtCopyUser-agent: * Allow: / Disallow: /wp-admin/ Sitemap: https://yoursite.com/sitemap.xml
     • User-agent: * applies to all crawlers.
     • Allow: / permits crawling the entire site.
     • Disallow: /wp-admin/ blocks the admin area.
     • Sitemap: points to your sitemap.
2. Access Existing File:
   • Visit https://yoursite.com/robots.txt in a private browser to view the current file (per video, ensures public access).
   • Alternatively, download it via:
     • cPanel: Go to File Manager > public_html > robots.txt.
     • FTP: Use FileZilla to access the root directory.
     • WordPress: Use a plugin like Yoast SEO (from your “Blog Page” guide) or access via hosting.
3. Edit or Create File:
   • Open in a text editor (e.g., Notepad, VS Code) with UTF-8 encoding.
   • Add rules based on your needs:
     • Block staging: Disallow: /staging/
     • Allow specific pages: Allow: /product/flying-ninja/
     • Block AI bots (e.g., for privacy): User-agent: GPTBot Disallow: /
   • Avoid blocking critical pages (e.g., /shop/, /blog/).
   • Save as robots.txt (no extension).
4. Test Syntax:
   • Use third-party tools since GSC’s Robots.txt Tester is discontinued:
     • Logeix Robots.txt Tester: logeix.com/tools/robots-txt-tester
     • Merkle: technicalseo.com/tools/robots-txt/
     • Ryte: ryte.com
   • Or, perform a GSC Live Test in URL Inspection for https://yoursite.com/robots.txt to check crawler access.
   • Fix errors (e.g., incorrect syntax like Disalow instead of Disallow).

Step 2: Upload Robots.txt to Your Site

1. Access Root Directory:
   • cPanel: Go to File Manager > public_html, upload or replace robots.txt.
   • FTP: Connect via FileZilla, navigate to the root, and upload robots.txt.
   • WordPress:
     • Use Yoast SEO: Go to SEO > Tools > File Editor to edit/upload robots.txt.
     • Or, use a plugin like All in One SEO to manage robots.txt.
   • CMS (e.g., Wix, Blogger): Check for search settings to manage crawler access if direct editing isn’t available.
2. Verify Upload:
   • Visit https://yoursite.com/robots.txt in a private browser to confirm the updated file is live (per video).
   • Ensure no 404 errors (indicates missing file).
3. Subdomains:
   • Each subdomain (e.g., shop.yoursite.com) needs its own robots.txt at its root (e.g., shop.yoursite.com/robots.txt).

Step 3: Submit Robots.txt in Google Search Console

1. Access Robots.txt Report:
   • Log in to GSC: search.google.com/search-console.
   • Select your domain property (e.g., yoursite.com, verified per your “Verify Domain” guide).
   • Go to Settings > Robots.txt Report (available since November 2023).
2. Check Current File:
   • The report lists robots.txt files for the top 20 hosts (e.g., https://yoursite.com/robots.txt), last crawl date, size, and any errors/warnings (e.g., parsing issues).
   • Click the file to view its contents and highlighted issues (e.g., syntax errors).
3. Request Recrawl:
   • If you’ve updated robots.txt, request a recrawl to refresh Google’s cached version:
     • In the Robots.txt Report, locate your file (e.g., https://yoursite.com/robots.txt).
     • Click Request Recrawl (available for emergency updates).
   • Note: Google updates its cache every 24 hours automatically, so recrawl is optional unless urgent.
4. Verify Update:
   • Refresh the Robots.txt Report to check the “Checked on” timestamp (per video, confirms Google’s latest crawl).
   • Use URL Inspection for https://yoursite.com/robots.txt to confirm the live version matches your upload.

Step 4: Verify and Troubleshoot

1. Test Crawling:
   • In GSC, use URL Inspection to test specific URLs (e.g., https://yoursite.com/blog/):
     • Check if they’re allowed or blocked per your robots.txt rules (e.g., “Crawling allowed: Yes”).
     • Look for errors in the Page Indexing Report (e.g., “Blocked by robots.txt”).
   • Search site:yoursite.com on Google to ensure only intended pages are indexed.
2. Troubleshoot Issues:
   • 404 Error (Not Fetched):
     • Confirm robots.txt is in the root directory (/robots.txt).
     • Check file permissions (should be 644) in cPanel or FTP.
     • If missing, Google crawls all URLs (not an issue unless you need restrictions).
   • Parsing Errors:
     • Fix syntax (e.g., use Disallow: /wp-admin/ not Disalow).
     • Reupload and request recrawl.
   • Blocked Critical Pages:
     • Review Disallow rules to ensure key pages (e.g., /shop/) aren’t blocked.
     • Use Allow for exceptions (e.g., Allow: /shop/flying-ninja/).
   • Old Version Cached:
     • Wait 24-72 hours or request recrawl.
     • Manually inspect via URL Inspection.
   • CMS Issues:
     • If using WordPress, ensure plugins like Yoast SEO aren’t overriding robots.txt.
     • Check hosting settings for Wix/Blogger.
   • Contact support at support.google.com/webmasters/ or your hosting provider.
3. Monitor Regularly:
   • Check the Robots.txt Report weekly for errors/warnings.
   • Update robots.txt for site changes (e.g., new subdomains, content types).

Step 5: Enhance Your Robots.txt Strategy

1. GSC Features:
   • Pair with Sitemaps submission to guide crawling (from your “Submit URL for Indexing” guide).
   • Monitor Page Indexing Report for robots.txt-related issues (e.g., “Blocked by robots.txt”).
   • Use Performance to track indexed page metrics.
2. WooCommerce Integration:
   • Allow product pages (e.g., /product/flying-ninja/) for indexing.
   • Block low-value pages (e.g., /cart/, /checkout/) with Disallow.
   • Pair with NotificationX for sales alerts on indexed pages (from your “NotificationX” guide).
   • Secure transactions with Razorpay or UPI QR Code (from your “Razorpay” or “UPI Payment” guides).
   • Use FiboSearch for indexed product searches (from your “FiboSearch” guide).
3. Advanced Rules:
   • Target specific crawlers: User-agent: Googlebot for Google-specific rules.
   • Use wildcards: Disallow: /*.pdf to block all PDFs.
   • Add sitemap: Sitemap: https://yoursite.com/sitemap.xml to aid discovery.
4. Alternative Blocking:
   • For sensitive pages, use noindex meta tags or password protection instead of Disallow, as robots.txt doesn’t prevent indexing if pages are linked externally.
   • Example: <meta name=\”robots\” content=\”noindex\”> in your page’s HTML.

Step 6: Best Practices

1. File Placement:
   • Ensure robots.txt is at the root (/robots.txt), not a subdirectory (e.g., /subfolder/robots.txt is invalid).
   • Use UTF-8 encoding and lowercase filename.
2. Rule Simplicity:
   • Keep rules minimal to avoid errors (e.g., 5-10 lines for most sites).
   • Avoid over-blocking to maintain SEO visibility.
3. Testing:
   • Test rules with Logeix or Merkle before uploading.
   • Use GSC’s URL Inspection for live tests.
   • Check crawl errors in Page Indexing Report.
4. Security and Compliance:
   • Secure blocked pages with Wordfence (from your “Malware Removal” guide).
   • Use GDPR Cookie Consent for compliant tracking (from your “Cookie Notice” guide).

Pro Tips

• Boost Engagement: Promote indexed pages via Welcome Bar or Join.chat (from your “Sticky Menu” or “Join.chat” guides).
• Ads: Run YouTube ads for crawlable content with Google Ads (from your “YouTube Ads” guide).
• Analytics: Track indexed page performance with Visualizer (from your “Visualizer” guide).
• Backup: Save robots.txt with UpdraftPlus (from your “Backup and Migration” guide).
• Styling: Align crawlable pages with Neve’s design (from your “Neve” guide).
• SEO: Pair with Search & Filter for indexed content navigation (from your “Search & Filter” guide).

Congratulations!

You’ve learned how to create, upload, and submit a robots.txt file in the new Google Search Console with 6fc Live! Your WordPress or WooCommerce site is now optimized for efficient crawling and indexing. For more details, explore Google’s Robots.txt Guide (

) or the Robots.txt Report help page (support.google.com/webmasters/answer/13085936). Combine with your other guides (e.g., “Verify Domain,” “Submit URL for Indexing,” “Search & Filter”) for a robust SEO strategy. Need help with robots.txt syntax, recrawling, or troubleshooting? Comment below or visit support.google.com/webmasters/