How to edit robots.txt in HubSpot
HubSpot generates a robots.txt file for every connected domain and lets you change it from inside your account. Here is where to find it, what to put in it, and the mistakes that quietly hide a site from Google.
Paul Lovell
SEO Consultant
Where to find robots.txt in HubSpot
In your HubSpot account, open Settings, go to the website pages settings, and choose the domain you want to edit. Look for the SEO & Crawlers tab. The robots.txt editor is on that tab. Menu names move around between HubSpot releases, so if you do not see the tab, search the settings for “robots”.
You can view the live file at any time by visiting yourdomain.com/robots.txt. Always check there after saving, because that is the version Google reads.
What the default file contains
The default file allows crawlers to access your content and points them at your sitemap. It also blocks a handful of HubSpot system paths that have no value in search. For most sites the default is correct, so edit it only when you have a specific reason.
The basic syntax
Robots.txt is a plain text file made of groups. Each group starts with the crawler it applies to, then lists the paths it may or may not visit.
User-agent: *
Disallow: /thank-you/
Sitemap: https://www.example.com/sitemap.xmlUser-agentnames the crawler.*means all of them.Disallowstops matching paths being crawled.Allowmakes an exception inside a disallowed path.Sitemaptells crawlers where your sitemap lives. Use the full URL.
Safe examples
Stop crawlers wasting time on thank-you pages
User-agent: *
Disallow: /thank-you/Block internal search results
User-agent: *
Disallow: /hs-search-resultsAllow a specific AI crawler
Name the crawler in its own group. Check each vendor's documentation for the current user-agent name, as they change.
User-agent: GPTBot
Allow: /Mistakes that hide a site from Google
- Leaving
Disallow: /in place. This blocks the entire site. It is often copied over from a staging setup. If your pages are not being crawled, check for this line first. - Using robots.txt to remove a page from search. Blocking a URL stops Google reading it, which also stops Google seeing a noindex tag on it. Use noindex instead and let the page stay crawlable.
- Blocking CSS or JavaScript files. Google needs them to render the page properly.
- Forgetting other domains. If you use a separate domain for a blog or knowledge base, it has its own robots.txt.
How to test your changes
- Save the file in HubSpot, then open
/robots.txton the live domain and confirm the change is there. - In Google Search Console, open the robots.txt report to see whether Google fetched the file and found errors.
- Use the URL Inspection tool on a page you care about and confirm it says crawling is allowed.
If a page still is not showing up after that, see why HubSpot pages are not indexed.
Frequently asked questions
Can I edit robots.txt in HubSpot?
Yes. HubSpot lets you edit the robots.txt file for each connected domain from the SEO and crawler settings in your account. You change the file as text and save it; there is no file to upload.
Does blocking a page in robots.txt remove it from Google?
No. Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it. To keep a page out of Google, use a noindex tag and leave the page crawlable so Google can see the tag.
Do I need a separate robots.txt for each domain?
Each domain connected to HubSpot serves its own robots.txt, so a change on one domain does not carry over to another. Check each domain you use.
Should I add my sitemap to robots.txt?
It is good practice to include a Sitemap line pointing at your sitemap.xml. HubSpot normally includes one by default, so check before adding a duplicate.
Want AI tools to find your best content?
robots.txt controls access. llms.txt tells AI tools what to read. Keep yours up to date automatically.
See llms.txt for HubSpot →