With the launch of HubSpot 3 and the new landing page tool, there’ve been a lot of features introduced that are geared to advanced users. If you know about robots.txt, and are familiar with HubSpot, you probably knew that in the old system you couldn’t edit the robots.txt on the HubSpot CMS. With the launch of the beta landing page tool, all that has changed.
What is Robots.txt?
If you’re unfamiliar with robots.txt, it’s a file that resides on a website’s server that gives instructions to search engine crawlers that index websites. The instructions in the robots.txt file dictate what content on your site can be crawled, indexed and ultimately displayed as search engine results. For example, if you want to have all your images hidden from search engines, you can include a rule in your robots.txt file that blocks access to your images folders (though, that’s not recommended).
Uses for the robots.txt with HubSpot
- You can tell search engines to not index your landing pages that contain links to downloads
- You can tell search engines to not index your PDFs that exist behind conversion forms, that way they can only be accessed via the landing page
Keep in mind, this new tool is only available in beta and for HubSpot customers who have the new beta landing page tool. For more info on how to use the feature, go here.
