Crawling & Indexing Settings

In Site Manager, you can manage your Crawling and Indexing Settings. These settings control how and what Google and other search engines do with your page. 

Typically, you want (or even need!) search engines to crawl and index all the pages on your site. This is absolutely vital to ensure that your site shows up during searches, including a Google Custom Search if you have that installed on your site. 

There are specific pages that you may not want to be found by users - therefore found by search engines. Examples would include draft content, sensitive or confidential information, or even certain one-time event webpages.

You can change the settings for crawling and indexing in the page properties. Doing this inserts specific instructions in the page code that search engines know to follow.

Before you edit these settings, it's important that you know that these changes can significantly impact your website, and should not be changed if you do not understand the full impact to your website.

To begin with, we need a basic understanding of what the terms are.

Indexing: indexing refers to the collecting, parsing and storage of data to facilitate fast and accurate information. Gathering information (crawling) by itself is useful, but not useful if it cannot be displayed at the right time with the right results. Indexing is the art that accomplishing this.

Crawling: crawling refers to search engine bots systematically browsing all the content on the World Wide Web. Crawlers (such as Googlebot) follow known links and crawl for new links and update dead links. They go from link to link and bring data back to the search engine's servers.

How To Modify Crawling & Indexing Settings in Site Manager

1. We recommend that you start with a brand new page, but you can modify an existing page.*
2. In Navigation Builder, find the page you need to edit, or build a new page.
3. In the Page Properties, click the SEO tab.
4. Towards the bottom of the SEO screen, using the drop down menu, select which setting you want.
5. Click save.
6. The settings are per page - you must repeat this process for each page you wish to modify properties for.

Changing the settings creates the proper tag in the page code.