Check whether search engines can access your important pages

Start with the complete address of the live page. Record whether it should be public, whether it is intended to appear in search and which version is preferred if there are similar pages. A private portal, a preview page and a live service page can reasonably have different settings.

For example, a new office refurbishment page might replace an old fit-out page. Before removing an instruction or redirect, agree which address should be kept and whether the old information has a valid replacement. The investigation is easier when the intended outcome is explicit.

Crawling is the retrieval of a page. Indexing is the process of analysing and storing information for search. Serving results is the selection of relevant information for a searcher's request. Success at one stage does not guarantee the next. Google does not guarantee that it will crawl, index or serve a page.

That distinction changes how you read an audit. “We could retrieve this page” is evidence about the audit's request. It is not evidence that Google has indexed it. “Indexed” also does not mean that the page must appear prominently for a particular search. Keep the question precise when asking for help.

Ask your developer to check the response at the live address, including the final destination after any redirects. A browser can show an attractive error page, so its appearance alone does not establish that the response is correct. Also check whether access changes for a visitor who is not logged in.

If an audit cannot retrieve the page, investigate the reason before treating the content as missing. A firewall, anti-bot rule, timeout or hosting problem may affect the tool. A test from one service cannot establish how every crawler is treated. Give the developer the URL, the time and the error or response you received.

Robots.txt manages crawler access; it is not a reliable way to keep a page out of Google results. A noindex instruction asks supporting search engines not to index the content. Google needs to be able to retrieve the page to see that instruction, which may appear in the HTML or an HTTP response header.

Do not remove every restriction because a checker calls it a warning. Establish whether the rule is intentional and which URL or crawler it affects. Keep private information behind appropriate access controls. The fact that a page should not be in search does not make robots.txt a security measure.

Canonical signals help indicate a preferred version when pages are duplicate or very similar. A canonical annotation is a signal rather than a command, and Google may select a different version. Review the actual relationship between the pages before assuming that a different canonical URL is an error.

Also confirm that customers and crawlers can reach the intended page through relevant internal links. Check its presence in the sitemap where appropriate and verify that those links use the live address. An important service page should not rely solely on someone guessing its URL. These checks help explain discovery; they still do not guarantee indexing.

With access to the relevant Search Console property, use URL Inspection to review Google's indexed information for the complete URL. A live test examines the current page and is different from the stored indexed result. The live test cannot predict Google's eventual canonical choice or guarantee indexing.

Save the finding before making the correction. After publication, check the live response and relevant instructions, then review Google's evidence again when it has had an opportunity to process the page. Record the date. If the intended settings are correct but the page still does not appear, widen the investigation to the content, duplication and the search question itself.