SEO Agent
Introduction to Search Engine Optimisation > Why Are Pages Not Being Spidered?

Why Are Pages Not Being Spidered?


SEO Agent - Search engine optimisation for websites. Improve rankingsWithin SEO there are certainly some things to take notice of. If some of your pages are being missed out by Google on a regular basis and you cannot see them being spidered at all, then your pages may have one of the following traits:

Firstly if your page starts with an underscore e.g _page. The spider may believe this to be a hidden page and avoid it.

A broken link may be your course and the spider cannot simply follow through to the page.

The page load time may be too great. It might be full of content that depicts a very long load time and this may have a negative effect. Try to optimise images and content to make your pages more efficient when loading for all your visitors including spiders.

A page may also not be spidered if it contains duplicate content. You may have another page on your site that is exactly the same remember Google is looking for informative relative unique content.

Something else that you may have missed is that your file attributes may not appear to Google as a normal file, it may again think it is hidden. Check your file attributes and also your files extension to make sure there is nothing out of the ordinary.

If you have checked all this and still your page is not being indexed take a look at which sites are linking into that page. A very simple explanation may be that a site linking to the page in question may be part of a link farm or a bad neighbourhood that Google regards as SPAM sites. If this is the case the link may be very hard to be removed, ultimately it could affect your whole site. It would be better to kill off the page and rename it. At least anything trying to come from a questionable site to yours would get a '404' not found.