In an online discussion, a webmaster asked why his dynamic website was not indexed properly.
Google’s John Mueller explained that Google might not index URLs that repeat the same path element multiple times:
“The two issues I see here which make this site very hard to crawl and index are:
- The URL structure (as mentioned above). By including multiple elements repetitively like this in the path, it looks like we’ve run into an invalid set of URLs. This is a common failure case with sites that rewrite URLs and use broken relative internal linking, so our systems tend to skip URLs that repeat the same path element many times.
- The internal linking. It’s really hard for us to discover URLs within your site since you mostly use the map for navigation. This is a bit hard to diagnose though, given that we tend not to crawl the repetitive path-element URLs anyway. Potentially your side-menu will help once we can crawl it.”
There are many more technical things that can keep Google from indexing your web pages. The website audit tool in SEOprofiler helps you to find them: