It doesn’t happen often, but when it does, it feels devastating. If you’re going to get any organic traffic from online searches, you need to make sure your site is visible—in other words, if you want to show up on Google’s search results pages, Google has to know that your site exists. And if your site isn’t being indexed by Google, it might as well not exist.
If your website isn’t appearing through organic search at all, fight the temptation to start panicking. Most of the time, this is simply an indication of some error or blockage that’s preventing Google from indexing your site—and these problems are easily fixed.
Take a look at these 10 reasons why Google might not be indexing your site—if you can’t be found in Google, chances are one of these is the culprit.
To the average web visitor, there’s no real difference between a URL that starts with http:// or http://www. Both of them ultimately lead you to the same place, so most users and webmasters don’t give it a second thought. But the www variant is actually a subdomain of the broader non-www version. In order to get your website indexed properly, you’ll need to verify your ownership of both in Google Webmaster Tools. You can also set your preferred domain, to inform Google which version you’d like to primarily use.
If you’ve just launched a site and you excitedly scoured through Google to see your site listed, relax. It usually takes Google at least a few days to index a new site. If several days have already passed and you still haven’t seen any results, it could mean that Google is having trouble indexing your site—and that usually means you’re having an issue with a sitemap. If you haven’t yet created or uploaded a properly formatted sitemap, that could be your problem. Once corrected, you can “force” Google to crawl your website through Webmaster Tools.
3. You’ve got a lingering robots file.
Robots.txt files are shockingly common to find. Occasionally, developers or content managers will use a robots.txt file to prevent a search engine from indexing a given page. Essentially, the file communicates with Google crawlers and tells them not to index a site—so if you remove the file, you’ll cease to have an indexing problem. Do a thorough scan of your website code, and remove any instances of robots.txt files that aren’t there for a specific reason. You’ll still need to give Google a few days to index your site after correcting the erroneous file.
It doesn’t happen often, but there is a chance that Google is having trouble crawling some of your web pages. If your home page is indexing, but not all of your internal pages are, it could be a symptom of a simple crawling error. Log into Google Webmaster Tools and click on “Crawl,” then “Crawl Errors.” This will lead you to a list of any pages on your site that are currently experiencing crawling errors. These errors are sometimes attributable to robots.txt files, detailed above, but can also be the result of DNS errors or server errors, both of which are easily correctable in most circumstances.
If you’re following best practices for content marketing, this shouldn’t be an issue, but there are circumstances where duplicate content can exist on your site—such as variations of a “master page” designed for slightly different audiences. If Google detects multiple instances of duplicate content, search engine crawlers can become confused and abandon indexing your site altogether. The easiest way to correct this is to get rid of the duplicate content. If deleting the duplicate content altogether isn’t an option, you can use 301 redirects or selective robots.txt files to ensure that Google only crawls one instance of each page.
If Google’s going to index your site, your site needs to be up. That means if you’re experiencing a loading problem when Google is attempting to index your site, you might miss the opportunity to be indexed. Ridiculously long loading times are sometimes the issue; if this is the case, you can decrease your loading times by setting up a decent caching system, reducing the size of your images, and installing a few applications to make the site run faster. It’s also possible that your hosting is unreliable, resulting in intermittent downtimes that are interrupting Google’s indexing attempts.
If you run a WordPress site, it’s possible you accidentally have privacy settings on—you can toggle this off by checking out “Privacy” under the Settings tab. It’s also possible that you’re using a .htaccess file for your website on the server. While .htaccess files are useful in most cases, they can sometimes interfere with site indexing.
Just like the robots.txt file, this is an addition that can mask your site’s pages from being found by search engine crawlers. Check your site’s code and look for the “noindex” tag somewhere in a meta title. If you find that somewhere, you’ve instantly diagnosed your indexing problem. Simply remove the tag and replace it if necessary, and you should be back on the fast track to search engine indexation.
When Google penalizes sites, it usually does so by dropping ranks and thus, visibility and traffic. However, there are rare and extreme cases when Google penalizes a site by completely removing it from indexes entirely. This is a type of manual penalty reserved for major infractions, so you don’t have to worry about this unless you’ve done something very wrong in the eyes of Google. If you’ve gotten deindexed this way, you’ve probably already been notified by Google, so unless that’s the case, you don’t have to worry that you’re not being indexed as a punishment.
Once your site is indexable, give Google a few days to catch up. You should start seeing your site in search engine results shortly. If you’re still having trouble, it’s possible your indexing problem could be more complex than usual. If you’re appearing, but you’re ranking very low, it could be an indication that your site is still new and doesn’t have much authority, or it could be an indication of a penalty. Either way, staying committed to best practices over an extended period of time is the best way to increase your visibility.