Common SEO Oversights: Site Maps

Every website should have a regular audit carried out in order to ensure that it’s performing at the top on its game when it comes to the SERPs. However, it’s also not the priority of many a site owner and so often, important SEO jobs get missed.

In these post Panda/Penguin times, not knowing the rules could have serious consequences for a website and for firms that rely on the web for any part of their business, then ignorance is really no excuse.

It’s a tough world out there on the World Wide Web, and if you’re to compete, then it’s wise to know what the competition are up to and how you can gain an edge over them.

Google algorithms and penalties

For the SEO professional, keeping up to date with Google’s ever-changing algorithms is a job in itself. If it wasn’t, then so many sites wouldn’t be trying to claw their way back up the rankings after a penalty was imposed now.

What I’m getting at here, is don’t think that just because everyone and their dog claims to be an expert in SEO when they land unsolicited in your inbox, that this means it’s simple to carry out. It’s not.

Let’s have a look now at some of the most common oversights people make when it comes to SEO, starting in this article with sitemaps.

Sitemaps

There are two kinds of sitemap, one for the search engine and one for the visitor. The search engine one should be an .xml file that lists all pages that you want indexing by the search engine.

Sitemaps help search engine spiders to locate all of the content on your site that it might miss whilst it’s carrying out a crawl. That’s not to say that your site won’t get indexed without a sitemap, but if you want all of your content to be indexed, then you really should have one.

If you do have a sitemap, then consider the following:

• Is the sitemap ever updated? If so, when?
• Do you have a separate one for images and videos?
• Does the sitemap get uploaded to the server?
• Does the sitemap you have list all indexable pages?
• How big is the sitemap, does it contain more than 50,000 URLs or is it bigger than 50MB when uncompressed (unzipped)

If it’s bigger than the above, then you should break down the site map into more than one and it’s also recommended that you resubmit whenever the content on your site changes.

Robots.txt – a note

People often confuse xml sitemaps and robots.txt files, so it’s worth knowing that the latter is different in one key manner. Robots.txt files tell the search engine which URLs you don’t want followed and which you do.

They don’t block these URLs, they simply don’t crawl and index them if certain parts of the site/directories are disallowed. However, it should also be noted that this is advisory and the search engine may still crawl the content, although most don’t.

For an example of what a robot.txt file looks like, check out Google’s here. As you can see, the file uses the terms Disallow and Allow to tell the spiders what to crawl and what not to.

User-agent: *

Disallow: /search

Disallow: /sdch

Disallow: /groups

Disallow: /images

Disallow: /catalogs

Allow: /catalogs/about

Allow: /catalogs/p?

Disallow: /catalogues

Disallow: /news

Allow: /news/directory

According to Google:

How to create an xml sitemap

You can create your own sitemap, but why go to the bother when there are lots of resources online that can generate one for you? If you do want to create your own then you will need some coding experience and you can find a full list of the protocols and tags that you will need to use on the sitemap.org website.

If you have a lot of URLs and content, then you may have to create more than one sitemap as well as a sitemap index file, which is the one you will then submit to the search engine. It’s also advisable to create one for media such as video and images too.

Once you have done all this, it’s just a case of submitting the sitemap or index file to the search engines.

That isn’t the end of it though, every time you post fresh content on the site, you should resubmit the sitemap, so that the search engines know to crawl it again. If you regularly put fresh content on the site (as you should), then the search engines will work this out and will be inclined to visit and crawl your site more often.

However, if you only update intermittently, then it’s likely that they will notice this too and come back less often, which will affect your position in the SERPs.

Using newsfeeds as sitemaps

Many of the major search engines also accept RSS (Real Simple Syndication), RSS 2.0, mRSS and Atom 1.0 feeds as a sitemap. Most blogs have one of these feeds and if you use WordPress, then it will have one by default without you having to do anything.

However, the feed might only provide the latest URLs that have been updated, so it’s not the perfect solution. For video content, you can use mRSS (media RSS) or if you just have a basic site that just contains text, with no imagery or video, then you can just provide a list of URLS in a text file, which should look like the below example.

http://www.example.com/file1.html

http://www.example.com/file2.html

http://www.example.com/file3.html

Guidelines:

• Ensure that you use the full URL
• Must use UTF-8 encoding
• List must contain URLs only
• The file can be called anything but must have a .txt file extension

Whilst UTF-8 encoding may sound like it’s something complex that you should be wary of, it isn’t and you can do this just by creating your file in a text editor such as Sublime Text 2 and hitting ‘save with encoding’ and choosing UTF-8.

To finish this up, I will say that it’s useful to have a sitemap for visitors on your site too, so that usability is taken into account, and it’s a simple matter for your visitors to find what they are looking for.

This can be broken down by:

• Home
• About
• Contact
• Products
• Privacy policy
• Terms and Conditions
• Blog
• Blog archives

This is of course a general outline and will depend entirely on how many pages you have on the site and what they are about. Try to make it as simple and logical as you can and it will prove an invaluable resource for your visitors.

Sitemaps are important if you want your regular content to be crawled and indexed by the search engines, and the more you update, the more you will be crawled and the better you will be indexed. This means that it’s not an aspect to SEO that should be ignored if you want to place as well as possible in the SERPs.