Like many additions to a website, sitemaps provide both users and search engines with a service, but unlike most content, it’s arguably more important for the search engine. For the user it’s a convenient way to provide a bird’s eye view of all your public web pages. In terms of SEO though, it provides an invaluable way to provide search engines with links to all your pages from a top level page.
Less clicks, better rankings
As explained in [information architecture], the more links a search engine needs to go through to find pages, the less likely it is to index them (and the less value it’s likely to attribute to them). Indeed site maps should be used as part of your information architecture and navigation strategy, if necessary using site maps at each levels of your navigation hierarchy, in the case of large sites. This helps to keep the structure flat and reduce the number of clicks to get to all of your pages.
In addition to having a sitemap for users, it’s worthwhile having an XML Sitemap. These use a standardised structure to encapsulate information about and links to your web pages in a format that spiders can easily understand. In addition to basic page information (which could be picked up from an html sitemap, an XML Sitemap holds information about last modified dates, change frequency and priority, which helps search engines to know when & how often to crawl certain pages in your site and how much relative importance to attribute to them.
XML Sitemap generators
It would be a bit of a slow and laborious process to create an XML Sitemap by hand, but fortunately there are tools that help you do this by spidering your website and automatically generating an XML Sitemap. One of these is XML Sitemap Generator. Bear in mind it’s hard for an automated tool to know the priority you wish to attribute to different pages on your site, so you may wish to edit manually before submitting.
If you have a CMS, you may have a built in tool that generates XML Sitemaps – this is very convenient, as the CMS knows your page structure, links and how frequently pages are updated, so it does a lot of the legwork for you.
How to submit an XML Sitemap to search engines
You can tell search engines where your XML Sitemap is by adding the following to your robots.txt file:
In addition, many search engines allow you to explicitly submit an XML Sitemap. For example, Google allows you to submit an XML Sitemap through it’s Webmaster Tools interface, or you can hit the URL www.google.com/webmasters/tools/ping?sitemap=http://mywebsite.com/sitemap_file.xml, which allows for automated scheduled submission.
Easier still, some CMS’s will automatically submit your XML Sitemap to various search engines as part of auto-generation mechanisms.