Use Google XML Sitemaps for Better Site Indexing

If you have a small, simple site that Google seems to index entirely, ignore what I say below. But if your site is getting pretty large and Google seems to be slighting you, here's an answer. Try Google Sitemaps, a system of submitting an XML-based sitemap directly to Google that will help their spider robot more easily index larger sites. This is especially important for sites with dynamically-generated webpages or product pages. For more information see Google Sitemaps FAQ (https://www.google.com/webmasters/sitemaps/docs/en/faq.html).

Essentially, you (1) use a software tool to create the sitemap in Google's designated XML format, (2) upload that to your website, and then (3) notify Google of its URL. While you're at it, why don't you make an HTML sitemap for Yahoo, MSN, and others, and put a small link to it at the bottom of your webpage. Some tools make that easy, too.

For smaller sites up to about 750 webpages, all you need is the free Google Sitemap Generator (Tarrant IT.com), written by Greg Tarrant of Australia. This online service generates a map with up to 750 entries. It includes a "filters and settings" section that allows you to restrict directories searched. There are options to include PDF, .doc, .xls, and .zip documents. The program can be instructed to obey robots' metatags about which directories and files to index and which to avoid. The program makes both an HTML and XML sitemap. I found that it works quite well, with its XML formatting accepted by Google. Highly recommended, even for non-programmers. If you like it, why don't you include a PayPal donation like I did? http://www.tarrantit.com/GoogleSiteMap/

For larger sites, you'll probably want to install a program on your site to index it and create the sitemap. This is a task for a trained webmaster, not a novice. However, there are a number of tools to do this. I'm listing some of the free ones I've come across. I have not tested them all.

TM Google Sitemap Generator 1.00 (TM Research). Written in Perl. This is a fairly simple program, with few directions. Once you install the program and set permissions correctly, you are asked to designate the types of files to be indexed, then given a list of directories which can be unchecked to prevent their being indexed. Very fast, even for a 1500 webpage directory. http://www.tm-research.com/products/google-sitemap/

Google Sitemaps (BETA). Written in Python 2.2, which must be installed on your server. But this is the one recommended by Google itself. http://www.google.com/webmasters/sitemaps/docs/en/sitemap-generator.html Also available from SourceForge. http://sourceforge.net/projects/goog-sitemapgen

phpSitemapNG 1.3.2b (enarion.it). A fairly sophisticated PHP script that allows you to exclude filetypes and directories not to be indexed. Nice. http://enarion.net/google/ 

Google Sitemap Generator 1.0 (AuditMyPC.com). Written in Java. http://www.auditmypc.com/free-sitemap-generator.asp

Google Sitemap Generator (DMXzone.com). Written by George Petrov of Dynamic Zones. http://www.dmxzone.com/ShowDetail.asp?NewsId=10538

VIGOS Gsitemap (VIGOS AG) for Windows. http://www.vigos.com/products/gsitemap/

GSiteCrawler (SOFTplus). Written by Johannes Miller for Windows 2000, XP. http://johannesmueller.com/gs/