The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is unrelated to, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.
add sitemap tag to robots.txt
keywordenvy Tutorial #1: robots.txt
Should I block duplicate pages using robots.txt?
Web Design Blog - Robots txt files explained
Online Marketing Quick Tip #1 - Search Engine Optimization - Robots.txt files
Uncrawled urls in search results
Will a link to a page disallowed in robots txt transfer pagerank
Use Google Webmaster Tools to Create a robots.txt File