robots.txt file

Quick wins for Magento SEO
robots.txt best practice guide