The robots.txt log is a direct message (no html) that is placed on your site to request the web index listing which pages to run and which not to. The scrolling guidelines are provided so that the web bot understands which pages to scroll, etc. There are some basic instructions given to web robots in the format of – User-agent: * Disallow: / User-agent: * Allow: / For example-For www.abc.com site: Robots.txt file URL: www.abc.com/robots.txt Blocking all web crawlers from all content User-agent: * Disallow: / Using this code in a robots.txt file would indicate all web crawlers to not to crawl any pages on www.abc.com, including the homepage. Allowing all web crawlers access to all content User-agent: * Disallow: Using this code in a robots.txt file tells web crawlers to crawl all pages on www.abc.com, including the homepage. If You are Looking for Affordable SEO Company India , to expand your business activities with diverse and enhanced marketing strategies to get higher ROI and to