How to Create robots.txt file for SEO
Category: Articles | 1,419 views | Add a Comment |
It has been a half month since ThemeLib.com was created, not a long time, huh ?. Today, I decided to test how is my site in two popular Search Engines : Google and Yahoo.
Here is the result :
- Google: get 48 results returned with keywords http://csshook.com
- Yahoo: get 35 results returned with keywords http://csshook.com
All of the results in Google is fine. They indexed my posts, my tags, my keywords, … But there are some problems with my site in Yahoo. Yahoo indexed my wp-login page and download links !!! How did it happen, huh ? If you have knowledge on SEO (Search Engine Optimization), these results are not good, really. After examining ThemeLib.com a few minutes, I realized that I have not created a Robots.txt file yet !!! Oh man, how can I forget it?
Introduction
The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images, download links, admin files, …) therefore creating a robots.txt file can actually improve your website indexation. Read more…
- How to Get Google to Index Your webSite too Fast
- Tips For Clean Link Checking
- Top Key to Search Engine Optimization
- Video Site SEO Tips and Techniques
- Importance of Anchor Text in SEO
- How to Make My Website Come Up Top in GOOGLE
- SEO DoFollow Backlinks
- What is Google Sandbox
- Submit your site in Top Search Engines and Directories
- Social Media metrics


