Internet Marketing - Matt Bailey [170]
Additional robots.txt Resources
Ideally, this has helped a few understand the place and purpose of the robots.txt file. Even more than that, I hope that it has taken the fear away from dealing with this file. Many site managers are very gun-shy, because they may have had a disallowed site from the search engines with a misplaced slash at one time or another.
If you want more information about the robots.txt file and all the things you can do with it, I suggest the following resources:
Robotstxt.org: www.robotstxt.org
Official Google Blog: The Robots Exclusion, pt 1: http://googleblog.blogspot.com/2007/01/controlling-how-search-engines-access.html
Google Blog: The Robots Exclusion, pt 2: http://googleblog.blogspot.com/2007/02/robots-exclusion-protocol.html
Robot.txt Code Generation Tool: www.mcanerin.com/en/search-engine/robots-txt.asp
The bottom line when dealing with robots.txt? Don’t be afraid to ask for help. Many marketers, business owners, and large companies have unwittingly blocked the search engines from accessing their websites. It’s a simple file that can cause a lot of problems, so don’t be afraid to have a third party check your work.
Review and Hands-On
Learn about your server and the language used to develop your website.
What type of server is your website hosted on? (Windows Server, Apache, other, and so on.)
What type of database does your website use?
What type of scripting language does your website employ?
Do you have a rewrite function on your URLs?
What function was used to develop the rewrite?
Provide three positive reasons that persuaded you or your company to choose that particular host server, database, and language selection for your site.
Find a software program or a website that will provide a header check. Check your page codes to ensure that mostly you have a 200 OK code for a page that was requested and delivered. If you have redirects in place, be sure to check them, and make sure that they are 301 redirects and they are going to the correct destination.
It’s time to bring out all of the tools that have been discussed about content duplication: webmaster tools, analytics, and search. Look in your analytics, and see whether there are any page duplications for the same content or page titles. Search for a unique sentence from a few of your pages and see whether there are any duplicates. Beyond searching on your website, search online to see whether you have syndicated content that is being duplicated online.
Check the status of your site in Google’s Webmaster Tools. Go to the Crawler Access section and the report, and test to see whether your website is accessible with your newly created (or revised) robots.txt.
Chapter 15
Week 12: Remember the Important Details
If you have been religiously reading this book an hour a day, then congratulations—you should now be finishing up the third month of your web marketing journey. You’re past the halfway point in learning about all about the development, design, content, architecture, and persuasiveness and are on your way to fully marketing your business through the use of analytics, PPC, social media, link building, blogging, and networking. At this point in the development and marketing cycle people tend to get excited about what’s to come, and they forget the important little details that can mean so much in a website. The river of marketing is very shallow yet very wide. They basic concepts are easy to grasp, but it is the thousands of little details.
Chapter Contents
Monday: Compare Tables and CSS
Tuesday: Create Better Error