More often than not you’re looking to have your website visibility increased when contracting an online branding agecny, but as a different train of thought, did you realize you can use them to hide your website as well from the search engines?
To clarify a tad, you don’t typically engage a marketing agency to hide your business, but there are cases where you would want to keep the search engines at bay. Depending on the way your site is constructed for example, you’ll want to keep them out of your image files, or perhaps you have an extensive pdf collection hosted on your site, and you’d rather keep them from being indexed as part of your website. Maybe it’s even a new website build being done to update your look and content, and that needs to be completely hidden from the search engines until you’re ready to go live, just in case the bots get confused – they’re not intelligent at all in this respect. For those reasons, I submit a short list, on how not to be found on the search engines.
One of the simplest and quickest ways to be hidden from the search engines is by editing your web hosts robots.txt file. Not all website owners have access to change this file, but when you can it can be used to restrict search engine behaviour like nothing else. With just a simple phrase, ‘disallow’, you can completely obscure your site or pages from being found by search engine spiders. But just as uncle Ben said ‘With great power comes great responsibility’ you need to be extremely careful with what you choose to disallow to the search engines, you may inadvertently block key components or pages of your site, rendering any search engine marketing efforts useless.
If the robots.txt change is too broad a stroke for your case, but you still have certain areas you would like to restrict the search engines from finding certain pages, you can use a meta string of code in the head of each webpage you’d like to keep from being indexed. It’s even as easy as it sounds, all you need to do is add the switch meta=noindex to hide a certain page, and there are a handful of additional switches which could be added to sculpt search engines flow through your site.
The two last methods for restricting bot activity on your website are the biggest guns in the game, and that’s using 404 pages for dead ends, and if you really get serious, 301 redirects to send bots and visitors alike to a different destination all together. These are the big redirects however, affecting your visitors and bots alike, and are not to be taken lightly. Sculpting your traffic flow entirely to a different destination is one that needs to be taken with care and foresight. One improperly executed 301 or 404 in your website, could bring your entire marketing campaign crumbling to the ground.