X-Robots-Tag Directive

X-Robots allow webmasters  flexibility control over which pages and documents are crawled and indexed by search engine.

Currently Yahoo X-Robots Tag Directive provide control for archiving and summarization of ANY file type with following tag:

NOINDEX, NOARCHIVE, NOSNIPPET, NOFOLLOW

Webmasters no longer need access to html templates in order to express exclusions for html files. Previously, these page level tags could only be expressed within html pages through the META directive (for e.g. <META NAME="Slurp" CONTENT="NOARCHIVE">), with X-Robots Tag Directive .

Webmasters has flexibility to achieve exclusions on PDF, Word documents, PowerPoint, video, and other file types, including html files, and increasing their coverage through a simplified process. Simply add the following page level tags to the X-Robots-Tag directive in the HTTP Header. Here are a few examples:

• X-Robots-Tag: NOINDEX —Do Not show the URL in the Yahoo! Search results.

  Note: They'll still need to crawl the page to see and apply the tag, so if you don’t wish to have the page crawled, use robots disallow on robots.txt.

• X-Robots-Tag: NOARCHIVE —Do Not display cache link in the search results page.
• X-Robots-Tag: NOSNIPPET — Do Not display summary in the search results page.
• X-Robots-Tag: NOFOLLOW — Do Not  crawl links in the page.

If you use the apache server, you may use the htaccess document to make the revision to a HTTP document. Revises a HTTP document with caution, if revise wrongly will possibly cause the search engine not to be able to carry on the index to your page. You may also use the following Meta label to replace X-Robots-Tag, they have the same function.

<meta name="robots" content="noarchive">
<meta name="robots" content="nosnippet">
<meta name="robots" content="nofollow">
<meta name="robots" content="noindex">

 

 

Source reference:Yahoo search.