Meta robots
The robots meta tag let we :
- Utilize a granular,
- Page-specific approach to controlling how an individual page should be indexed and served to users in search results.
<!DOCTYPE html>
<html><head>
<meta name="robots" content="noindex" />
(…)
</head>
<body>(…)</body>
<html><head>
<meta name="robots" content="noindex" />
(…)
</head>
<body>(…)</body>
</html>
The robots meta tag at the above, instructs search engine not to show the page in search results.
Robots-text attribute value, specifies that the directive applies to all crawlers.
To address a specific crawler, replace the robots value of the name attribute with the name of the crawler that you are addressing. Specific crawlers are also known as user-agents.
