Pages Should Not Appear in Search Engine

Not all the pages need to appear in search engine search result.

 

eg:

1. Error Pages (general error, page not found, forbidden, etc)

Reason: We don't want people click the link in Google and directly show error page.

 

2. Pages in pop up iframe
Reason: We don't want people click the link in Google and see incomplete page.

3. Maintenance Pages
Reason: We don't want people click the link in Google and see maintenance page.

4. Pages with Affiliate ID

Reason: We don't want people click the link in Google and let affiliate earn the money.

 

We can update robots.txt to disallow search engine web crawler crawl the page and show in search engine search result.

 

eg:

http://www.xxx.com/robots.txt

Portal totally do not want to show in search engine search result.

Other that robots.txt, we also can add meta in the page to disallow search engine crawl the page.