To avoid undesirable material inside the research indexes, site owners can instruct spiders never to crawl sure information or directories with the common robots.txt file in the foundation directory in the area. Also, a page may be explicitly excluded from a online search engine's database by using a meta tag https://news.rafeeg.ae