In order to avoid undesirable information within the look for indexes, webmasters can instruct spiders never to crawl specific data files or directories with the common robots.txt file in the root directory with the domain. Moreover, a page might be explicitly excluded from the internet search engine's database through the https://news.rafeeg.ae