Search engines such as Google, Yahoo and Bing find information using links and sitemaps to other websites. The search engines can find some of the most obscure information and include it in the index. This type of deep search technology has some advantages and disadvantages for end users.
Inner Web Pages
Inner web pages consist of the website's pages that take several clicks before the user is able to view them. These pages may be product pages, content or a database of searches that web search engine spiders cannot normally crawl. The advantage of this type of crawl technology is that website owners can have products, shops, information and other valuable links included in a web search engine's index. This leaves more visibility on the Internet for the website owner.
One disadvantage of a deep search engine crawler is personal information being indexed regardless of privacy. Information such as social security numbers, financial information or geographic locations can be indexed even if it's posted to a personal website. Search engines have given users the ability to block some information from the index using a file called "robots.txt." Search engines such as Google allow users to remove URLs after the website owner has removed the offending information, which deletes it from the index.
Website owners can have web pages indexed by dropping a few links on another website or within their own site. This practice is called "backlinking." A website that is backlinked automatically gets crawled by the search engine, which can then map and index a website. This makes it much easier for a website owner since they do not need to submit a domain name to the search engines. Automation of the index makes it easy to be found by readers on the Internet.