The invisible side of the Internet

The Internet has reached an unimaginable size: more than 3 billion people surf the net tapping more than 50,000 search request into Google’s search engine per second. Even harder to believe is the fact that the search results produced by the most important search engines only cover a fraction of the web.

Google ceaselessly searches for new websites and changes to existing sites. The search bot (Googlebot) continuously scours the Internet in a tireless effort to index websites. If you tap a search into Google, the search engine will show you the pages that have been found and indexed by the Googlebot. However, the displayed search results only represent the public part of the Internet.

The majority of websites may be found in the deep web

When large search engines such as Google, Bing or Yahoo do not show certain websites in the search result rankings, the sites are simply missing in their index. Since the search engines could not find and list these pages, one must know their exact address to be able to access them. The number of non-indexed pages is significantly higher than the number of indexed pages. Hence the deep web is much larger than the easily accessible Internet. The lack of indexation in the deep web can be caused by a number of different reasons:

• The pages are password protected.
• The pages are not linked anywhere.
• Code has been integrated in the pages to keep search engines from finding them.
• The pages do not contain any text.
• The pages have simply drowned in the colossal amount of content.

The dark web: flight from censorship and prosecution

Anonymous tor networks represent another category in the web. These networks are often described under the term dark web, because they can be used for illegal activities. However, the dark web might also be used to escape censorship and repressive governments. In order to be able to access services in the dark web (so-called hidden services) one must first be connected to the appropriate tor network. This can be done through the tor browser, which is an adapted Mozilla Firefox. Unlike the «normal» Internet, data is always transferred in an encrypted format when being sent via the tor network.

Source: www.internetlivestats.com