A new search engine is now capable of investigating the deep web. Read more about the hidden internet that most of us never see and what is being done to find it.
The deep web is far too big for anyone to grasp. In fact, it’s estimated that we tap into less than 5% of all the content on the internet via search engines such as Google or Bing. If the internet is really this big, why are we limited to a measly 5%?
First off, let’s define deep web and surface web. The surface web is basically what we see every day. It is made up of news sites, Facebook, Amazon, ESPN.com and millions of other such websites. The surface web is what we get back when we enter a search in Google. However, this makes up a very small portion of the internet.
The deep web makes up the other 95%. It consists of forgotten databases, academic journals, public documents posted by organizations, and Tor. Tor is the dark and many times illegal portion of the internet. It contains millions of websites that end in .onion which require special software to access them. This is done to keep those who crawl around the deep web anonymous.
As you can imagine, the deep web is commonly used for illegal activity. When talking about this dark corner, the deep web is referred to as the dark web, or anonymous web. Pornography, human trafficking, drugs and illegal weapons are just some of the illegal actions going on in the dark web. Since it is very difficult to access, the general public rarely comes across these highly illegal activities.
Those who do search into the deep web can snoop around without being tracked. This is done by onion routing, a technique originally developed by the U.S. Navy that enables anonymous communication over a computer network. Tor, as mentioned above, is the free software that enables onion routing. Without Tor, the deep web would not be accessible.
So why don’t we see the deep web? Search engines use “crawlers” or “spiders” in order to fetch search results. This works similar to a spider web in the sense that everything branches out and expands. These crawlers use hyperlinks and words within the HTML code to jump from site to site. The crawlers form an index that eventually becomes the search results. Many websites fall into the deep web because there is no way to link to them, keeping crawlers from finding these sites.
Crawlers don’t reach the deep and at times dark parts of the web that requires Tor to be accessed. This – as stated above – makes it the perfect place to spread illegal content.
However, the Defense Advanced Research Projects Agency, or DARPA, has recently released a new project that is basically a search engine that sneaks into the deep and dark web. This new project is called Memex. According to DARPA.mil, Memex is designed to “revolutionize the discovery, organization, and presentation of search results.”
Memex was particularly designed to track illegal activity that is occurring in the dark web. One of the main focuses of DARPA’s innovative project is to target sites that are involved in human trafficking.
Memex made its public debut on 60 minutes. The inventor of Memex, Chris White, explained that Memex creates a web that can be used to track what is going on in the dark web. Through this web, government officials can actually make connections between sites that contain illegal activity. Once a connection has been made, the idea is to pinpoint who is behind the connected sites.
Cyrus Vance Jr.,the incumbent New York County District Attorney, said that Memex is “Google on steroids.” He, along with thousands of others, hopes to see Memex being used to stop human trafficking around the world.
In fact, it is already being used in the United States to monitor and track the dark web. Scientific American stated that DARPA is currently working with 17 different teams of researchers to organize and interact with data in order to track illegal activity.
DARPA plans on moving the Memex program forward, eventually tracking the deep and many times dark web.