The term Dark Web is evocative. It conjures up images of hitmen, illegal drugs, and pedophilia. One imagines a place where the dark side of human nature flourishes away from the eyes – and laws – of society at large.
The above infographic cuts through the mystique and provides an entertaining and practical overview of the Deep Web and the Dark Web.
Layers (Part 1)
Much like the ocean, the internet is divided into defined layers.
The internet most people are familiar with is called the Surface Web. Websites in this layer tend to be indexed by search engines and can be easily accessed using standard browsers. Believe it or not, this familiar part of the web only comprises less than 10% of the total data on the internet.
The next layer down, we encounter the largest portion on the internet – the Deep Web. Basically, this is the layer of the internet that is quasi-accessible and not indexed by search engines. It contains medical records, government documents, and other, mostly innocuous information that is password protected, encrypted, or simply not hyperlinked. To reach beyond this layer of the internet, users need to use Tor or a similar technology.
Layers (Part 2)
Tor, which stands for “The Onion Router”, is how the majority of people anonymously access the Dark Web. Tor directs internet traffic through complex layers of relays to conceal a user’s location and identity (hence the onion analogy).
In 2004, Tor was released as an open source software. This allowed the Dark Web to grow as people could anonymously access websites.
Since anonymity is sacrosanct in the deep reaches of the Internet, transactions are typically conducted using cryptocurrencies like Bitcoin or Ethereum. People making purchases in Dark Web markets are (understandably) concerned with privacy, so they often use a series of methods to transfer funds. Below is a common transaction flow on the Dark Web.
Tumblers are used as an extra step to ensure privacy. A conventional equivalent would be moving funds through banks located in countries with strict bank-secrecy laws (e.g. Cayman Islands, Panama).
What’s Going on Down There?
The concept of the Dark Web isn’t vastly different from the Surface Web. There are message boards (e.g. 8chan, nntpchan), places you can buy things (e.g. Alphabay, Hansa), and blogs (e.g. OnionNews, Deep Web Radio). The rules, or rather a lack thereof, is what makes the Dark Web unique.
Anything that is illegal to sell (or discuss) on the Surface Web is available in the Dark Web. Personal information, drugs, weapons, malware, DDoS attacks, hacking services, fake accounts for social media, and contract killing services are all available for sale.
The Dark Web is full of criminal activity, but it’s also place where dissidents and whistle-blowers can anonymously share information. In countries with restrictive internet surveillance, the Dark Web may be the only place to safely voice criticisms against government and other powerful entities.
Measuring in the Dark
Many .onion sites are only up temporarily, so determining the true size of the Dark Web is nearly impossible. That said, Intelliagg and Darksum recently attempted to map out the Tor-based Dark Web by using a script to crawl reachable sites. They found 29,532 websites; however, 54% of them disappeared during the course of their research. 87% of Dark Web sites don’t link to any other sites.
It is more accurate to view the darkweb as a set of largely isolated dark silos.
Recent changes to Tor, such as 50-character hidden service URLs, have made the Dark Web an even more untraceable place, so we may never fully know what lies beneath the surface of the internet. Based on the parts we have seen, perhaps that’s for the best.
A digital replica is a digital representation of an individual, object, or asset. Such a representation is constructed based on an individual, object, or asset’s interactions with its environment. In physical objects and assets, the concept has recently picked up steam with the easy availability of sensors and internet-of-things (IoT) connectivity. For instance, GE reportedly has 66,000 jet engine, locomotive, and turbine assets, each of which has a unique digital replica. The digital replica of its jet engine, for instance, relies on a variety of sensors that capture and transmit detailed information as the jet engine operates. This information is processed on a platform (Predix) using advanced modeling techniques. As a consequence, GE not only gets accurate predictions of how each component in the jet engine is faring but also is able to optimize the operation of the jet engine for maximum efficiency. The more elaborate and accurate the sensors, and the more the asset operates and interacts with its environment, the more precise its digital replica becomes. Using digital replicas, GE offers its customers new services such as predictive maintenance, which helps them plan for maintenance before a critical part of the engine breaks down. GE also offers outcome-based pricing, appropriating a part of the cost savings in fuel because of increased efficiency in the engine’s performance.
Digital replicas are found in a growing number of physical assets, such as mattresses, toothbrushes, cars, excavators, CT scanners, dishwashers, and refrigerators. As most of such assets have a finite range of functions and interactions with their environments, it is understandable that the firms that own and operate them also have complete control over their digital replicas. Individuals, however, are far more complex than physical assets; their digital replicas understandably are much harder to construct and control. Yet because of the ubiquity of e-commerce platforms, social media, IoT devices, smartphone apps, and the internet, most individuals leave substantial digital traces of their preferences, behaviors, and personas. They do so because the more intricate information they provide about themselves, the more customized services they get in return. This provides the digital titans with an unprecedented opportunity to collect all possible facets of this information and to construct as complete a digital replica as possible for every individual, one that most closely predicts peoples’ behavior. The company that monopolizes this realm will wield far more power than what we have seen thus far.
When Google developed its search engine, it probably never thought that it would start a journey toward building digital replicas for individuals. Every search we conduct offers select facets of our persona to Google. Similarly, every movie we watch through Netflix, every question we ask Alexa or Siri, and every interaction we have with our friends on Facebook imparts different slices of our persona, each contributing to a full digital replica of ourselves built by a digital titan. It is not surprising that these titans are trying everything they can think of to capture different facets of our personas. This in part explains many of their new forays outside of traditional tech services into domains such as automobiles and health care. A car can provide a treasure trove of information about our behavior. Most industrial-age firms, despite venturing into digital replicas, appear more interested in embellishing their traditional product sales and services. GE may make its dishwashers and refrigerators more attractive because of predictive maintenance features through its digital replicas. By contrast, a digital titan such as Amazon appears more interested in how we interact with a whole host of appliances and other objects in our homes through Alexa.
Historically, many firms have had deep knowledge about only specific facets of an individual’s life. For example, financial institutions knew the financial lives of their customers; retailers accumulated knowledge about their customers’ buying habits; and even libraries had information on users’ reading habits. What they lacked, however, was a composite knowledge about individuals that came through the pooling of such information. Since competition was sector by sector, firms did not have the incentive to pool such information. Moreover, pooling disparate information was an onerous task, and no firm had the desire to expend resources on such an effort. In other words, the analog versions of digital replicas were only partial pictures of individuals possessed by different market participants.
The arrival of the digital age has changed this picture completely. Digital titans like Google, Amazon, and Facebook now recognize that having a complete digital replica is not only desirable but also feasible. It is desirable because of enormous cross-selling and cross-advertising opportunities that would accrue for the firm that gets there ahead of others. It is becoming feasible because of how digitization has enveloped our lives, how we routinely leave digital traces as we interact in the digital world, and through technologies, such as APIs, that are making it easier for companies to share or access different slices of information that they possess.
If access to high-quality data on customer preferences is what companies are seeking, the ability to collect, connect, and integrate data from various sources will become the new competitive moat. This moat will restrict other competitors from gaining access to and control over individual digital replicas. Such new-age monopolies may not be visible through traditional industry concentration measures, but they will wield tremendous influence over consumers. Their allure will be their ability to provide unprecedented personalization based on the information they hold. Yet with such personalization customers may be restricted to see only what the provider wants them to see.
Monopolies in traditional markets are easy to detect. But a monopoly in who gets access to how individuals think, behave, and make day-to-day decisions is not. Individuals getting lured into providing monopoly access to their digital replicas because of the convenience of personalized services may never realize that their choices are being made not by them but by their providers.
Customers could try to control their identifying information and be careful about what access they provide to companies when they are in the market for a product or service. However, that’s easier said than done. The more that personalization kicks in, the more we will expect sharing information to be the new norm.
Regulators will need to understand how to detect and prevent digital-age monopolies that enable control over individual digital replicas. While they should continue to watch out for mergers, acquisitions, and alliances that lead to market consolidation, regulators’ attention should also be trained on efforts to consolidate access to information, enabling only a few firms to piece together full individual digital replicas. More important, they should also be watching out for how firms manage their APIs. These tactics may not be as visible as alliances or mergers, but they are potent tools for data integration and piecing together digital replicas.
Governments can control the infrastructure that provides identity services, with which they can prevent any one company from having a complete digital replica. The Indian government, with its Aadhaar card and UPI services, is one such example. With this control, the government can seek users’ permission before allowing the search provider to integrate information.
Digital titans can create a protocol to share information and prevent anyone from getting an unfair advantage. This doesn’t address privacy concerns, but it would go a long way toward assuaging our fears about this new form of information monopoly. When search results are provided, the titan should show the sources from which the information is gathered. This type of transparency will allow the customer to determine the sources of bias in the information provided.
In sum, data acquisition for digital replicas can manifest itself in many forms. These quests for data may pose major challenges for regulatory authorities keen on maintaining a well-functioning market system. As digital titans expand their reach and ownership of data across multiple domains, they may create a world where the atomistic competition so beloved by economists may become impossible.