THE HIDDEN WEB
There's More to the Internet Than You Can See
There's something rather amusing about the way people talk about the hidden web.
Apparently, if a website doesn't appear when you search for it, it has somehow wandered off into the murky depths of the Internet and is probably up to no good.
This seems to be particularly true once somebody mentions the words deep web or dark web. At that point the sensible conversation about how the Internet actually works tends to disappear and we're suddenly talking about drugs, stolen credit cards, hackers and all manner of things that would make a Daily Mail headline writer rub his hands together with excitement.
But there's a rather more interesting story here.
The Internet has always contained far more information than the stuff that is immediately visible. Search engines are incredibly useful, but what they give us is a way of finding information that has been made available to their indexing systems. They don't somehow shine a light over the entire Internet and tell us everything that exists.
This matters because not being indexed is not the same thing as being hidden in any sinister sense.
Take something as ordinary as a database.
A website might contain millions of records, but those records aren't necessarily sitting there as millions of separate pages. You go to the website, enter what you're looking for, and the server produces the result. The information has been there all along; you just asked the system to show it to you.
There are vast numbers of websites that work like this.
There are also plenty of places on the Internet where the information is intended for a particular audience. A members' area, a company system, an archive, a specialist research collection, an online service requiring an account. None of this is particularly mysterious. It is simply information that isn't being offered to the entire public as a collection of searchable web pages.
And this is where the phrase deep web comes in.
It sounds rather grand, but in practice it covers an enormous amount of completely mundane Internet activity. It's the stuff that's there but isn't sitting out in the open where a search engine can wander through it.
That is a much more useful way of looking at it than the usual picture of an iceberg, with the innocent Internet floating on top and some enormous dark mass lurking underneath.
The iceberg picture makes it look as though the Internet has been deliberately divided into levels of increasing unpleasantness.
It hasn't.
What we actually have is an enormous collection of information and services with different ways of controlling access to them. Some are openly available, some require you to know where to look, some require an account, some are generated when you ask for them and some aren't intended to be indexed at all.
That's the deep web.
The dark web is where things become technically rather more interesting, because now we're talking about something different.
A dark-web service isn't merely a normal website that has managed to avoid being indexed. It operates through a network or system specifically designed to provide a different sort of access. Tor is the best-known example, and its onion services allow websites and other services to operate without exposing their conventional network location in the way an ordinary website does.
And that's where the subject usually gets hijacked by the word dark.
It's a wonderfully dramatic name, and the media has made very good use of it.
You'd think we'd discovered an underground version of the Internet where everyone has a false moustache, a stolen passport and a large quantity of drugs.
There are criminals using it, obviously. There would be something seriously wrong with the world if a system designed to provide anonymity wasn't eventually discovered by people who wanted to remain anonymous while doing things they shouldn't.
But anonymity itself isn't a criminal activity.
That's the bit I think gets missed.
There are perfectly legitimate reasons for not wanting every Internet connection to reveal who you are and where you are. There are people who need to communicate without identifying themselves, people working under censorship, journalists dealing with sources, whistleblowers, activists and people who simply don't think that privacy should be considered suspicious behaviour.
The same technology can serve all of them.
That's hardly unique to the Internet. A telephone can be used to order a pizza or organise a bank robbery. The telephone itself doesn't particularly care.
The dark web is much the same.
What makes it interesting isn't that criminals have discovered it. Criminals have discovered practically every useful technology humanity has invented.
What's interesting is the technology that makes anonymous communication possible in the first place, and why anybody might want that ability.
Once you get past the silly iceberg diagrams and the inevitable television programme with ominous music, the whole subject becomes considerably less mysterious.
The hidden web isn't really a secret second Internet.
It's more like the enormous amount of the Internet that doesn't happen to be sitting in the part you can see.
And that's probably the most important thing to remember.
We have become so accustomed to finding things by searching that we've started confusing searchable with existent.
Those aren't the same thing.
There can be an enormous amount of information sitting behind a web interface that isn't in a search index. There can be communities and services that deliberately keep themselves out of general search. There can be information that only appears when somebody asks for it. There can be entire services running through networks that were never intended to be part of the conventional web in the first place.
None of that tells you whether the material is good, bad, useful, boring, illegal, fascinating or utterly pointless.
It tells you how it is made available.
There's another reason this matters, and it's becoming harder to ignore. Search engines have become remarkably good at giving us an answer while becoming rather less useful at helping us find anything. Increasingly, the search result itself is the destination. You ask a question and the system extracts what it thinks you wanted to know, wraps it up neatly and puts the answer in front of you, often without giving you much reason to visit the website where that information actually came from. That's convenient if all you wanted was the answer to one question, but it's a terrible way of discovering things. Quite often the interesting bit is what you find while you're looking for something else. You arrive at a website because it has the information you were after, notice something else that catches your eye, follow a link, read an article you hadn't been looking for and suddenly you're somewhere completely different. That's how you discover things. As search becomes increasingly stuffed with advertising, commercial priorities and AI-generated answers designed to keep you on the search engine rather than send you elsewhere, that sort of discovery is becoming harder. Which makes learning how to find information outside the surface web rather less of a curiosity and rather more of a useful skill. The further this goes, the more important it may become to know how to look beyond the convenient answer somebody has decided to put in front of you.
That's an important distinction, because the moment we start treating invisible as a synonym for sinister, we've stopped explaining the Internet and started telling ghost stories about it.
And the Internet is quite weird enough without us having to make things up.
FINDING THE INTERNET FOR YOURSELF
If all of this has made you wonder how you're actually supposed to find some of this stuff, there are plenty of ways to start looking without relying on the same handful of enormous search engines.
One of the most interesting is Marginalia Search. It's an independent search engine built around discovering the less commercial parts of the web, and its philosophy is almost tailor-made for what we're talking about here. It deliberately gives more attention to smaller, independent and less search-engine-optimised websites. In other words, the sort of places that can be remarkably difficult to find when you're searching through the commercial web.
Then there's Wiby, which takes a similar idea in a slightly different direction. Wiby describes itself as a search engine for the classic web, and it can be particularly good at turning up small personal websites and other places that have somehow escaped the attention of the modern search machinery. It even has a Surprise Me option, which is possibly the closest thing a search engine has to saying, "Go on then, let's see what we find."
Mojeek is another interesting alternative because it maintains its own independent search index rather than simply borrowing the results of one of the major search engines. That doesn't mean every result will suddenly become wonderful, but it does mean you're looking at a different map of the web.
Brave Search is another independent search option, with its own index rather than relying entirely on one of the major search providers. It's worth having around simply because different indexes can produce very different views of the same Internet.
Then there is the Internet Archive's Wayback Machine, which is useful for a completely different reason. The current web is constantly changing, disappearing and being redesigned into something that somehow manages to be both more complicated and less useful than what was there before. The Wayback Machine lets you look back at previous versions of websites, which means that something disappearing from the current web doesn't necessarily mean it has vanished forever.
For anyone who wants to go considerably further down the rabbit hole, Common Crawl provides a huge open repository of web-crawl data. This is much more technical territory, but it demonstrates something important: there are ways of working with collections of web data that don't involve asking a commercial search engine what it thinks you ought to see. Common Crawl has been collecting web data since 2008 and makes its crawled data available for research and analysis.
And then we come to something which fits this whole idea rather nicely: webrings.
A webring is almost the complete opposite of the modern search-engine model. Instead of an algorithm deciding what you should see, somebody who runs a website simply says, "If you like what I'm doing, here's a bunch of other people doing something similar."
You follow the ring.
You find another site.
That site points you towards another one.
Before long you've wandered into some completely unexpected corner of the Internet and discovered something you weren't actually looking for.
That is a very different sort of discovery from typing a question into a search box and having an AI produce three paragraphs which apparently contain everything you need to know.
The IndieWeb Webring page is a good modern example of the idea still being alive.
And perhaps that's the real lesson here.
You don't necessarily need to go looking for the dark web to find the parts of the Internet that are difficult to discover. There is an enormous independent web sitting perfectly openly on the ordinary Internet. You just need to stop expecting one search box to introduce you to all of it.
There are search engines which index different parts of the web, archives which preserve what has disappeared, specialist databases, independent websites, communities, directories, RSS feeds and webrings. There are people still building websites because they have something they want to share rather than because somebody has worked out how to monetise the page.
That's where the fun starts.
Think of it less as searching for the hidden web and more as going exploring.
And if that means occasionally finding a website last updated in 2007, written by somebody who is inexplicably obsessed with restoring Sinclair computers, documenting obscure railway signalling systems or breeding unusually aggressive tomatoes, then frankly you've probably found the Internet working properly again.
If you want to explore Tor
Tor is worth including here because it is an important part of understanding the dark web, but it shouldn't be confused with the whole idea of finding information outside the mainstream search engines.
The safest place to get Tor Browser from the Tor Project is the Tor Project itself. Tor is designed around privacy, anonymity and resistance to surveillance and censorship, and its onion services provide access to sites which aren't part of the conventional web.
For discovering publicly indexed onion services, Ahmia provides a search engine for Tor hidden services. That makes it a useful illustration of the distinction made earlier in this article: even something described as "hidden" can have ways of being discovered.
Just remember that Tor is a privacy technology, not a magic certificate of trustworthiness. A site being difficult to find doesn't make it interesting, trustworthy or worthwhile. The same rule applies everywhere else on the Internet.
The point isn't to go hunting for somewhere dark.
The point is to realise that you don't have to accept somebody else's version of the Internet as the whole thing.