All Posts
Routing and BGP: How the Whole Internet Agrees Where to Send Your Data

Routing and BGP: How the Whole Internet Agrees Where to Send Your Data

Series: Networking Foundations | Post 9 of 14 Everything in this series so far has quietly assumed one thing: that a packet can reach any destination on earth. But the internet is not one network. It is tens of thousands of separate networks, owned by rival companies, different countries, and organisations that have never met, with no central map and nobody in charge. So how does your data find its way across all of them? The answer is routing, and at global scale it is a protocol called BGP, which holds the entire internet together with something surprisingly close to a handshake and a promise.

We have come a long way. We have addresses, names that resolve to them, reliable connections, and encryption that protects what those connections carry. But every single one of those posts rested on an assumption we never examined: that a packet leaving your device can actually find its way to any server in the world, wherever it happens to be.

That assumption hides one of the most remarkable feats in all of engineering. Because the internet is not a single, tidy network with a master plan. As we first noted back in Post 1, it is a network of networks, tens of thousands of them, each independently owned and operated, by internet service providers, corporations, universities, governments, and rivals who may compete fiercely or even distrust one another. There is no central authority that knows where everything is. There is no master map of the internet stored anywhere.

And yet, somehow, a packet from your laptop can reach a server on the other side of the planet in a fraction of a second, crossing a dozen of these independent networks on the way, each one correctly handing it onward. How? That is the question this post answers, and the answer comes in two parts: routing, the general idea of moving data toward a destination, and BGP, the specific protocol that lets all those independent networks agree on how to reach each other.

Let us build it from the ground up.


What Routing Actually Is

At its simplest, routing is the process of choosing where to send a packet next so that it moves closer to its destination. The devices that do this are routers, and they sit at the boundaries between networks, forwarding packets from one to the next.

Here is the single most important idea about routing, and it is worth pausing on because it is deeply counterintuitive: no single router knows the entire path to the destination. A router does not plan the whole journey. It knows only one thing: given a packet's destination address, which neighbour should I hand it to next to move it in the right direction? That is all. Each router makes one local decision, passes the packet along, and the next router makes the next decision.

This is hop-by-hop forwarding, and the analogy that captures it is a traveller with no map crossing a country by asking for directions at each town. Nobody along the way knows the entire route. Each person knows only which road leads toward the next town in the right direction. Follow enough of those local pointers, and you arrive, without anyone ever having held the whole journey in their head.

How does a router know which neighbour is the right next step? It keeps a routing table, a list that maps ranges of destination addresses to the next hop that leads toward them. When a packet arrives, the router looks up its destination in this table, finds the best matching entry, and forwards the packet to the corresponding next hop. The entire art of routing comes down to how these tables are built and kept correct, because a routing table is only as good as the information in it.

There is one rule about that lookup worth knowing now, because it becomes critical later: when several entries in the table could match a destination, the router chooses the most specific one, the one covering the smallest, most precise range of addresses. A route to a narrow, exact block of addresses beats a route to a broad one. Remember this. It is the mechanism at the very centre of the attack we will examine at the end.


Two Kinds of Routing: Inside a Network, and Between Networks

Routing happens at two very different scales, and telling them apart is the key to understanding the whole system.

Interior routing happens within a single organisation's network, one autonomous system, which we will define properly in a moment. Inside your company or your ISP, routers need to know how to reach every internal destination, and they figure this out among themselves using interior routing protocols, with names like OSPF and IS-IS. These protocols let routers inside one network automatically share what they know and calculate the best internal paths. Because everyone inside is under one administration and mutually trusts one another, these protocols can afford to share detailed information freely. This interior world is, incidentally, exactly where much of the deeper CCNA-level networking lives, and it is a topic worthy of its own series later.

Exterior routing happens between these independent networks, and it is a fundamentally different problem. Here, the networks do not trust each other by default, do not share internal details, and each wants to enforce its own policies about who it will carry traffic for and how. This is the harder, more political problem, and solving it is the job of one protocol above all others: BGP. To understand BGP, we first need to understand what these independent networks actually are.


Autonomous Systems: The Building Blocks of the Internet

The internet is divided into tens of thousands of independent networks called autonomous systems, or AS for short. An autonomous system is a network, or a collection of networks, under the control of a single organisation that presents one consistent routing policy to the outside world. Your ISP is an autonomous system. A large company like Amazon or Google is an autonomous system, often several. A university is one. Each is a large pool of routers and addresses run by one operator.

Every autonomous system is assigned a unique identifying number, an Autonomous System Number, or ASN, handed out by a central numbering authority. These numbers are how networks refer to one another. Amazon's main autonomous system, for example, is AS16509, a number you will see again shortly, because it is central to our closing story. Originally these numbers were 16-bit values, allowing about 65,000 of them, but the space was later expanded to 32-bit to accommodate the growth of the internet.

alt The Internet's True Structure: Autonomous Systems
alt The Internet's True Structure: Autonomous Systems

So the true shape of the internet is this: tens of thousands of autonomous systems, each an island of networks run by one organisation, all needing to interconnect so that any device in any of them can reach any device in any other. The autonomous systems link up at exchange points and through business agreements, one carrying another's traffic, forming a vast, messy, decentralised mesh. The question BGP answers is: across this mesh of mutually independent, sometimes rival networks, how does each one learn how to reach all the others?


BGP: The Protocol That Holds the Internet Together

BGP, the Border Gateway Protocol, is the protocol that autonomous systems use to exchange routing information with each other. It is, without exaggeration, the protocol that makes the internet one internet rather than tens of thousands of disconnected islands. When someone in Singapore loads a website hosted in Argentina, BGP is what worked out a path between them across all the networks in between.

The core mechanism is beautifully simple, almost startlingly so given what rests on it. BGP works by advertisement. Each autonomous system announces to its neighbours which ranges of IP addresses it can reach. In effect, a network stands up and says to the networks it connects to: "I can deliver traffic for these address ranges. Send them to me." Its neighbours record this, and then pass the announcement onward to their neighbours, adding themselves to the description of the path as they go. Those neighbours tell their neighbours, and so the knowledge of how to reach that range of addresses ripples outward across the entire internet, autonomous system by autonomous system.

As an announcement travels, it accumulates a record of the autonomous systems it has passed through, called the AS path. Each network that forwards the announcement adds its own ASN to the front of this path. So by the time an announcement has spread across the internet, it carries a full list of the autonomous systems you would traverse to reach that destination, something like "to reach these addresses, go through AS X, then AS Y, then AS Z." This AS path serves two purposes: it lets a router compare competing routes and choose among them, and it lets BGP detect loops, because if a router sees its own ASN already in a path, it knows the announcement has come back around to it and can discard it.

When an autonomous system hears about several possible paths to the same destination, it selects the best one according to a set of rules and its own policies, often preferring shorter AS paths, but also honouring the business and policy preferences of the network operator, who has the final say in how traffic flows. The chosen route goes into the routing tables, and packets follow it. Multiply this across every autonomous system on earth, each advertising what it can reach and each choosing best paths from what it hears, and out of this vast, continuous conversation emerges a working global routing system with no central coordinator at all. It is genuinely one of the most impressive emergent systems humanity has built.

alt BGP Advertisement and the AS Path (the anchor diagram)
alt BGP Advertisement and the AS Path (the anchor diagram)

But read that mechanism again, and you may notice something unsettling. A network announces which addresses it can reach, and its neighbours believe it. And that is the crack at the heart of the internet.


The Uncomfortable Truth: BGP Runs on Trust

Here is the thread this entire series keeps pulling, arriving now at perhaps its starkest example. BGP was designed in the early days of the internet, for a small community of network operators who knew and trusted one another. And so, at its core, BGP has no built-in mechanism to verify that an announcement is truthful. When a network announces that it can reach a particular range of addresses, its neighbours, by default, simply accept the claim. Route announcements are accepted on trust.

Think about what that means. Nothing in the base protocol checks whether a network actually has the right to announce the addresses it claims. If an autonomous system announces "I can reach these addresses," the announcement propagates and is believed, whether or not those addresses genuinely belong to it. The system assumes good faith. For a small, trusting network of acquaintances, that was a perfectly reasonable design. For the backbone of global commerce, it is a profound vulnerability, and, exactly as with the DNS and the certificate systems we have already studied, it is not a bug that someone forgot to fix. It is a direct consequence of the trusting design that made the system simple enough to work and to spread in the first place.

When a network announces routes it should not, we call it BGP hijacking. It can happen by accident, through a simple misconfiguration, a network mistakenly announcing addresses it does not own, and this has taken large services and even whole countries' access offline by drawing traffic into a black hole. Or it can happen on purpose, as a deliberate attack. And because of the "most specific route wins" rule from earlier, an attacker has a devastating lever: they do not even need to out-compete the legitimate announcement. They only need to announce a more specific, narrower slice of the victim's address range. Every router that hears it will prefer the more specific route, and traffic for that slice will bend toward the attacker. This is precisely how the incident this series has alluded to three times finally unfolded.

alt BGP Hijacking: Why the More-Specific Route Wins
alt BGP Hijacking: Why the More-Specific Route Wins


The MyEtherWallet Heist, Explained at Last

Back in the DNS post, and again since, I mentioned an attack where people typed the correct web address and still lost everything, and I promised the full explanation would come once we understood routing. Here it is.

On the morning of 24 April 2018, for roughly two hours, a small internet service provider in Ohio, eNet, autonomous system AS10297, began announcing BGP routes for a set of IP address ranges belonging to Amazon's Route 53 DNS service. Those addresses legitimately belonged to Amazon, AS16509. But eNet announced more specific slices of them, and by the rule we now understand, more specific routes win. Networks across the internet that heard these announcements believed them, exactly as BGP is designed to, and began sending traffic destined for Amazon's DNS servers toward infrastructure the attackers controlled instead.

Now recall what Route 53 does: it is a DNS service, the phone book from Post 5, answering the question "what is the IP address for this domain?" With the traffic to Amazon's DNS servers hijacked, the attackers' machines stepped in and impersonated that phone book. When users tried to visit MyEtherWallet, a cryptocurrency wallet site, their computers asked for its address, and the imposter DNS server answered with a lie: the address of a phishing site the attackers had built.

Here is the detail that ties this entire series together. MyEtherWallet's own servers were never breached. Amazon's DNS servers were never breached. The DNS records themselves were never even altered. Nothing was hacked in the way people usually imagine. The attackers simply hijacked the route to the DNS service, poisoning the answers at the level of the internet's own plumbing. The users did everything right. They typed the correct address. And the internet, trusting a false announcement at its most fundamental layer, delivered them to a thief.

There was one last line of defence, and it connects directly to the previous two posts. The phishing site could not present a valid TLS certificate for MyEtherWallet's domain, because, as we learned in Post 8, the attackers did not control a certificate the browser's trust store would accept. So users were shown a certificate warning, the browser correctly telling them something was wrong. Tragically, many clicked past the warning and proceeded anyway, and roughly $150,000 in cryptocurrency, by some later accounts considerably more, was drained in those two hours. The encryption layer did its job and raised the alarm. The humans overrode it. That, too, is a lesson this series keeps teaching: the technical safeguards are only as strong as our willingness to heed them.

alt The MyEtherWallet Attack Chain, Finally Visualised
alt The MyEtherWallet Attack Chain, Finally Visualised

One attack, and it wove together three layers we have studied: a BGP route hijack, enabling a DNS impersonation, defeated only partly by a TLS certificate warning that users ignored. You now understand all three well enough to see exactly how each link in that chain worked.


Defending Against It: Hardening a Trusting System

As with DNS and certificates, the fix is not to throw away BGP, which is impossible, no more able to be swapped out overnight than any other piece of the internet's foundation, but to add layers of verification and vigilance on top of its trusting core. The main efforts are worth knowing at a high level.

The most important is RPKI, Resource Public Key Infrastructure. It lets the legitimate holder of a range of addresses publish a cryptographically signed record, essentially a statement declaring "this autonomous system is authorised to announce these addresses." Other networks can then check announcements against these signed records and reject ones that fail, a mechanism called route origin validation. Notably, in the months after the MyEtherWallet incident, Amazon published exactly these records for its address ranges, so that the specific hijack that occurred would be caught in future. RPKI is, in spirit, doing for routing what certificate authorities do for websites: adding a layer of cryptographic authorisation to a system that originally ran on trust alone.

Alongside it sit route filtering, where networks configure their routers to reject obviously improper announcements from their neighbours, and route monitoring, services that continuously watch global BGP announcements and raise the alarm when someone announces addresses they should not, so an operator can react fast. In the MyEtherWallet case, some large, well-run networks had filters in place that limited how far the bad announcement spread, which is why the damage, though serious, was not far worse.

But, and this is the familiar refrain, none of these is universally deployed, and the same deployability wall we met with DNS and IPv6 applies here too: you cannot force the entire internet, tens of thousands of independent operators, to adopt anything at once. Adoption of RPKI has grown substantially, but large portions of the global routing table still remain unprotected. So the honest picture is that the internet's routing core still rests, to a significant degree, on trust and good behaviour, propped up by a growing but incomplete scaffolding of verification. It works remarkably well, most of the time, for the same reason it always has: because most operators are honest most of the time.


Commands to See It Yourself

You can watch routing happen from your own machine.

Trace the path your packets take across the internet:

traceroute example.com        (on Linux or macOS)
tracert example.com           (on Windows)

This shows each hop, each router, that your packets pass through on the way to the destination. You are watching hop-by-hop forwarding directly: each line is one router handing your packet onward toward its destination. You will often see the packets crossing between different networks and providers as they travel.

See the autonomous system a route passes through:

mtr example.com               (a live, continuously updating traceroute on Linux/macOS)

Some versions and related tools can annotate hops with the autonomous system numbers they belong to, letting you watch your traffic cross from one AS to the next.

Explore real global BGP data: public "looking glass" services and route-collector projects, run by various network operators and organisations, let you query the actual BGP routing tables of the internet from your browser. You can look up any address range and see which autonomous system announces it and what AS paths lead there, the real advertisements this whole post described, live.


What You Now Understand

You came in with a question you had never thought to ask: how does a packet cross an internet made of tens of thousands of independent, mutually distrustful networks, with no central map? Now you know.

You understand routing: the hop-by-hop process where no single router knows the whole path, each just choosing the best next step from its routing table, preferring the most specific matching route. You understand the split between interior routing, which happens within one trusting organisation using protocols like OSPF, and exterior routing, which happens between the independent networks and is far harder.

You understand the true structure of the internet as a mesh of autonomous systems, each an organisation's network identified by an ASN, and you understand BGP, the protocol that stitches them together by having each announce what it can reach, propagating those advertisements with their AS paths across the whole world until a global routing system emerges from the conversation, with nobody in charge.

And you understand the crack at its foundation: BGP runs on trust, believing announcements by default, so a false or too-specific announcement can hijack traffic, as it did in the MyEtherWallet attack, where a route hijack enabled a DNS impersonation that sent users who did everything right to a thief, stopped only partly by a certificate warning they overrode. You understand the defences, RPKI, filtering, and monitoring, and why, like every fix in this series, they are partial, held back by the same wall: you cannot upgrade the whole internet at once.

Three times now this series has arrived at the same shape of truth. DNS trusts its answers. Certificates trust their authorities. And routing trusts its announcements. The internet was built for connection, not control, and its greatest vulnerabilities are the shadow side of the very openness that let it grow.

In the next post, we come back down to earth, to the edge of your own network, and examine the device and the technique standing between your private devices and the whole wild internet we have just been touring. We will look at NAT, network address translation, what it really does, and the honest story of what protection it does and does not provide. That is Post 10.


This is Post 9 of the Networking Foundations series. If BGP and the trust at the heart of the internet finally make sense, and you can now picture how the MyEtherWallet attack actually worked, share it with someone who assumes the internet has a map. New here? Start with Post 1, and Post 8 on TLS and certificates leads into this one. Subscribe to our newsletter to get each new post as it publishes.

Enjoyed this post?

Get notified when I publish next.

No spam — only new posts on networking, security, DevOps and infrastructure.

Comments

Leave a comment