move over, zerotier —

VPN mesh networks allow nearby nodes to rapidly communicate peer-to-peer.


Last month, the engineering department at Slack—an instant messaging platform commonly used for community and small business organization—released a new distributed VPN mesh tool called Nebula. Nebula is free and open source software, available under the MIT license.

It’s difficult to coherently explain Nebula in a nutshell. According to the people on Slack’s engineering team, they asked themselves “what is the easiest way to securely connect tens of thousands of computers, hosted at multiple cloud service providers in dozens of locations around the globe?” And (developing) Nebula was the best answer they had. It’s a portable, scalable overlay networking tool that runs on most major platforms, including Linux, MacOS, and Windows, with some mobile device support planned for the near future.

Nebula-transmitted data is fully encrypted using the Noise protocol framework, which is also used in modern, highly security-focused projects such as Signal and WireGuard. Unlike more traditional VPN technologies—including WireGuard—Nebula automatically and dynamically discovers available routes between nodes and sends traffic down the most efficient path between any two nodes rather than forcing everything through a central distribution point.

Getting started with Nebula isn’t too difficult, although the documentation is a little sparse. The Github repository for Nebula makes binaries for Windows, Linux, and MacOS available, along with a sample configuration file. The config files consists of pki, static_host_map, lighthouse, listen, tun, and firewall sections.

  • The first three sections of config.yml define crypto assets, already-known hosts, and “lighthouses”—Nebula nodes available over normally routable networks.


    Jim Salter

  • “Listen” and “tun” define device attributes for the local Nebula node, including bound IP and port.


    Jim Salter

  • The Firewall section of config.yml is one that’ll trip you up if you aren’t careful. It defaults to deny, so if you haven’t set any allow rules, no traffic will make it into your node.


    Jim Salter

Nebula’s term for a publicly-available node in the network is a “lighthouse.” Lighthouse nodes should be available over underlying network connections without Nebula up and running, and they represent a point of entry for new nodes joining the network. But once a node has joined the network, it won’t need to route all its traffic through a lighthouse—nodes will automatically discover the most efficient path between themselves, and if that doesn’t include a lighthouse node in the middle, that’s fine. Even if traffic needs to punch across NAT “the wrong way,” there’s no problem—because each node behind NAT will make and keep open tunnels to all nodes it knows about.

  • This diagram shows the “real” network connectivity between nodes on our test network. Nat0 and Nat1 are on the same subnet and can reach one another directly; doublenat can reach nat0 and nat1 via NAT, but they can’t reach it. All can reach lighthouse via one or more NAT connections.


    Jim Salter

  • Once all nodes are connected to the Nebula mesh, they’re capable of reaching one another directly, despite the NAT barriers between them—distinctions between one real-world subnet and another melt away.


    Jim Salter

  • We tested interconnect speeds with Nebula running—and it’s obvious that each node connected optimally, rather than routing everything through normal network paths.


    Jim Salter

We tested this by connecting four nodes in a small Nebula network: a lighthouse node hosted at Digital Ocean—which we creatively named lighthouse—and three member nodes in a small office. Two of our member nodes (nat0 and nat1) are on the main LAN in the office, and the third member node, doublenat, is on a separate subnet, connected behind node nat0.

It didn’t take much testing to confirm that Nebula’s promise to automatically discover the best route works as advertised. When running an iperf3 network speed test from nat1 to doublenat, we got 674Mbps throughput, making it painfully clear that packets aren’t being routed through lighthouse, which is in Digital Ocean’s New York datacenter several hundred miles away. Instead, doublenat has punched a tunnel outward through the NAT (Network Address Translation) layer directly to nat1, and the two hosts can use that established tunnel to communicate directly.

We can already hear some of you clamoring, “can I use this to escape obnoxious networks with overbearing firewalls?” and the answer—sorry!—is “probably not.” Just like WireGuard, Nebula operates over UDP only—so overzealous firewalls that won’t allow WireGuard connections won’t allow Nebula either. This also sharply limits its value as an exfiltration tool, since a big wash of outbound traffic on an arbitrary UDP port will stick out like a sore thumb to any network analysis tools, even if the firewall allows it.

We think that the greatest potential value to using Nebula over a more conventional VPN tool like WireGuard is its ability to discover the most efficient routes wherever it happens to be. If you have Nebula running on your laptop, your home PC, and a Digital Ocean droplet, the laptop will communicate with the PC at LAN speeds when it’s at home and Internet speeds when it’s on the road.

Listing image by NSSDC Photo Gallery

Read More