What the internet was supposed to be — the federated peer-to-peer original vision
The hypermedia tradition treated above is one side of the frame. The other side is the internet itself, considered apart from the web that runs on top of it. The two are distinct in ways that have been blurred almost beyond recovery. The web is an application; the internet is the substrate. The web was proposed in 1989, became dominant by 1995, and is now thirty-six years old. The internet, depending on where you place the origin, is between fifty and sixty years old, and was for the first two decades of its life a coherent proposal about how networked machines should relate to one another. The proposal was federated and peer-to-peer. Every node was an equal. The work of the network was carried at the edges; the middle was deliberately stupid. There was no central operator and no central registry of services. This was not an accident. It was a design choice, made by specific people for specific reasons, and held in clear view for long enough to become culture before the web arrived and rewrote the cultural assumption to “client-server.” This chapter is about the proposal as it was originally made.
The Cold War origin of resilience
The first published proposal for a packet-switched communications network was Paul Baran’s series of memoranda at RAND, drafted between 1960 and 1964 and published in 1964 as On Distributed Communications. Baran had been given a problem by the Air Force: design a communications system that could survive a nuclear first strike. The existing telephone network was hierarchical, with a small number of long-distance switching centers that aggregated traffic from local exchanges. A small number of nuclear weapons aimed at those switching centers could disable national communications. Baran’s response was a network of equal-status nodes connected by redundant links, in which messages were broken into pieces that found their own routes, hop by hop, from source to destination. The destruction of any single node forced the surviving pieces of the message to route around it. There was no center to destroy.
Baran called the technique “distributed adaptive message-block switching.” The phrase did not catch on. Three years later, Donald Davies, working at the National Physical Laboratory in Teddington, England, independently arrived at the same scheme and coined the term that did catch on: “packet switching.” Davies’ motivation was different from Baran’s. He was thinking about how to share an expensive computer among many users efficiently, not about nuclear survivability, and he proposed packet switching as a way to multiplex traffic onto a single transmission line without dedicating the line to any one conversation. The two motivations — resilience under attack and efficient sharing of expensive resources — produced the same architecture. A network of peers, each capable of forwarding packets, with no central authority and no privileged path.
When Lawrence Roberts began designing what became the ARPANET in 1966 and 1967 at the Advanced Research Projects Agency in the U.S. Department of Defense, he drew on both Baran’s and Davies’ work. The ARPANET’s first four nodes — UCLA, SRI, UC Santa Barbara, and the University of Utah — came online in late 1969. The first message attempt was sent from UCLA to SRI on October 29, 1969, by a graduate student named Charley Kline, working under Leonard Kleinrock. The system crashed after the letters “L” and “O” had been transmitted; the full word was supposed to be “LOGIN.” This is the founding story, and the founders enjoy telling it. The point worth keeping is what the architecture was. Four computers, in four cities, treated as equals. Any of them could send to any of them. None was the center.
The choice mattered. The competing architecture at the time was the centralized timesharing model, in which dumb terminals connected to a single mainframe and the mainframe did everything. The ARPANET could have been built that way; many corporate networks of the same era were. The choice to make the network a mesh of peers rather than a hub of clients was deliberate and was justified by specific arguments — survivability, efficient use of long-distance lines, the ability of any researcher to log into any machine — that we have largely forgotten the force of because the choice has since been so thoroughly undone.
The end-to-end principle
The peer-to-peer architecture acquired, in 1981, a written justification that would shape three decades of internet engineering. The paper was “End-to-End Arguments in System Design,” by Jerome Saltzer, David Reed, and David Clark, all then at MIT. The argument was technical but its consequences were political. Saltzer, Reed, and Clark observed that many functions a network might perform — reliable delivery, encryption, deduplication, ordering — can only be done correctly at the endpoints of a communication, because only the endpoints know what counts as a correct outcome. If the network tries to do them in the middle, it either does them imperfectly (and the endpoints have to redo them anyway) or it does them so perfectly that it has to know everything the endpoints know, which makes the middle of the network as complicated as the endpoints. Therefore, the paper argued, functions that could be implemented at the endpoints should be implemented at the endpoints, and the middle of the network should be kept as simple as possible.
The technical consequence was that the internet’s middle would be a dumb packet-forwarding fabric. The intelligence would live at the edges, in the operating systems and applications on each host. The political consequence, which Saltzer and his coauthors did not fully spell out but which their successors did, was that a dumb network is a network that cannot discriminate. A router that knows nothing about what it is forwarding cannot favor one kind of traffic over another, cannot inspect content, cannot bill differently for different services, cannot be a chokepoint for control. The end-to-end principle was a technical argument that doubled as a charter for a network in which the operator of the network had no business meddling with what passed through it.
The end-to-end principle made certain things easy and certain other things difficult. It made it easy to add new applications: anyone could write a new protocol that used the network in a new way, and the network did not have to be modified. The arrival of the web in 1991 was possible precisely because the people running the network did not need to be consulted. It made it difficult to provide quality-of-service guarantees, because the network was not allowed to know whether a packet contained voice or email or a file transfer. It made it difficult to charge for individual applications, because the network was not allowed to identify them. It made it difficult to block specific kinds of content, for the same reason. These difficulties were largely seen, at the time, as features.
Smart edges, federated control
What the end-to-end principle implied for the social organization of the internet was that there was no central operator. The ARPANET had been run by ARPA but ARPA’s role was funding and coordination; the actual machines were operated by the universities and companies that owned them. When the ARPANET split into MILNET (for the Department of Defense) and the civilian internet in 1983, and when the National Science Foundation built NSFNET as the new backbone between 1985 and 1995, no single entity controlled the network. Backbones were operated by separate organizations; regional networks by separate organizations; campus networks by separate organizations. Each peered with the others through bilateral agreements. The internet was, structurally, a federation of independently administered networks that had agreed to exchange traffic on common terms.
This federation was visible at every level of the system. Email addresses had two parts: the user and the host. The host was reached through DNS, the Domain Name System, designed by Paul Mockapetris in 1983 and 1984 and rolled out in the mid-1980s. DNS was itself federated: the root servers were a small set of machines operated by different organizations, and below them every domain owner ran their own name servers for their own zone. There was no central registry of all names, only a hierarchical delegation in which each domain owner was responsible for the names below them. When you looked up a name, you walked the delegation. This is still how it works today, mostly, although the political economy around the root has become much heavier than the technical design intended.
Routing was federated. The Border Gateway Protocol, finalized in 1989, lets independent networks announce which addresses they can reach and lets other networks decide whether to believe the announcements. There is no central routing table. Each network operator makes their own decisions and the global table is the emergent product of all those decisions. This works astonishingly well and has been the substrate for most of the internet’s growth.
Governance was federated. The Internet Engineering Task Force, formed in 1986, made standards by a process David Clark described in 1992 as “rough consensus and running code.” Anyone could attend a working group meeting. Anyone could submit a draft. Drafts that worked, in the sense that multiple independent implementations could interoperate, were promoted to standards. There was no formal membership. The Internet Society was founded in 1992 as an institutional home for IETF activities, but the IETF itself remained, and remains, an open process. Compare with the telecommunications standards bodies of the same era, which were formal intergovernmental organizations with national delegations and weighted voting. The IETF was, by design, not that. The internet’s standards were made by the people building the internet, in working groups that admitted anyone willing to show up and do the work.
Numbers were federated. The Internet Assigned Numbers Authority (IANA), run for most of its history by Jon Postel personally out of his office at USC’s Information Sciences Institute, allocated address blocks and protocol numbers. Postel’s authority was technical and personal; he had been editor of the Request for Comments series since the late 1960s and was trusted to do the right thing. The institutional structures that succeeded him — ICANN, founded in 1998 — formalized what had been informal, with mixed results that are still being debated. But for the first quarter-century of the internet’s existence, the allocation of names and numbers was done by a federation of mostly volunteers, working in the open, with the rough consent of the people who used the resources.
The applications that ran on this thing
Before the web, the internet had applications. They were not built around a single uniform interface; each had its own protocol and its own conventions. They are worth listing because the diversity of the pre-web application landscape is one of the things the web’s dominance has obscured.
Email was the first and remained the largest. RFC 821, the SMTP standard, was published by Jon Postel in August 1982, codifying practices that had been in use on the ARPANET since the early 1970s. Mail was federated end-to-end: any host running an SMTP server could send mail to any other host running an SMTP server. There were no central servers, no obligatory intermediaries, no required directory of users. By the mid-1980s, email had become the dominant traffic on the internet and the dominant medium for technical and academic correspondence. It still works, almost exactly as Postel specified it, and the chapter on email’s near-miss treats what has happened to it since.
File Transfer Protocol, RFC 959 in 1985 (consolidating much older drafts), let any user on any host transfer files to and from any other host. Anonymous FTP — using “anonymous” as the username and an email address as the password — let public archives be set up at any institution that wanted to publish files. Universities ran anonymous FTP archives. Companies ran them. Hobbyists ran them on machines in their basements. The total volume of material available through anonymous FTP in 1993 was substantial — the major archives like wuarchive at Washington University and ftp.uu.net at UUNET held tens of gigabytes each, which was a great deal at the time — and there was no central catalog. You had to know where to look, or use Archie, a search tool released in 1990 by Alan Emtage at McGill that indexed anonymous FTP sites.
Usenet, treated in its own chapter, was the federated discussion system. NNTP, the Network News Transfer Protocol, was standardized in RFC 977 in 1986. By 1991 Usenet carried thousands of newsgroups and was the substrate of most online community life.
Telnet, RFC 854 in 1983, let users log into remote machines as if they were local. The Internet Relay Chat protocol, by Jarkko Oikarinen in 1988, let users participate in real-time text conversations across federated servers. Gopher, treated in its own chapter, was released in 1991. WAIS, also treated in its own chapter, was released in 1991. The MUDs and MOOs — text-based virtual worlds — accumulated their own protocols and communities. Each application had its own client. Each had its own conventions. Each was federated end-to-end on top of the same TCP/IP substrate.
The thing to take from this list is that the pre-web internet was not impoverished. It was, by the early 1990s, a fully populated services landscape with several million users across thousands of institutions, with mature governance, with rich application diversity, and with a culture of operation that took the federated peer model as a given. The web did not arrive into an empty room because the internet was empty; the internet was the room.
The structural promises
What was structurally available, before 1993, on the open internet, that has since become difficult or impossible to find:
Anyone with a static IP could run a server. There were no asymmetric residential connections in the modern sense. If your machine was on the internet, it could accept incoming connections, and so it could be a peer to any other machine. The split between “users” who only consume and “servers” that produce, which now defines almost every consumer’s relationship to the network, did not exist as a structural property. It came in with consumer broadband in the late 1990s, when ISPs began offering asymmetric DSL and cable modem service with the upload bandwidth deliberately constrained, and when they began prohibiting servers in terms of service.
There was no platform layer between you and the network. If you wrote an application that used TCP/IP, it could reach any other application on the internet that was willing to talk to it. There was no Apple, Google, Facebook, or AWS standing between your packets and the recipient. The intermediation that defines the modern web — that almost every commercially significant exchange of bits passes through one of a small number of platform companies’ infrastructures — was specifically not a property of the original design. It was added on, over decades, by a combination of commercial incentives and technical accidents that the chapters in Part IV trace.
Naming was personal or institutional. Email addresses were yours or your institution’s; FTP and Usenet handles were under your control; Gopher and WAIS resources were addressable by the running server, not by some platform’s permission. The migration of personal naming to platform-issued identities — your handle is on a site, owned by the site, and dies if the site dies — was not a property of the substrate. The substrate let you use your own name on your own machine. You can still do this, formally; almost nobody does.
The cost of participating in the network was, for most users, the cost of being at an institution that was on the network. For users with their own dial-up accounts, it was the price of an account at one of the small ISPs that proliferated in the early 1990s, which was a real expense but not a structural barrier. There were no gatekeepers between a user with an account and the rest of the internet. This was the explicit consequence of end-to-end design. The user’s host could speak any protocol it wanted to any other host. The ISP was a transit, not a platform.
These structural promises were not equally well kept by every part of the system. The pre-web internet had its own pathologies — homogeneous and overwhelmingly American, mostly male, mostly white, mostly inside academia or the defense establishment, with a culture that was open in some senses and exclusionary in others. The web’s broader social reach was a real and significant gain. But the structural openness was real, and it was the substrate the web ran on for the first half-decade of its life, and the slow loss of that structural openness across the 2000s and 2010s is the substance of the consolidation chapters in Part IV.
What the web kept and what the web did not keep
The web, considered as an application of the internet, kept some of the original peer-to-peer vision and discarded much of it. It kept the protocol-level openness: HTTP runs over TCP, TCP runs over IP, and the network does not need to know what the bits mean. It kept the standards process: HTTP was standardized through the IETF, HTML through the W3C, and both processes are descendants of the rough-consensus-and-running-code tradition. It kept the addressing model, at least at the URL layer: a URL points to a host and a path, and the host is reached through DNS, which is still federated.
What the web did not keep — and this is the slow erosion that the rest of the book is about — was the practical equality of nodes. The web is technically peer-to-peer; everybody could in principle run a web server. In practice, the asymmetry between client and server, between consumer and provider, between the people reading the web and the people publishing it, became sharper and sharper as the web grew. Consumer connections were asymmetric. Consumer machines were behind NAT. Consumer operating systems made running a server difficult. Consumer ISPs forbade running servers in their terms of service. The mass-market reality of the web was a mass-market reality of clients. The publishers were a different and smaller group, and the platforms in the middle were a still smaller group, and the structure that emerged was the opposite of the structure the internet had originally proposed.
This is not, again, a fault of the web’s designers. Berners-Lee, Andreessen, and the people building browsers were not the people building the cable modems. The structural collapse from peer to client happened in the infrastructure layer, and the web inherited the consequences. But the consequences are real. The internet was originally a federation of peers. The web on top of it became, by accident and by design, a hierarchy of clients with platforms in the middle. The recovery work in Part V — local-first, ActivityPub, the IndieWeb, the static renaissance — is in large part work to reassert at the application layer what the infrastructure has stopped offering at the transport layer.
The two halves of the frame are now in place. Hypermedia was supposed to be an extension of authorship: machines that helped people make and read structured documents, with the structure itself being the primary intellectual product. The internet was supposed to be a federation of peers: a substrate in which anyone could publish, anyone could connect, and no one had to ask permission. The web inherited both visions, kept fragments of each, and was carried by its trade-offs to a position of dominance from which the original visions are now hard to see. The next part begins with the most ambitious and least successful of the hypermedia proposals, which was also the oldest, and which has been in development almost as long as the internet itself.