The technical decisions that mattered — stateless HTTP, URLs without coordination, lossy linking, view-source
The web won. The last ten chapters have been about the alternatives it displaced; this chapter is about what made it win at the protocol level. The web is, considered as a piece of engineering, a particular set of choices made by particular people for particular reasons. The choices were not obvious at the time. Several of them were specifically rejected by competing systems. Several are still controversial in the niche literature on hypermedia. All of them were made deliberately, by Tim Berners-Lee and the small group of collaborators around him at CERN through 1989 and 1990, with subsequent refinement by the wider community as the protocol grew. This chapter treats the choices that mattered most: statelessness, uncoordinated URLs, one-way lossy linking, permissive parsing, and view-source. Each was a trade-off. Each had a cost. Each was, in the context of 1990, defensible, and several of them turned out, at the scale the web eventually reached, to be the right call. Understanding what the web chose is the first step in understanding why it won.
Where the web came from
In March 1989, Tim Berners-Lee, a software engineer at CERN — the European Organization for Nuclear Research — wrote a short proposal titled “Information Management: A Proposal” and submitted it to his manager, Mike Sendall. The proposal addressed a practical problem at CERN: the laboratory was a large and complex organization with thousands of researchers, many of them visiting for short periods, working on documentation distributed across many systems and many media. New researchers spent weeks figuring out where the information they needed lived. Established researchers had no efficient way to follow cross-references between documents on different systems. The institutional knowledge of CERN was fragmented across NeXT machines, VMS systems, Unix workstations, and paper, with no uniform way to access any of it.
Berners-Lee proposed a system in which any document on any of these systems could be referenced by a uniform identifier, requested through a uniform protocol, and rendered through a uniform format. The system would be a federated hypermedia application running on top of the internet, with no central server, no required registration, and no required coordination beyond the network already provided. Sendall, after some delay, approved the proposal with the note “Vague but exciting” written on the cover. Berners-Lee revised the proposal with Robert Cailliau in November 1990 and began implementation. By late 1990 he had a working server (running on a NeXTcube) and a working browser (running on the same NeXT). On August 6, 1991, he posted an announcement of the World Wide Web project to the alt.hypertext newsgroup on Usenet, inviting interested parties to participate. The web was, at that moment, one server, one client, a small set of documents, and Berners-Lee’s promise that the design was worth taking seriously.
The design Berners-Lee had committed to had three principal components: URLs for addressing resources, HTTP for transferring representations of resources, and HTML for marking up the representations. Each component made choices that distinguished the web from the alternatives in the previous chapters. The choices were not made in isolation; they were a coherent set, each reinforcing the others.
Statelessness
The most fundamental of the choices was that HTTP would be stateless. Each HTTP request from a client to a server stood alone. The server did not maintain memory of previous requests from the same client. The connection (in HTTP/0.9 and 1.0) was closed after each response. There was no concept of a session in the protocol; there was no concept of a user; there was no concept of a logged-in state. A request asked for a representation of a resource; the server returned the representation; the connection closed; the next request, if any, was a fresh start.
This was a deliberate departure from how most networked services worked. Telnet maintained a session. FTP maintained a session. The hypermedia systems treated in the academic-hypermedia chapter — Intermedia, KMS, NoteCards — maintained extensive per-user state. Gopher was stateless, but Gopher was an outlier; most networked services at the time assumed that the server had ongoing knowledge of what the client was doing.
The argument for statelessness was scaling. A stateful server has to remember things; the memory consumes resources; the resources limit how many clients the server can handle simultaneously. A stateless server forgets immediately; it can handle any client whose request arrives and immediately return to a clean state. The server’s capacity is bounded by request-processing throughput, not by concurrent-session count. For a system that might have to serve thousands of concurrent users on commodity hardware, statelessness was the architecture that let the hardware reach its limit.
The argument against statelessness was that it threw away features the stateful systems had been providing. Awareness of who was reading what, of where in the document graph a user had been, of which links a user had already followed — all of these required state, and the web specifically did not have it. The cost was paid in two ways. Applications that needed state (essentially all of them, eventually) had to invent it on top of the stateless protocol, using cookies, session tokens, server-side session stores, and the elaborate apparatus of modern web authentication. The application layer absorbed the complexity the protocol had refused to take on. And users lost the benefit of features (collaborative awareness, link history, position tracking) that the stateful systems had offered as a matter of course.
The trade was, in retrospect, the right one. The stateless protocol let the web grow to scales the stateful alternatives could not have reached. The added complexity at the application layer became, eventually, the platform on which sophisticated stateful applications were built. But the cost — that the application layer is where state has to live, that protocol-level features cannot count on it, and that the web’s basic substrate is, structurally, a series of disconnected transactions — has shaped everything that has happened since.
Uncoordinated URLs
The second of the deep choices was that URLs would be uncoordinated. Anyone with a host on the internet could mint URLs for resources on that host. There was no central authority that approved URL allocations. There was no registry of URLs that resolved them to canonical destinations. The system was: you put a server up on your machine, you decided what paths it served, and the URLs were http://yourmachine/yourpath. The hierarchy of authority was DNS, which already existed, and DNS was federated.
This was a rejection of what every serious hypermedia system had proposed. Xanadu had a global address space with tumblers and a registry. The academic hypermedia systems had link databases that registered links separately from documents. Gopher had a centralized hierarchy of servers indexed by Veronica. WAIS had a directory of servers that registered which databases were available where. Each of these systems assumed, in one form or another, that some authority had to know about each resource for the resource to be findable. The web’s answer was: no such authority. The internet’s name resolution (DNS) was enough; below that, each domain owner managed their own namespace; below that, there was no coordination required.
The benefit of uncoordinated URLs was that the web could grow without permission. Anyone with a host could publish; no application to a central authority was required; no coordination with other publishers was needed. The cost was that links could break for any reason at any time, that there was no system-level guarantee that a URL would continue to work, and that the citation network of the web was constructed by individual link-makers with no system-level support for keeping the citations valid.
This cost is what we call link rot. A 2010 study by Klein, Van de Sompel, and others found that approximately half of the URLs cited in academic papers became unreachable within a decade. Subsequent studies have found similar patterns across many domains. The web’s citation infrastructure is structurally fragile in a way that print’s was not, and the fragility traces directly to the choice to forgo URL coordination. The trade-off was: pay the cost of broken links to get the benefit of permissionless growth. The web has, on balance, paid the cost.
The recoveries are imperfect. The Internet Archive’s Wayback Machine, founded in 1996, has been the closest thing to a retrofitted versioning layer for the web. Persistent identifier systems — DOIs for academic publishing, ARKs for cultural heritage materials, Handles for various domains — provide URL-like identifiers with explicit promises of permanence. Each of these systems has carved out a niche where the web’s default uncoordinated URLs were too fragile to be relied on. None has reached the scale where it changes the web’s basic relationship to its own past.
Lossy one-way linking
The third deep choice was that links would be one-way and lossy. A link from page A to page B was a property of page A. Page B did not know about the link. If page A changed or was removed, the link disappeared from the system’s record without page B noticing. If page B moved or was renamed, the link in page A would silently break, with no notification to page A’s author and no automatic update of the link.
The contrast with the alternatives is sharp. Xanadu had two-way links with version permanence. Intermedia had two-way links with linkbases. NoteCards had backlinks. The academic hypermedia community had spent two decades developing the theoretical and practical apparatus for richer link models. Berners-Lee’s 1989 proposal specifically considered two-way linking and specifically rejected it, on the grounds that requiring the destination of a link to know about the link would prevent the system from being permissionless: you cannot link to my page unless I let you, if my page has to be updated whenever you link to me.
The choice was, structurally, the right one for a permissionless system. The cost was that the citation structure of the web exists only in the source documents; from the destination, you cannot see what links to you. Search engines, by crawling the web, reconstruct this information at enormous cost — much of what makes search engines work, in fact, is the inferred backlink graph that the protocol itself refused to maintain. The information is recoverable, by external means; it is not native to the system.
Smaller-scale recoveries have been deployed. Webmentions (W3C recommendation since 2017) let a citing page notify a cited page of the citation, and let the cited page display incoming mentions; the cited page has to opt in to the system, but where both sides opt in, two-way linking is recovered. Pingbacks (a precursor of webmentions, dating from 2002) provided similar functionality for the WordPress era. RSS feed enclosure links, the Linkback systems in some publishing platforms, and the various comment-thread aggregators all provide pieces of the property the protocol does not natively support. None of these are universal; all are bolt-ons.
The deeper cost, beyond the technical one, is that the web has an asymmetric model of authorship. The link is a unilateral act by the source author; the destination is a passive recipient. Citation as a social act is asymmetric on the web: the citer chooses, the cited is informed only by the analytics they install. This is not how citation works in print, where the cited work, once cited, has a verifiable existence in the citing work’s bibliography that the cited work’s author can see and respond to. The web’s one-way linking has eroded the social dimension of citation, in ways that have been felt particularly sharply in academic publishing and in the various reputation systems that depend on citation.
Permissive parsing
The fourth choice, sometimes treated as an implementation detail and sometimes as a deep architectural commitment, was that HTML would be permissively parsed. The original HTML specification was small; browsers were expected to render the document even when the markup was malformed; authors who produced bad HTML would see something on the screen, possibly not what they intended, but not nothing. The browser’s job was to make the best of whatever input it got.
This was a sharp contrast with SGML, the markup language family HTML descended from, which had a strict validation regime: documents either conformed to a Document Type Definition or did not, and non-conformant documents were not processed. SGML’s strictness was a feature for the kinds of professional publishing it had been designed for, where document validity was checked by tools and authors had time and incentive to fix errors. SGML’s strictness was a barrier for the much more casual authorship the web was about to invite.
The web’s permissive parser was, in effect, a statement that authorship would not require validation. Anyone who could write something that looked like HTML could publish. The browser would deal with the rest. This made authorship dramatically more accessible; it also made the population of HTML in the wild dramatically less clean. By the mid-1990s, much of the HTML on the web was malformed in some way, and browsers had built elaborate “tag soup” parsers — heuristics for guessing what the author had meant when the markup was ambiguous — to handle it.
The cost of tag soup was a competitive pressure. A browser that did not handle malformed HTML the way Netscape or Internet Explorer did would render existing pages incorrectly, and users would conclude that the browser was broken. New browsers entering the market had to reverse-engineer how the existing browsers handled malformed input. This was substantial work, and it became a significant moat protecting incumbent browsers. The eventual standardization of HTML5 in the late 2000s included, for the first time, a specification of how parsers should handle invalid input, codifying what had been informal practice. The HTML5 parsing algorithm is hundreds of pages of specification text dedicated to the question of what to do with input that is, by older standards, not HTML at all.
The recovery from tag soup has been XHTML, then HTML5’s strict parsing for application/xhtml+xml documents, and then the broader return to validation in many publishing workflows (static site generators, content management systems with template validation, the markup-aware editors used in modern web publishing). The web’s raw substrate remains permissive; the layers above it have, in many cases, recovered the validation discipline that the substrate refused.
View-source
The fifth choice was a feature, not a protocol decision, but it was load-bearing for the web’s culture. View-source — the menu item in early browsers that opened the page’s HTML source in a separate window — let any reader of a web page see how the page was constructed. The HTML was visible, readable, and copyable. Anyone curious about how a page achieved an effect could open the source, find the relevant markup, and learn.
The cultural effect was that the early web was a school. A generation of web authors learned HTML by reading other people’s HTML. Pages were borrowed, modified, republished. Techniques spread through view-source the way recipes had spread through cookbooks. The activation energy for becoming an author of the web was reading some pages, viewing their source, and trying things; no formal instruction was required and no special tools beyond a text editor.
The cultural effect lasted as long as the web’s source was readable. Through the late 1990s and into the 2000s, view-source continued to be a learning tool. As JavaScript grew and pages became applications, the source became less readable; modern JavaScript bundles are minified, obfuscated, and unreadable to anyone but the author. View-source still works as a menu item; what it shows, on most modern web pages, is incomprehensible. The cultural pattern that view-source enabled has weakened, although it persists for the parts of the web that still use legible HTML and CSS.
The choice to make source visible was not specifically against any of the alternatives, but it was a meaningful contrast with several. HyperCard had similar transparency: stacks could be opened and inspected. Most desktop applications did not, and most platform-mediated content does not now. The web’s transparency was a real and substantial property, particularly in the years it lasted, and its erosion is one of the quieter losses that the platform decade has produced.
The IMG tag and the inflection toward applications
A specific choice within the early web that is worth singling out is the IMG tag, added to HTML by Marc Andreessen at NCSA in 1993 and implemented in NCSA Mosaic. The IMG tag let HTML include an image inline with text, with the image fetched from a separate URL and rendered alongside the surrounding text. The tag was added to Mosaic without going through any formal standards process; it shipped, and other browsers adopted it because users wanted the feature. Berners-Lee was, at the time, opposed to the specific design Andreessen chose, preferring an extensible OBJECT-style mechanism that would have handled images as one case of a general inclusion mechanism. The OBJECT-style alternative was eventually specified, but IMG had already won the market.
The argument over IMG was, in retrospect, a smaller version of a larger pattern. The web’s design needed to handle media beyond text. Andreessen’s choice was pragmatic: a single-purpose tag for the most common case. Berners-Lee’s preference was structural: a general mechanism that handled images, video, embedded documents, and other inclusions through a unified abstraction. Andreessen’s approach shipped; Berners-Lee’s approach was not implemented for years; the web ended up with a proliferation of single-purpose tags (IMG, EMBED, OBJECT, VIDEO, AUDIO, IFRAME, CANVAS) each handling its own narrow case, with no unified inclusion mechanism. This is a recurring pattern: pragmatic single-purpose features beat general abstractions, the web ships, and the long-term cost is a tangle of overlapping mechanisms.
The deeper consequence of IMG, beyond its specific design, was that it made web pages visual. Pre-IMG pages were text-with-links; post-IMG pages were illustrated documents, which led, by stages, to formatted documents, which led to designed pages, which led to interactive applications. The IMG tag is the moment the web stopped being a hypertext system and started being a presentation medium. The application web — JavaScript, CSS, the modern browser-as-platform — is a descendant of the IMG tag’s commitment to visual presentation as a first-class concern.
The HyperCard chapter argued that what we lost was the unified read-and-write tool with a low floor for end-user authorship. The web’s evolution from hypertext to visual presentation to application is one part of why we lost it. The web could have remained a hypertext system, with images as one kind of resource alongside text, but the visual ambition that IMG enabled created the conditions for the application-platform web that followed. The choice was not made deliberately as a turn away from authorship; it was made for good reasons in 1993. The consequences for the broader trajectory were significant and largely unanticipated.
Putting the choices together
The five choices treated above — statelessness, uncoordinated URLs, lossy linking, permissive parsing, view-source — and the sixth (the visual evolution started by IMG) form a coherent design. Each reinforces the others. A stateless protocol works with uncoordinated URLs because the URLs don’t need to be remembered between requests. Uncoordinated URLs require lossy linking because two-way links would need coordination. Lossy linking implies permissive parsing because the system has to tolerate the broken links and other malformed input that uncoordinated growth produces. Permissive parsing enables view-source because the source is whatever the author wrote and the browser figures out the rest. The whole design is internally consistent.
The consistency is what made the web fast to learn, fast to deploy, and fast to grow. The web’s competitors had richer designs but were not internally consistent in the same way; each made some features powerful by requiring infrastructure the web did not require. Xanadu’s two-way links required a global registry. Intermedia’s linkbases required institutional infrastructure. KMS’s speed required dedicated hardware. The hypermedia systems were rich in particular dimensions and demanding in correspondingly particular ways. The web was thin in every dimension and demanding in none. The thinness was the price for the scaling.
The chapters in Part V take, in different ways, the position that some of these trade-offs need to be revisited. Local-first wants to recover the per-client persistence the web threw away with statelessness. Personal knowledge tools want to recover the two-way linking the web threw away with one-way links. The static renaissance wants to recover the document focus the web threw away in its turn toward applications. Each recovery is a way of negotiating with the trade-offs the protocol chose. None is a wholesale rejection of the web’s design.
Roy Fielding and REST
A useful retrospective frame for the web’s design came in 2000, when Roy Fielding completed his PhD dissertation at UC Irvine. Fielding had been one of the principal authors of HTTP/1.1 and was, by 2000, a thoughtful theorist of the web’s architecture. His dissertation, “Architectural Styles and the Design of Network-based Software Architectures,” named the web’s underlying style REST — Representational State Transfer — and articulated the constraints that the style imposed on architectures that followed it. The constraints were six: client-server separation, statelessness, cacheability, uniform interface, layered system, and (optional) code-on-demand.
Fielding’s framing has been influential beyond its original purpose. The REST acronym has been adopted, often imprecisely, by an entire industry of web API designers who use it to mean “we have HTTP endpoints with reasonable URLs.” The original meaning was more specific: a REST architecture is one in which the server exposes resources at addresses, the client navigates between resources through hypermedia links embedded in the representations, and the state of the application moves from the server to the client as the client follows links. This is, in essence, the structure of the web of documents: you fetch a page, the page contains links, you click a link, you fetch the next page. Fielding was articulating, in 2000, the design that the web had implemented in 1991, and showing why the design’s constraints had been load-bearing for its scaling.
The REST constraints were not arbitrary. They corresponded to the choices treated in this chapter: client-server separation made the protocol pluggable; statelessness made it scalable; uniform interface made it interoperable; layered system made it evolvable; code-on-demand let the protocol grow without breaking existing clients. The web’s success had been a demonstration that an architecture with these constraints could grow; Fielding’s dissertation was the analytical articulation of why.
REST as an architectural style has been important for the design of web APIs and for the broader thinking about how internet services should be structured. It has also been routinely misunderstood. Most “REST APIs” in practice violate Fielding’s constraints in significant ways, particularly the hypermedia-as-the-engine-of-application-state constraint. The web’s actual practice has drifted from the architectural style its underlying protocol embodied. The drift is one of the things that Part V’s chapters track: the web increasingly is not architecturally a hypermedia system; it is increasingly a thin client to remote applications.
The costs paid
The chapter has been about the technical decisions that gave the web its shape. Each decision was a trade-off; each trade-off has had downstream costs. The summary is worth being explicit about.
The cost of statelessness has been the application-layer scaffolding that recovers state where applications need it. Cookies, sessions, OAuth tokens, the complex authentication and authorization machinery of modern web applications — all of it exists because the protocol does not provide what applications need. The cost has been complexity at the application layer, and the cost has been distributed across millions of applications, each solving the same problem incompletely.
The cost of uncoordinated URLs has been link rot. Half the citations in academic papers break within a decade. The web’s relationship to its own past is, structurally, fragile, and the recovery (the Internet Archive, persistent identifier systems, web archiving projects) has been retrofit and incomplete.
The cost of lossy linking has been the citation network being invisible from the cited side. The deep structure of the web is reconstructable only by external means, and the institutions that reconstruct it — search engines, primarily — have become structurally central in a way the protocol did not require.
The cost of permissive parsing has been browser implementations being enormous and difficult to write, a structural barrier to new entrants in the browser market, and a stream of subtle compatibility issues that have absorbed engineering attention for thirty years.
The cost of view-source’s effective erosion — as pages have become applications and source has become minified bundles — has been the loss of the teaching tradition that view-source had supported. The activation energy for becoming a web author is now higher than it was in 1996.
These are real costs. The web is the right system for the world to have built in 1991, given what the world wanted networked computing to do. The trade-offs the web chose were defensible at the time and have, on balance, been right at scale. The recovery work in Part V is not work to replace the web; it is work to recover, on top of the web’s substrate, properties that the substrate did not provide. The web is not going away. The question for the recovery is which of the original alternatives’ features can be reassembled into the web’s world and which cannot.
The next chapter takes the moment the web became commercial. The technical decisions Berners-Lee and his early collaborators made were one thing; the commercial inflection that came with Mosaic and Netscape was another, and the two together produced the trajectory of the next thirty years. Andreessen and the people around him are the protagonists of the next chapter; Berners-Lee and the people around him recede into the standards-body history that comes after.