Content Addressing

Content addressing is the practice of identifying data by what it is (its cryptographic hash) rather than where it lives (a server URL). This single shift enables offline verification, natural deduplication, and immutable references—properties that are foundational to P2P systems.

Why Content Addressing Matters for P2P

In a centralized system, you trust the server to give you the right data. In P2P, any peer might serve you data—including a malicious one. Content addressing solves this: if you know the hash of what you want, you can accept it from anyone and verify correctness locally. This property is what makes trustless storage and distribution possible.

Hash-Linked Structures

Merkle trees and DAGs are the core data structure. Each node's identifier is the hash of its contents (including child hashes), creating a recursive integrity guarantee:

Any tampering invalidates the root hash—a single bit flip propagates upward.

Subtrees can be verified independently (partial verification), so you don't need the whole tree to check a piece of it.

Identical subtrees are automatically deduplicated across different data sets.

Merkle DAGs generalize trees by allowing nodes to reference multiple parents, enabling richer structures:

Git commit graphs: each commit references its parents and a tree of files.

IPFS UnixFS: files and directories as DAG nodes, enabling deduplication across files.

Hypercore append-only logs: a Merkle tree over sequential entries, enabling efficient range proofs.

Matrix event DAGs: room history as a hash-linked DAG of events.

BitTorrent piece hashes: per-piece SHA-1 (v1) or per-file Merkle trees (v2).

Content Identifiers (CIDs)

A CID is a self-describing content address combining:

Multihash: the hash algorithm identifier + digest (e.g., SHA-256, BLAKE2b-256, BLAKE3). Using algorithm-agile hashing means the system can migrate to stronger hashes without breaking the addressing scheme.

Multicodec: the encoding/format of the data (e.g., dag-cbor, dag-pb, dag-json, raw bytes). This tells consumers how to interpret the bytes.

Version: CIDv0 (legacy, bare SHA-256 multihash) or CIDv1 (with explicit codec, version, and multibase encoding).

CIDs enable a universal content namespace: any peer holding data matching a CID can serve it, and the requester can verify correctness without trusting the source. CIDs are the lingua franca of IPFS and Filecoin, used by libp2p for block exchange, and the CID concept (hash-based self-describing identifiers) appears in various forms across most P2P systems.

Chunking Strategies

Large files must be split into blocks before hashing. The choice of chunking strategy affects deduplication, diff efficiency, and transfer performance:

Fixed-size chunking (e.g., 256 KB blocks): simple and fast, but insertions at the beginning shift all subsequent boundaries, destroying deduplication.

Content-defined chunking (e.g., Rabin fingerprinting, Buzzhash): block boundaries are determined by content patterns (e.g., when a rolling hash hits a specific value). Insertions only affect nearby chunks, producing stable, diff-friendly splits. This is what makes Syncthing block exchange and IPFS file updates efficient.

Manifests and archives: CAR (Content Addressable aRchive) files bundle multiple blocks for efficient transport and storage. Useful for bulk import/export and backup.

Mutable Pointers over Immutable Data

Content addresses are immutable by definition—changing the data changes the hash. Mutability is layered on top through signed pointers:

IPNS (InterPlanetary Name System): a mutable pointer keyed by a public key that resolves to a CID, updated via signed records in the DHT. Anyone who knows the public key can resolve the latest CID; only the private key holder can update it.

Signed heads: append-only logs where the latest entry is signed by the author's identity key. Used by Hypercore and SSB. The log itself is content-addressed (Merkle tree); the head pointer is authenticated.

DNSLink: map a DNS TXT record to a CID, using the existing DNS infrastructure for human-readable names over IPFS content.

Pinning: explicitly marking content for durable retention against garbage collection. Pinning doesn't create mutability—it ensures availability.

This immutable-data-plus-mutable-pointer pattern is one of the most important design patterns in P2P, enabling aggressive caching, CDN-like behavior, and trustless mirroring.

Content Addressing and Verification

Content addressing is the foundation of defense against content poisoning: if a peer serves you corrupted data, the hash won't match and you reject it. Combined with Merkle proofs, you can verify individual blocks without downloading the entire dataset. This is what makes trustless multi-peer swarming safe.

How Content Addressing Connects to Everything Else

Content addressing depends on Overlay Networks and DHTs for provider discovery ("who has this CID?"), Transport and Connectivity for fetching blocks, and Identity and Capabilities for signing mutable pointers. Storage and Distribution governs how content-addressed blocks are replicated and cached. State Sync and CRDTs builds mutable collaborative structures on top of content-addressed building blocks. Privacy is complicated by content addressing: requesting a specific CID reveals what you're looking for. Compliance must address the tension between immutable data and the right to erasure.