An overview of the 3Box user data management system


Web3 is a broad ecosystem of technologies that together aim to minimize trust while giving users greater control and agency over their assets and interactions online. 3Box, a decentralized user data management system, plays an important role in realizing this vision of a better, more self-sovereign, user-centric web by providing developers with the ability to build applications on a user-controlled data architecture. This model of application development allows developers to separate their application from their users’ data, thereby making their infrastructure more secure and trustworthy, while limiting their data liability.

The 3Box network is a secure and decentralized system for storing and managing user-controlled data on web3. Each 3Box is a collection of distributed databases that together store a user’s public and private data from across the web, including every application and service that they use. By aggregating this information in one logical place and putting it in control of the user, we can make data interoperable and shareable across applications, wallets, accounts, blockchains, and networks.

By composing together powerful, user-centric web3 data technologies — including IPFS, OrbitDB, 3ID, and blockchain wallets — into a network, along with providing simple and easy-to-use solutions, 3Box allows developers to build better web3 applications, faster. With 3Box, developers can build and scale their application without needing to worry about user data.

Architecture Overview

There are four layers to the 3Box user data management system. We will explore each in more detail.

  1. Wallets

An overview of two users on 3Box

Please note that we are in the process of improving various parts of 3Box so some of this information is subject to change. Reach out on Discord if you find this interesting and would like to contribute.

Wallets: Key Management and Authentication

In 3Box, wallets perform the role of holding users' private keys, as well as using those keys for authenticating data access control, signing, and encryption via 3ID. More on 3ID below. 3Box allows users to bring their own existing wallet and relies on third-party wallet providers rather than providing a stand-alone wallet.

Data Authentication

3Box is blockchain-agnostic by design, but currently supports all standard Ethereum wallet providers. There are two ways wallets can authenticate databases on 3Box.

  1. EthereumProvider: The basic way that Ethereum wallets can support 3ID database authentication is by supporting the personal_sign method in the standard EthereumProvider API. When apps call openBox() or openSpace() in the 3Box API to gain access to a user's database(s), the request is transported over the Ethereum JSON-RPC interface to the wallet which displays a personal_sign request to the user. If this request is approved, the message is signed with the user's Ethereum private key and the resulting signature is returned to the application over Ethereum JSON-RPC. The combination of the user's unique private key and message is the material which is used to generate a 3ID and additional database keys, which are then stored in localStorage and used by the app to perform database operations.

  2. 3IDProvider: The more advanced way that Ethereum wallets can support 3ID database authentication is by supporting the authenticate method in the standard 3IDProvider API. When apps call openBox() or openSpace() in the 3Box API to gain access to a user's database(s), this request is transported over the 3ID JSON-RPC interface to the wallet which calls authenticate and displays a request to the user. This request can be visualized however the wallet likes, and we will likely provide a standard UI module to make this easy. If this request is approved, the message is signed with the user's Ethereum private key and the resulting signature is kept inside the wallet and is used to generate a 3ID and its additional database keys used to perform database operations. Since the keys never leave the wallet, this method of authentication offers improved key security for the user. All future database operation requests from the application will be sent to the wallet over the 3ID JSON-RPC interface via the 3Box API. In addition to improved database key security, the 3IDProvider architecture allows wallets to perform other important functions, such as linking multiple blockchain key-pair accounts to the same 3ID (great for contract wallets and for users with multiple accounts), in addition to improving the UX with autonomic signature permissions (users can permit wallets to automatically approve database requests). Systems like MetaMask Snaps, a plugin architecture for web3 wallets, are leading this effort.

Support for additional blockchains: We will soon be adding support for additional blockchain APIs and wallets via the 3ID helper library js-3id-blockchain-utils. If you have a request for supporting another network, reach out on Discord.

Additional Ethereum Resources

Here is a list of some of the most popular JavaScript EthereumProvider libraries for applications: Web3.js, Ethers.js, and Web3Connect.js.

And here is a list of some of the most popular Ethereum wallet providers for users: MetaMask, Portis, Fortmatic, Authereum, Universal Login, and WalletConnect (which supports Gnosis Safe, Argent, Rainbow, and other mobile wallets).

3ID: Data Identity, Access Control, Encryption

3ID is a decentralized identity (DID) protocol and standard providing data identity, access control, and encryption for user data stored on the decentralized web in decentralized databases, or just IPFS.

Data Identity

At the core of every 3Box account is a 3ID, a decentralized identifier (DID), which acts as a unique data identity and allows users to interoperably manage their data and information across various databases, accounts, and decentralized networks. All data associated with a user, and all messages and communications sent from a user, are controlled by their 3ID.


Creating or Authenticating a 3ID

As mentioned above, a 3ID is deterministically generated by the combination of the user's private wallet key and a 3ID consent message.

"This app requests to view and update your 3Box profile."

When the user approves this message by signing it, they will create their 3ID. Apps can request this function by calling the openBox() method available in the 3Box API. If a user does not yet have a 3ID, one will be created and instantiated for them. A link proof will also be added to the 3ID's root store so others can verify this link. If users already have a 3ID, it will simply be instantiated.

Additional public keys can later be added to the 3ID, which enables the user to authenticate their 3ID and additional databases using various private keys and associate their data across multiple wallet accounts. Wallets can link multiple key-pair and contract accounts to the same 3ID using the addAuthMethod() method in the standard 3IDProvider API, which is available in the IdentityWallet SDK.

Creating Database Keys

When the user approves a consent message in their wallet to generate a 3ID, additional database authorization keys are derived from the user's 3ID, including: a signing key (sK) to sign all database updates; and an encryption key (eK) to encrypt and decrypt data for that database.

With the EthereumProvider wallet integration, these database keys are kept in localStorage for the duration of the user's application session. This means that the user only needs to approve the consent message once, but then can perform unlimited operations on the approved database using these keys.

With the 3IDProvider wallet integration, these database keys are kept inside the wallet for a duration decided by the wallet. The wallet can choose to surface a confirmation screen for every request made by applications, or alternatively, sign all requests automatically on behalf of the user.

Resolving a 3ID

Participants wanting to lookup the databases associated with a public wallet address need to know its 3ID. Since only the owner of the public address can deterministically generate the 3ID on their own using their private key and consent message, others need to lookup this mapping from a public address to its 3ID.

3Box stores public key to 3ID mappings on the 3Box Address Server. Once a participant queries the address server to get a 3ID for a given public address, the result can then be verified in a decentralized way by observing a link proof stored publicly in the user's 3Box root store. We are in the process of decentralizing the address server and placing it in the public domain so initial lookups don't rely on any single service.

Data Access Control

3ID allows a single user to orchestrate and control many decentralized databases, each of which is separately access controlled. While traditional server-based systems have separate notions of access control and encryption, 3ID utilizes a model of Cryptographic Access Control (CAC) which uses signatures (sK) and encryption (eK) to create a system of serverless data access control. 3Box uses 3ID's CAC because data on 3Box is stored on the open IPFS network and is not kept behind a server, therefore access control must be performed in a serverless manner.

Fetching Public Data

Data stored publicly in databases on the 3Box network is kept in plaintext. Data is signed by sK but not encrypted by eK, so it is publicly available for others to consume without any access control approvals required from the user. Participants can fetch all public data from a user's root store, profile, spaces, or threads databases using any of the static 3Box get API methods, which fetch data from the 3Box REST API.

Performing Operations on Data

With 3ID, all operations on data happen client-side after the user has authenticated to a particular database. All database writes, updates, and messages must be signed by the 3ID's sK; additionally, private data must be symmetrically encrypted/decrypted by the 3ID's eK.

As mentioned previously, depending on the wallet's 3ID integration these keys may exist in localStorage in the user's browser or inside the wallet. This means that apps and services wanting to interact with a user's 3Box in additional ways beyond reading public data, such as writing data, encrypting/decrypting data, or deleting data, will need to request authentication from the user's wallet via a consent message to generate the signing key sK and encryption key eK required to perform these actions.

Accessing the Profile Database

Access control is granular for each database controlled by the user, so apps need to separately request permission for specific databases including the user's profile and each of their spaces.

Apps wanting to access a user's 3Box profile database to perform updates, decryption, or deletion of data must request authentication of the basic 3ID consent message by calling openBox() in the 3Box API. This is the same message that will create or instantiate the user's 3ID and generate derived signing and encryption keys, which means that by simply authenticating a user to your application you can read and/or update the profile database.

"This app requests to view and update your 3Box profile."

Accessing Spaces/Threads Databases

Apps wanting to access a user's Space database (or the Threads databases contained therein) to perform updates, decryption, or deletion of data must request authentication of the space's specificSpace 3ID consent message by calling openSpace() in the 3Box API.

This is different than the consent message for the profile, and upon approval will generate an entirely different 3ID, and subsequently a different signing key (sK) and encryption key (eK). This is how 3ID can access control each database separately and allow for selective disclosure of information.

"This app wants to open your <MetaMask> space."

Data Encryption

When applications want to store data privately inside a database, there is actually a series of operations that happen behind the scenes to go from the database's eK to the symmetrically encrypted key-value pair. As a reminder each database has a separate 3ID, and so is separately access controlled with a different eK. Here's how data is encrypted:

  1. When a user signs a database consent message, an sK, eK, and a salt is derived from the 3ID

  2. The key (PLAIN_KEY) and the value are stringified together and padded to have a length multiple of 24

  3. The output of step 2 is encrypted with eK using tweetnacl secretbox to get the ciphertext value

  4. Compute an obfuscated key by taking hash(PLAIN_KEY | salt)

  5. Store the key-value pair, with the key from step 4 and ciphertext value from step 3

OrbitDB: Peer to Peer Databases

3Box data is stored in various OrbitDB database instances. OrbitDB is a serverless, distributed, peer-to-peer database. 3Box uses a Javascript implementation of OrbitDB which works both in Browsers and Node.js.

OrbitDB uses IPFS as its data storage and IPFS Pubsub to automatically sync databases with peers. It's an eventually consistent database that uses CRDTs for conflict-free database merges making OrbitDB an excellent choice for decentralized apps (dApps), blockchain applications and offline-first web applications.

Database Types

OrbitDB provides various types of databases for different data models and use cases. At this time, 3Box makes use of the following types:

  • keyvalue: a key-value database just like your favorite key-value database

  • feed: a mutable log with traversable history. Entries can be added and removed. Useful for "shopping cart" type of use cases, or for example as a feed of blog posts or "tweets"

In our current implementation of 3Box, key-value stores are used for the profile and spaces databases, while feed stores are used for threads.

Data Model

All OrbitDB databases are implemented on top of ipfs-log, an immutable, operation-based conflict-free replicated data structure (CRDT) for distributed systems. CRDTs use a logical clock to allow updates from multiple peers to come to consensus on order without a strict sense of time.

IPFS: Peer to Peer File Storage

As 3Box is a system for managing user data on the decentralized web, we need a place to store and persist this data. IPFS is the underlying peer to peer file storage network used in 3Box.

About IPFS

IPFS (the InterPlanetary File System) is a content-addressed, versioned, peer to peer file system. IPFS powers the data that enables the creation of completely distributed applications. It aims to make the web faster, safer, and more open.

IPFS is a distributed file system that seeks to connect all computing devices with the same system of files. In some ways, this is similar to the original aims of the Web, but IPFS is actually more similar to a single bittorrent swarm exchanging git objects. You can read more about its origins in the paper IPFS - Content Addressed, Versioned, P2P File System.

Data Routing

Data objects in IPFS are content-addressed (indicated by a multi-hash of the contents, also known as a content identifier (CID)) instead of location-addressed (http endpoint) like data is in traditional http web architectures. If the contents of an object change, the object gets a new address.

Participants only need to ask the network for a content multi-hash and the network will perform the appropriate routing to find and return the contents, regardless of which node is persisting the information. This reduces reliance on centralized servers to host information, but rather allows for public, open distribution.

Data Persistence

IPFS has been developing a base protocol for data persistence, called Filecoin, which aims to guarantee that data added to the IPFS network remains available to users as long as some payment is received. However, Filecoin is not yet suitable for commercial use. For this reason, we operate a network of IPFS/OrbitDB pinning nodes to ensure that 3Box data remains available. Read more about 3Box pinning nodes on the next page (coming next).