Dolos

Dolos is a new type of Cardano node, fine-tuned to solve a very narrow scope: keeping an updated copy of the ledger and replying to queries from trusted clients, while requiring a small fraction of the resources

Motivation

Cardano nodes can assume one of two roles:

• block producer: in charge of minting blocks
• relay node: in charge relaying blocks from / to peers.

Each of these roles has concrete responsibilities and runtime requirements. Criteria such as network topology, resource allocation, backup procedures, etc vary by role.

We argue that there’s a 3rd role that should be treated independently with the goal of optimizing its workload: nodes that are used with the sole purpose of resolving local state queries or serving as data source for downstream tools that require ledger data.

There are many potential optimizations for nodes performing this type of workload that are not currently possible with the Cardano node:

• drastically limiting the amount of memory required to execute the node
• switching to storage solutions with different trade-offs (eg: S3, NFS, etc)
• providing alternative wire protocols more friendly for data queries (eg: REST, gRPC)
• providing an auth layer in front of the API endpoints

The goal of this project is to provide a very limited and focused version of the Cardano node that can be used by DevOps as a cost-effective, performant option to deploy data nodes side-by-side with the producer / relay nodes.

This new role would be useful in the following scenarios:

• As data source for well-known tools such as DB-sync, Ogmios, CARP, Oura, etc.
• As a fast, low resource node for syncing other producer / relay nodes.
• As a ledger data source that scales dynamically according to query load.
• As a node that leverages network / cloud storage technology instead of mounted drives.
• As a node that scales horizontally, allowing high-availability topologies.
• As a low resource local node for resolving local state queries.

Detailed design

Data nodes will share some of the features with the mainstream Cardano node:

• Node-to-Node and Node-to-Client Chain-Sync mini-protocol
• Node-to-Node Block-Fetch mini-protocol
• Node-to-Client Local-State-Query mini-protocol

This new type of node will also provide features not currently available in the mainstream Cardano node:

• HTTP/JSON endpoint for common local state queries
• gRPC endpoint for local state queries and chain-sync procedure
• Different storage options including NFS, S3 & GCP Buckets
• Low memory consumption (allowed by the trade-offs in scope)

Drawbacks

• Although the scope is very narrow compared to a real, full-blown node, this tool has a large LoE.
• There's overlap with some TxPipe tools such as Oura and Scrolls. The mitigation plan is to hoist individual components into Pallas to achieve DRY.
• Some components, such as the gRPC interface, might be useful even in environment running the full-blown Cardano node. To mitigate this we will architect the system in such a way that different entry-points (aka: binaries) can perform different roles. The gRPC bridge would be one of this.

Alternatives

• Use the full-blown Cardano node even for the scenarios described in this RFC.
• Split the project into sub-components that can be orchestrated to achieve the same result.

Unresolved questions

• Performance gains and resource allocation optimizations are theoretical, these were extrapolated from our experience implementing Cardano data processing pipelines using components written in Rust. We won’t have a strict, quantifiable measurement until we develop a PoC of this project. To mitigate this issue, our development process will include performance benchmarks execution at each development milestone. Reports will be included as part of each release.
• There’s some documentation lacking regarding local state queries wire-format which will need some reverse engineering from the mainstream Cardano node. We have experience with this approach but the level-of-effort associated with the task is hard to anticipate. To try mitigate this issue, we'll reach out to IOG for advise and documentation in case it's available.

Installation

Depending on your needs, Dolos provides different installation options:

• Binary Release: to use one of our pre-compiled binary releases for the supported platforms.
• From Source: to compile a binary from source code using Rust's toolchain
• Docker: to run the tool from a pre-built docker image
• Kubernetes: to deploy Dolos as a resource within a Kubernetes cluster

Binary Releases

Dolos can be run as a standalone executable. The Github release page includes the binaries for different OS and architectures. It's a self-contained, single-file binary that can be downloaded directly.

For simplicity, we also provide an install script that can be executed from your terminal to automate the installation process. This install script will only work for MacOS and Linux for either x86_64 and arm64 architecture. You can execute the install by running the following command from your terminal. You'll need curl and sudo access.

curl -fsSL https://github.com/txpipe/dolos/raw/main/.github/public/install.sh -o get-dolos.sh
sudo sh ./get-dolos.sh

From Source

The following instructions show how to build and install Dolos from source code.

Pre-requisites

• Rust toolchain

Procedure

git clone git@github.com:txpipe/dolos.git
cd dolos
cargo install --all-features --path .

Docker

Dolos provides already built public Docker images through Github Packages. To execute Dolos via Docker, use the following command:

docker run ghcr.io/txpipe/dolos:latest

The result of the above command should show Dolos' command-line help message.

Entry Point

The entry-point of the image points to Dolos executable. You can pass the same command-line arguments that you would pass to the binary release running bare-metal. For example:

docker run -it ghcr.io/txpipe/dolos:latest \
    --config my-custom-config.toml

For more information on available command-line arguments, check the usage section.

Using a Configuration File

The default daemon configuration file for Dolos is located in /etc/dolos/daemon.toml. To run Dolos in daemon mode with a custom configuration file, you need to mount it in the correct location. The following example runs a docker container in background using a configuration file named daemon.toml located in the current folder:

docker run -d -v $(pwd)/daemon.toml:/etc/dolos/daemon.toml \
    ghcr.io/txpipe/dolos:latest daemon

Versioned Images

Images are also tagged with the corresponding version number. It is highly recommended to use a fixed image version in production environments to avoid the effects of new features being included in each release (please remember dolos hasn't reached v1 stability guarantees).

To use a versioned image, replace the latest tag by the desired version with the v prefix. For example, to use version 0.6.0, use the following image:

ghcr.io/txpipe/dolos:v0.6.0

Multiple Architectures

Dolos docker image is multi-arch, meaning that it can be used from different CPU architectures. We currently support amd64 (aka x86_64) and arm64.

The Docker daemon will detect your architecture and use the correct manifest to run the image. The usage procedures are the same regardless of the architecture.

Kubernetes

Dolos can be implemented as a standalone Kubernetes StatefulSet resource.

Please note that the amount of replicas is set to 1. Dolos doesn't have any kind of "coordination" between instances. Adding more than one replica will just create extra pipelines duplicating the same work.

apiVersion: v1
kind: ConfigMap
metadata:
  name: dolos
data:
  daemon.toml: |-
    [upstream]
    # REDACTED: here goes your `upstream` configuration options

    [rolldb]
    # REDACTED: here goes your `rolldb` configuration options
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dolos
spec:
  template:
    spec:
      containers:
        - name: main
          image: ghcr.io/txpipe/dolos:latest

          # we mount the same volume that the main container uses as the source
          # for the Cardano node unix socket.
          volumeMounts:
            - mountPath: /var/dolos/db
              name: db
            - mountPath: /etc/dolos
              name: config
          resources:
            requests:
              memory: 1Gi
              cpu: 1
            limits:
              memory: 1Gi
      volumes:
        # an empty-dir to store your data. In a real scenario, this should be a PVC
        - name: db
          emptyDir: {}

        # a config map resource with Dolos' config, particular for your requirements
        - name: config
          configMap:
            name: config

Usage

This section provides information on how to configure Dolos.

Upstream

[upstream]
peer_address = "preview-node.world.dev.cardano.org:30002"
network_magic = 2

RollDB

[rolldb]
path = "./data"
k_param = 1000

Serve

[serve.grpc]
listen_address = "[::]:50051"

Byron

[byron]
path = "./byron.json"

Logging

[logging]
max_level = "debug"

Example Configuration

Preview Network

[upstream]
peer_address = "preview-node.world.dev.cardano.org:30002"
network_magic = 2

[rolldb]
path = "./data"
k_param = 1000

[serve.grpc]
listen_address = "[::]:50051"

[byron]
path = "./byron.json"

[logging]
max_level = "debug"

Usage

This section provides information on how to use Dolos.

• There are 2 ways to start the server that provides access to the endpoints: the serve command and the daemon command

Sync

serve mode

The serve command starts Dolos just with the purpose of exposing the client endpoint, no other functions are executed

To start Dolos in serve mode run the following command from the terminal:

dolos serve

daemon mode

The daemon command starts Dolos with all of its main functions enabled (syncing from upstream, updating the ledger, etc) which includes the client endpoint server

To start Dolos in daemon mode run the following command from the terminal:

dolos daemon

eval command

The eval is an utility for evaluating transactions. It takes tx data from an external source and uses the current ledger state to evaluate phase-1 validation rules.

Usage

To execute the evaluation, run the following command from your terminal:

dolos eval --file <FILE> --era <ERA> --magic <MAGIC> --slot <SLOT>

The args should be interpreted as:

• --file <FILE>: the path to the file containing the tx data as hex-encoded cbor.
• --era <ERA>: the id of the era that should be used to interpret the transaction data.
• --magic <MAGIC>: the protocol magic of the network.
• --slot <SLOT>: the slot that should be used for retrieving protocol parameters.

API

gRPC

Introduction

• Dolos exposes a gRPC endpoint allowing clients to query data
• The endpoint adheres to the UTxO RPC interface definition found in https://utxorpc.org
• The current implemented module is the Sync module that allows clients to sync with the state of the chain stored by Dolos
• Dolos endpoint also supports gRPC-web, a variant of gRPC that can be used directly from browsers

Connecting to the server

• Once started, the server exposes TCP port
• The default port number is 50051, but can be changed via configuration
• This port accepts http/2 connections following the standard gRPC mechanism
• The port also accepts http/1.1 connections following the gRPC-web protocol
• Developers can make use of UTxO-RPC SDK libraries to interact with the endpoint programmatically

Authentication Mechanism

• Dolos has a built-in mechanism for authenticating clients using TLS
• By specifying a specific CA authority, Dolos can allows clients that provide a matching certificate
• The CA authority is specified by pointing to the corresponding .pem file through configuration

Ouroboros

Benchmarks

Long-running operation - June test

A "long-running" test refers to a type of test that is designed to evaluate the system's behavior under sustained use over a long period of time. These tests can be used to identify problems that might not be apparent in short-term testing scenarios.

Among other things, long-running tests are often used to identify memory leaks, resource leaks, or degradation in system performance over time. If a system runs perfectly for an hour but starts having issues after several hours or days, a long-running test would help identify such problems.

In particular, we want to measure CPU and memory usages of Dolos compared to the Haskell node. The main goal of Dolos is to provide a lightweight alternative (with substantially reduced features) for data access use-cases. To perform the comparison, both implementations were hosted under the equivalent conditions:

• same hardware
• both fully-synced to the Preview network
• similar client request profile (1 chain-sync consumer)

CPU Usage

In this analysis we compare CPU usage. It is expressed as shares of a vCPU (~core). 1 share represents 1/1000 of a vCPU. Each bucket represent the average of shares utilized by each process in a 30-minute period. The information was gathered after continuous operations throughout a 2-day period.

Memory Usage

In this analysis we compare memory usage. It is expressed as total amount of data (KB, MB, GB). Each bucket represent the average of memory utilized by each process in a 30-minute period. The information was gathered after continuous operations throughout a 2-day period.

Long-running operation - October test

A "long-running" test refers to a type of test that is designed to evaluate the system's behavior under sustained use over a long period of time. These tests can be used to identify problems that might not be apparent in short-term testing scenarios.

Among other things, long-running tests are often used to identify memory leaks, resource leaks, or degradation in system performance over time. If a system runs perfectly for an hour but starts having issues after several hours or days, a long-running test would help identify such problems.

In particular, we want to measure CPU and memory usages of Dolos compared to the Haskell node. The main goal of Dolos is to provide a lightweight alternative (with substantially reduced features) for data access use-cases. To perform the comparison, both implementations were hosted under the equivalent conditions:

• same hardware
• both fully-synced to the Preview network
• similar client request profile (1 chain-sync consumer)

CPU Usage

In this analysis we compare CPU usage. It is expressed as shares of a vCPU (~core). 1 share represents 1/1000 of a vCPU. Each bucket represent the average of shares utilized by each process in a 15-minute period. The information was gathered after continuous operations throughout a 24 hr period.

Memory Usage

In this analysis we compare memory usage. It is expressed as total amount of data (KB, MB, GB). Each bucket represent the average of memory utilized by each process in a 15-minute period. The information was gathered after continuous operations throughout a 24 hr period.

Byron Phase-1 Sync - Resource Footprint Test

A "resource footprint" test refers to a type of test designed to evaluate the amount of resources, usually memory and CPU, that a software component requires to perform a particular action.

Among other things, these tests are often used to ensure that theoretical estimations of resource consumption match reality. These test are also useful as reference profiles that allow system operators to anticipate resource requirements.

The goal of this particular test instance is to understand the impact of Byron Phase-1 validations on Dolos resource footprint.

The data presented was gathered during a chain sync process against a single upstream Haskell node while executing Byron Phase-1 validations. The Dolos version used for this test matches the code in branch feat/byron-phase-1-validations

CPU Usage

In this analysis we track CPU usage. It is expressed as shares of a vCPU (~core). 1 share represents 1/1000 of a vCPU. Each bucket represent the average of shares utilized by each process in a 15-minute period. The information was gathered after continuous operations throughout a 1-day period.

Memory Usage

In this analysis we track memory usage. It is expressed as total amount of data (KB, MB, GB). Each bucket represent the average of memory utilized by each process in a 15-minute period. The information was gathered after continuous operations throughout a 1-day period.