Operating rippled Servers

The core server of the XRP Ledger peer-to-peer network is rippled. Anyone can run their own rippled server that follows the network and keeps a complete copy of the XRP Ledger. You can even have your server take part in the consensus process.

This page contains instructions for:

Types of rippled Servers

The rippled server software can run in several modes depending on its configuration, including:

  • Stock server - follows the network with a local copy of the ledger.
  • Validating server, or validator for short - participates in consensus.
  • rippled server in stand-alone mode - for testing. Does not communicate to other rippled servers.

You can also run the rippled executable as a client application for accessing rippled APIs locally. (Two instances of the same binary can run side-by-side in this case; one as a server, and the other running briefly as a client and then terminating.)

Reasons to Run a Stock Server

There are lots of reasons you might want to run your own rippled server, but most of them can be summarized as: you can trust your own server, you have control over its workload, and you're not at the mercy of others to decide when and how you can access it. Of course, you must practice good network security to protect your server from malicious hackers.

You need to trust the rippled you use. If you connect to a malicious server, there are many ways that it can take advantage of you or cause you to lose money. For example:

  • A malicious server could report that you were paid when no such payment was made.
  • It could selectively show or hide payment paths and currency exchange offers to guarantee its own profit while not providing you the best deal.
  • If you sent it your address's secret key, it could make arbitrary transactions on your behalf, and even transfer or destroy all the money your address holds.

Additionally, running your own server gives you admin control over it, which allows you to run important admin-only and load-intensive commands. If you use a shared server, you have to worry about other users of the same server competing with you for the server's computing power. Many of the commands in the WebSocket API can put a lot of strain on the server, so rippled has the option to scale back its responses when it needs to. If you share a server with others, you may not always get the best results possible.

Finally, if you run a validating server, you can use a stock server as a proxy to the public network while keeping your validating server on a private subnet only accessible to the outside world through the stock server. This makes it more difficult to compromise the integrity of your validating server.

Reasons to Run a Validator

The robustness of the XRP Ledger depends on an interconnected web of validators who each trust a few other validators not to collude. The more operators with different interests there are who run validators, the more certain each member of the network can be that it continues to run impartially. If you or your organization relies on the XRP Ledger, it is in your interest to contribute to the consensus process.

Not all rippled servers need to be validators: trusting more servers from the same operator does not offer better protection against collusion. An organization might run validators in multiple regions for redundancy in case of natural disasters and other emergencies.

If your organization runs a validating server, you may also run one or more stock servers, to balance the computing load of API access, or as a proxy between your validation server and the outside network.

Properties of a Good Validator

There are several properties that define a good validator. The more of these properties your server embodies, the more reason others have to include your server in their list of trusted validators:

  • Availability. An ideal validator should always be running, submitting validation votes for every proposed ledger.
    • Strive for 100% uptime.
  • Agreement. A validator's votes should match the outcome of the consensus process as often as possible. To do otherwise could indicate that the validator's software is outdated, buggy, or intentionally biased.
    • Always run the latest rippled release without modifications.
  • Timeliness. A validator's votes should arrive quickly, and not after a consensus round has already finished.
    • A fast internet connection helps with this.
  • Identified. It should be clear who runs the validator. Ideally, a list of trusted validators should include validators operated by different owners in multiple legal jurisdictions and geographic areas, to reduce the chance that any localized events could interfere with the validator's impartial operation.

At present, Ripple (the company) cannot recommend any validators aside from those in the default validator list. However, we are collecting data on other validators and building tools to report on their performance. For metrics on validators, see validators.ripple.com.

Capacity Planning

This section describes some of the challenges related to scaling rippled servers for testing and production deployments. Additionally, it describes how hardware and configuration settings relate to those challenges. Finally, this section offers recommendations to assist you in properly setting up rippled to meet the goals for the use case of your deployment.

Even the most minimally functional rippled server must contain the most recently validated ledger versions to submit transactions to the network and verify the integrity of the XRP Ledger. Beyond these requirements, consider the following possible business needs:

  • Handling ever-increasing transaction volume
  • Servicing transaction reporting information to clients
  • Maintaining varying amounts of historical data

To meet your rippled capacity requirements, you must address these technical factors:

Configuration Settings

Ripple recommends the following guidelines to improve performance. You can set the following parameters in the rippled.cfg file to improve performance for your rippled server.

Node Size

The node_size parameter determines the size of database caches. Larger database caches decrease disk I/O requirements at a cost of higher memory requirements. Ripple recommends you always use the largest database cache your available memory can support. See the following table for recommended settings.

Recommendation

Available RAM for rippled node_size value Notes
< 8GB tiny Not recommended
8GB low
16GB medium
32GB huge Recommended for production servers

Node DB Type

The type field in the node_db section of the rippled.cfg file sets the type of key-value store that rippled uses to persist the XRP Ledger in the ledger store. You can set the value to either rocksdb or nudb.

rippled offers a history sharding feature that allows you to store a randomized range of ledgers in a separate shard store. You may want to configure the shard store to use a different type of key-value store than the ledger store. For more information about how to use this feature, see History Sharding.

RocksDB vs NuDB

RocksDB requires approximately one-third less disk storage than NuDB and provides a corresponding improvement in I/O latency. However, this comes at a cost of increased memory utilization as storage size grows. NuDB, on the other hand, has nearly constant performance and memory footprint regardless of storage.

rippled servers that operate as validators should keep only a few days' worth of data or less. Ripple recommends using RocksDB for validators. For all other uses, Ripple recommends using NuDB for the ledger store.

RocksDB has performance-related configuration options you can modify to achieve maximum transaction processing throughput. (NuDB does not have performance-related configuration options.) Here is an example of the recommended configuration for a rippled server using RocksDB:

[node_db]
type=rocksdb
open_files=512
file_size_mb=64
file_size_mult=2
filter_bits=12
cache_mb=512
path={path_to_ledger_store}

Historical Data

The amount of historical data that a rippled server keeps online is a major contributor to required storage space. At the time of writing (2018-03-01), a rippled server stores about 12GB of data per day. You can expect this amount to grow as transaction volume increases across the XRP Ledger network. You can control how much data you keep with the online_delete and advisory_delete fields.

Online deletion enables pruning of rippled ledgers from databases without any disruption of service. It only removes records that are not part of the current ledgers. Without online deletion, those databases grow without bounds. Freeing disk space requires stopping the process and manually removing database files.

Log Level

The default rippled.cfg file sets the logging verbosity to warning. This setting greatly reduces disk space and I/O requirements over more verbose logging. However, more verbose logging provides increased visibility for troubleshooting.

Caution: If you omit the log_level command from the [rpc_startup] stanza, rippled writes logs to disk at the debug level and outputs warning level logs to the console. debug level logging requires several more GB of disk space per day than warning level, depending on transaction volumes and client activity.

Network and Hardware

Each rippled server in the XRP Ledger network performs all of the transaction processing work of the network. It is unknown when volumes will approach maximum network capacity. Therefore, the baseline hardware for production rippled servers should be similar to that used in Ripple's performance testing.

Recommendation

For best performance in enterprise production environments, Ripple recommends running rippled on bare metal with the following characteristics:

  • Operating System: Ubuntu 16.04+
  • CPU: Intel Xeon 3+ GHz processor with 4 cores and hyperthreading enabled
  • Disk: SSD
  • RAM:
    • For testing: 8GB+
    • For production: 32GB
  • Network: Enterprise data center network with a gigabit network interface on the host

SSD Storage

SSD storage should support several thousand of both read and write IOPS. The maximum reads and writes per second that Ripple engineers have observed are over 10,000 reads per second (in heavily-used public server clusters), and over 7,000 writes per second (in dedicated performance testing).

CPU Utilization and Virtualization

Ripple performance engineering has determined that bare metal servers achieve maximum throughput. However, it is likely that hypervisors cause minimal degradation in performance.

Network

Any enterprise or carrier-class data center should have substantial network bandwidth to support running rippled servers. The minimum requirements are roughly 2Mbps transmit and 2Mbps receive for current transaction volumes. However, these can burst up to 100MBps transmissions when serving historical ledger and transaction reports. When a rippled server initially starts up, it can burst to over 20Mbps receive.

Storage

Ripple recommends estimating storage sizing at roughly 12GB per day of data kept online with NuDB. RocksDB requires around 8GB per day. However, the data per day changes with activity in the network. You should provision extra capacity to prepare for future growth. At the time of writing (2018-03-01), a server with all XRP Ledger history requires 6.8TB.

Memory

Memory requirements are mainly a function of the node_size configuration setting and the amount of client traffic retrieving historical data. As mentioned, production servers should maximize performance and set this parameter to huge. You can set the node_size parameter lower to use less memory, but you should only do this for testing. With a node_size of medium, a rippled server can be reasonably stable in a test Linux system with as little as 8GB of RAM.

Amazon Web Servives

Amazon Web Services (AWS) is a popular virtualized hosting environment. You can run rippled in AWS, but Ripple does not recommend using Elastic Block Storage (EBS). Elastic Block Storage's maximum number of IOPS (5,000) is insufficient for rippled's heaviest loads, despite being very expensive.

AWS instance stores (ephemeral storage) do not have these constraints. Therefore, Ripple recommends deploying rippled servers with host types such as M3 that have instance storage. The database_path and node_db path should each reside on instance storage.

Caution: AWS instance storage is not guaranteed to provide durability in the event of hard drive failure. Further, data that is lost when the instance stops and restarts (but not when just rebooted). This loss can be acceptable for a rippled server because an individual server can usually re-acquire that data from its peer servers.

Installing rippled

For development, you can compile rippled from source.

Production rippled instances can use Ripple's binary executable, available from the Ripple yum repository.

Minimum System Requirements

A rippled server should run comfortably on commodity hardware, to make it inexpensive to participate in the network. At present, we recommend the following mimimum requirements:

  • Operating System:
    • Production: CentOS or RedHat Enterprise Linux (latest release) or Ubuntu (16.04+) supported
    • Development: Mac OS X, Windows (64-bit), or most Linux distributions
  • CPU: 64-bit x86_64, 2+ cores
  • Disk: Minimum 50GB SSD recommended (1000 IOPS, more is better) for the database partition
  • RAM:
    • Testing: 8GB+
    • Production: 32 GB

Amazon EC2's m3.large VM size may be appropriate depending on your workload. A fast network connection is preferable. Any increase in a server's client-handling load increases resources needs.

Tip: For recommendations beyond the minimum requirements, see Capacity Planning.

Installation on CentOS/Red Hat with yum

This section assumes that you are using CentOS 7 or Red Hat Enterprise Linux 7.

  1. Install the Ripple RPM repository:

    $ sudo rpm -Uvh https://mirrors.ripple.com/ripple-repo-el7.rpm
    
  2. Install the rippled software package:

    $ sudo yum install --enablerepo=ripple-stable rippled
    
  3. Configure the rippled service to start on system boot:

    $ sudo systemctl enable rippled.service
    
  4. Start the rippled service

    $ sudo systemctl start rippled.service
    

Installation on Ubuntu with alien

This section assumes that you are using Ubuntu 15.04 or later.

  1. Install yum-utils and alien:

    $ sudo apt-get update
    $ sudo apt-get install yum-utils alien
    
  2. Install the Ripple RPM repository:

    $ sudo rpm -Uvh https://mirrors.ripple.com/ripple-repo-el7.rpm
    
  3. Download the rippled software package:

    $ yumdownloader --enablerepo=ripple-stable --releasever=el7 rippled
    
  4. Verify the signature on the rippled software package:

    $ sudo rpm --import https://mirrors.ripple.com/rpm/RPM-GPG-KEY-ripple-release && rpm -K rippled*.rpm
    
  5. Install the rippled software package:

    $ sudo alien -i --scripts rippled*.rpm && rm rippled*.rpm
    
  6. Configure the rippled service to start on system boot:

    $ sudo systemctl enable rippled.service
    
  7. Start the rippled service

    $ sudo systemctl start rippled.service
    

Postinstall

It can take several minutes for rippled to sync with the rest of the network, during which time it outputs warnings about missing ledgers. After that, you have a fully functional stock rippled server that you can use for local signing and API access to the XRP Ledger.

rippled commands can be run with:

    $ /opt/ripple/bin/rippled <command>

Updating rippled

You can subscribe to the rippled Google Group to receive notifications of new rippled releases.

Automatic Update on CentOS/Red Hat

Automatic rippled updates can be enabled with a one-time Cron configuration:

  1. Check that /opt/ripple/bin/update-rippled.sh exists. If it does not, update manually.

  2. Install crond:

    $ sudo yum install cronie
    
  3. Open the crontab file for editing

    $ sudo crontab -e
    
  4. Add the following to the crontab file. Be sure to add a blank line at the end of the file.

    RANDOM_DELAY=59
    0 * * * * /opt/ripple/bin/update-rippled.sh
    

The script updates the installed rippled package within an hour of each new release.

Manual Update on CentOS/Red Hat

Run the following commands to update to the latest release of rippled:

    $ sudo rpm -Uvh --replacepkgs https://mirrors.ripple.com/ripple-repo-el7.rpm
    $ sudo yum update --enablerepo=ripple-stable rippled
    $ sudo systemctl daemon-reload
    $ sudo service rippled restart

Manual Update on Ubuntu

Run the following commands to update to the latest release of rippled:

    $ sudo rpm -Uvh --replacepkgs https://mirrors.ripple.com/ripple-repo-el7.rpm
    $ yumdownloader --enablerepo=ripple-stable --releasever=el7 rippled
    $ rpm -K rippled*.rpm
    $ sudo alien -i --scripts rippled*.rpm
    $ sudo systemctl daemon-reload
    $ sudo service rippled restart

Running a Validator

Running a rippled validator that participates in the Consensus process is simple:

  1. Enable validation on your rippled server.
    • At first, your server is an untrusted validator. Others can see the validations your server issues, but they disregard them in the consensus process.
  2. Share the public key with the public, especially other rippled operators.
  3. When other rippled operators add your public key to their list of trusted servers, you have become a trusted validator.

Validator Setup

The validator-keys tool (included in the rippled RPM) is the recommended means to securely generate and manage your validator keys.

  1. Install a rippled server.

  2. Generate a validator key pair:

    $ /opt/ripple/bin/validator-keys create_keys
    

    Warning: Store the generated validator-keys.json key file in a secure but recoverable location, such as an encrypted USB flash drive. Do not modify its contents.

  3. Generate a validator token and edit your rippled.cfg file to add the [validator_token] value.

    $ /opt/ripple/bin/validator-keys create_token --keyfile /path/to/your/validator-keys.json
    

    If you had previously configured your validator without using the validator-keys tool, you must also delete the [validation_seed] from your rippled.cfg file. This changes your validator public key.

  4. Start rippled:

    $ sudo service rippled restart
    

See the validator-keys-tool GitHub repository for more information about managing validator keys.

Public-Facing Server

To protect a production validator from DDoS attacks, you can use a stock rippled server as a proxy between the validator and the outside network.

  1. Set up the rippled validator.

  2. Set up one or more stock rippled servers.

  3. Configure the validator and stock rippled servers to be clustered with each other.

  4. Make the following configuration changes to your validator:

    • Copy the [ips_fixed] list and paste it under [ips]. These fields should contain only the IP addresses and ports of the public-facing rippled(s). The validator connects to only these peers.
    • Change [peer_private] to 1 to prevent its IP address from being forwarded.
  5. Configure the validator host machine's firewall to only accept inbound connections from its public-facing rippled(s).

Remember to restart rippled for config changes to take effect.

Take care not to publish the IP address of your validator.

Domain Verification

Network participants are unlikely to trust validators without knowing who is operating them. To address this concern, validator operators can associate their validator with a web domain that they control.

  1. Find your validator public key by running the following on the validator server:

    $ /opt/ripple/bin/rippled server_info -q | grep pubkey_validator
    
  2. Sign the validator public key (from step 1) using the SSL private key used for your domain. The SSL private key file does not need to be stored on the validator server.

    $ openssl dgst -sha256 -hex -sign /path/to/your/ssl.key <(echo <your-validator-public-key>)
    
  3. Using validator-keys tool (included in the rippled RPM), sign the domain name:

    $ /opt/ripple/bin/validator-keys --keyfile /path/to/your/validator-keys.json sign <your-domain-name>
    
  4. To have the verified validator domain included on XRP Charts, submit this Google Form with your validator's information.

Additional Configuration

rippled should connect to the XRP Ledger with the default configuration. However, you can change your settings by editing the rippled.cfg file (located at /opt/ripple/etc/rippled.cfg when installing rippled with yum). For recommendations about configuration settings, see Capacity Planning.

See the rippled GitHub repository for a description of all configuration options.

Changes to the [debug_logfile] or [database_path] sections may require you to give the rippled user and group ownership to your new configured path:

    $ chown -R rippled:rippled <configured path>

Restart rippled for any configuration changes to take effect:

    $ sudo service rippled restart

Parallel Networks

Most of the time, we describe the XRP Ledger as one collective, singular entity -- and that's mostly true. There is one production XRP Ledger peer-to-peer network, and all business that takes place on the XRP Ledger occurs within the production network.

However, sometimes you may want to do tests and experiments without interacting with the core network. That's why Ripple started the Ripple Test Net, an "alternate universe" network, which can act as a testing ground for applications and the rippled server itself, without impacting the business operations of everyday XRP Ledger users. The Ripple Test Net (also known as the AltNet) has a separate supply of TestNet-only XRP, which Ripple gives away for free to parties interested in developing applications on the Test Net.

Caution: Ripple makes no guarantees about the stability of the test network. It has been and continues to be used to test various properties of server configuration, network topology, and network performance.

Over time, there may also be smaller, temporary test networks for specific purposes.

Parallel Networks and Consensus

There is no rippled setting that defines which network it uses. Instead, it uses the consensus of validators it trusts to know which ledger to accept as the truth. When different consensus groups of rippled instances only trust other members of the same group, each group continues as a parallel network. Even if malicious or misbehaving computers connect to both networks, the consensus process overrides the confusion as long as the members of each network are not configured to trust members of another network in excess of their quorum settings.

Clustering

If you are running multiple rippled servers in a single datacenter, you can configure those servers into a cluster to maximize efficiency. Running your rippled servers in a cluster provides the following benefits:

  • Clustered rippled servers share the work of cryptography. If one server has verified the authenticity of a message, the other servers in the cluster trust it and do not re-verify.
  • Clustered servers share information about peers and API clients that are misbehaving or abusing the network. This makes it harder to attack all servers of the cluster at once.
  • Clustered servers always propagate transactions throughout the cluster, even if the transaction does not meet the current load-based transaction fee on some of them.

To enable clustering, change the following sections of your config file for each server:

  • List the IP address and port of each other server under the [ips_fixed] section. The port should be the one from the other servers' protocol = peer setting in their rippled.cfg. Example:

    [ips_fixed]
    192.168.0.1 51235
    192.168.0.2 51235
    
  • Generate a unique seed (using the validation_create command) for each of your servers, and configure it under the [node_seed] section. The rippled server uses this key to sign its messages to other servers in the peer-to-peer network.

  • Add the public keys (for peer communication) of each of your other servers under the [cluster_nodes] section.