I just got back from New York, where I spent a lot of time talking to people. A lot of my discussions were about operating validators on the XRP Ledger. And a few people took me to task for not having delivered on my promise to write a blog post about how to operate a validator securely.

Vali-what-now?

First things first. The XRP Ledger is a decentralized peer-to-peer network of computers which, collectively, share a distributed database, processing transactions which alter the state of that database.

Some servers operate as “validators” which means that they participate in the process that is used to decide the order in which transactions are to be executed.

The basics

Ripple’s brand-spanking new developers.ripple.com portal is a wonderful resource and it includes a lot of great content; and it does, of course, include a step-by-step introduction to validators and how to go about running one.

The not-so-basics

So, installing the software and configuring it to run as a validator is not hard. What is hard is running the validator securely.

Validators contain sensitive cryptographic material: material which is used to derive the validator’s secret key; protecting it is extremely important.

Cryptographic Keys

This is, without a doubt, one of the most important topics in applied crypto—regardless of whether you think that the word is short for “cryptography” or “cryptocurrency”.

I’ll have more to say on this topic in a future blog post, but for now, I’ll choose to focus only on a set of “best practices” that I believe that validator operators should implement.

Lockdown

One of my primary considerations was the physical security of the machines. This may seem like an odd place to start, but it helps inform all our other decisions.

Disclaimer

Let’s be clear from the outset: no set of steps can guarantee security. The idea here isn’t to implement a hack-proof system, for no such thing exists.

The idea is to batten the hatches down as tightly as possible because the Internet is a hostile place.

Physical security

Rather than choosing to operate my validator from home or an office at a strip mall, I opted to co-locate it at a commercial hosting facility which uses sophisticated electronic monitoring equipment as well as around-the-clock security guards.

The server is located in a secure part of the facility, where access is restricted, and is further segregated from other equipment by a cage. The enclosure itself, while not strictly tamper-evident, is protected with tamper-evident stickers and is rigged so that an attempt to open it will write an event to the BIOS log.

Hosting, co-location and cloud computing providers

There are plenty of hosting and co-location options when it comes to operating a validator. Most larger cities have at least one facility that is suitable. Google is your friend here.

I’d generally recommend avoiding using a cloud computing provider. I have nothing against cloud computing; it is wonderful and I use it. But I wonder how many validators are running on any given cloud?

The hardware

Picking hardware is fun, even if it ends up expensive. The excellent capacity planning guide offered by Ripple should be the first stop here. The “Network and Hardware” section has more details.

For a production validator that will be sitting at a co-location facility, I strongly recommend purchasing server equipment as opposed to building a server using commodity hardware.

Ripple’s recommendations are good, but I chose to go above and beyond for my server vis-à-vis RAM and CPU; my suggestion is to not use less than 32GB of RAM and, if possible, to go for more than that—one can never have too much RAM. I also opted for a server with a TPM.

When it comes to disk storage, I use solid-state devices for everything except machines that are used as archiving servers. For a rippled server, whether a regular node, a validator or a full history server, I recommend using only solid-state storage; my personal preference is to use Samsung SSDs, but there are other other excellent options available. If you have an option, then you should prefer NVMe over SATA.

Bare metal

Part of the reason for upgrading the specs was that I was planning on running the validator as a VM guest, rather than directly on the bare metal.

This is not something I generally recommend at this time; I have very specific reasons for doing that, which are almost certainly not applicable for most other validator operators. I am only mentioning it here for completeness. If anyone is really interested, I could write a follow-up.

Randomness

While rippled doesn’t need a lot of random data, it does need some and it should be of good quality. Operating systems typically use an entropy pool that they seed using several sources of randomness. Rather than rely solely on the system’s entropy pool, rippled uses OpenSSL which provides APIs specifically designed to produce cryptographically secure random data.

But the more entropy is available to the system, the better off you will be and it is generally, it is a good idea to ensure that the system always has a sufficient amount of entropy available to feed into the entropy pool, especially when generating public-private keypairs.

One way to do that is to use a hardware true random number generator. These are devices which will generate a stream of random data from some physical process—often relying on tapping quantum mechnical physical randomness.

Some newer CPUs have such devices embedded into them, and modern operating systems will automatically use them. But external devices are also available and should, in my opinion, always be used, especially on servers.

These devices come in a wide range of form factors and prices, ranging from tiny to large and from affordable to… well… pricey. There are even several guides on how to make your own for those who like tinkering with electronics.

If you’re interested in purchasing such a device, I know of two affordable options that work well: the TrueRNG and the OneRNG. Both come with instructions on how to integrate them with your operating system; no configuration specific to rippled is needed.

The software

My validator is running on CentOS, which is one of the platforms that Ripple recommends (the other being CentOS’s enterprise cousin, RedHat Enterprise Linux).

Setup

Installing Linux is pretty straight-forward these days; I installed the bare minimum, mostly using the defaults, and made sure that the server was up to date.

The only changes I made was to dedicate an entire disk to rippled; I chose to have it mounted it under /opt/ripple, formatted it using ext4 and tweaked the default mount options to improve the performance of the filesystem, by using writeback mode and upping the commit timeout to 30. I don’t recommend similar tweaks unless your machine is connected to a UPS and configured to properly shut down in the event of a power outage.

If my validator was running on bare metal I would have, also, opted to encrypt the disks; on a modern CPU with AES-NI support, the overhead is minimal and the increased security is a small price to pay. Choosing to encrypt does have its drawbacks, however: whenever the machine restarts you would need to enter a passphrase. Unless you have the ability to connect to the machine’s console over the Internet, a reboot might mean a trip down to the colocation facility.

Sidenote: if you’re using Linux, you could rig the system to store the key in the TPM, but that is still unnecessarily hard on Linux, especially when compared to BitLocker on Windows. Better TPM integration is desperately needed.

Security Hardening

In addition to some basic administrative housekeeping, I also used some of the publicly available “recipes” to harden the operating system. The guides available for Linux are pretty thorough. Spending some time on this now can save you time and frustration in the future. Harden your operating system and restrict what is running to a bare minimum.

I also chose to configure some very restrictive firewall rules; I do not allow any incoming connections to the server, and only allow outgoing connections from the rippled process to the two servers that were going to be “fronting” my validator.

rippled

How you choose to install rippled is up to you. I download and install rpm packages. I probably should be compiling from source and installing, but I know, first hand, how the packages built and signed by Ripple are built and signed, so I feel confident that using them is safe.

Your mileage may vary.

Compiling from scratch affords you the opportunity to tweak compilation options to optimize the compiled executable for your specific architecture but you’ll need to make sure that you have a new enough version of the compiler and the various libraries.

Configuration

I make a few tweaks to the default rippled.cfg file.

As I mentioned earlier, I do not allow incoming connections, so there is really no point in specifying a port for peers to connect to, so I removed the [port_peer] stanza and the port_peer entry from the [server] stanza.

I also change [node_size] to huge. The value you have depends on the amount of RAM your server has available; consult with Ripple’s documentation on what the value should be.

I also choose to remove the [debug_logfile] stanza entirely; the information in those logs is only rarely useful outside of helping troubleshoot issues during development.

Using RocksDB is acceptable for a validator. But if you want to maintain full history, you should probably opt for NuDB.

I make some changes to the default settings, by changing the online_delete from 2000 to 30000. At a high-level, this value controls how often the server will remove old information from its databases. Keeping 2000 ledgers means that the server will perform this housekeeping approximately every two hours which is just too often. Unless disk space is at a premium, I would suggest changing this value to at between 30000 and 60000.

Node Seed

By default, the first time it starts, rippled will randomly generate a node seed, which is used to derive a unique cryptographic identity for each server. The seed is generated once, and is stored in wallet.db so that it can reused after every restart.

You can manually configure a node seed, but it is almost never required.

Validation Seed

In order to operate as a validator, rippled requires a unique public-private key pair; it will use the private key to sign validations, and others will use the public key to verify that the validations are genuine.

If you install rippled using the supplied packages, you will also have access to the validator-keys tool which simplifies the processes of generating a key.

The documentation outlines the steps needed to generate a new validation seed.

For more information on how to use the tool to do things like key revocation you can check out this guide.

Go forth and validate

A diverse group of validators is important for the long term health and stability of the XRP Ledger. No one group or entity should have control of the network, and the more secure validators we have, the less likely it becomes that any one group or entity will have such control.

If you are an exchange, gateway, or business transacting over the XRP Ledger, then it is in your interest to operate a validator, and there is no excuse to not do so.

Other ways to contribute

Operating a validator isn’t the only way to contribute to the XRP Ledger. If you are a cloud or hosting provider, you can contribute significantly by operating a high-capacity hub server.

If you are a developer or work with developers, you can contribute by developing new tools and applications, or by contributing code to rippled.

If you are an enthusiast, you can help educate others about the XRP Ledger and the broader space. There is a thriving community of similarly minded people on xrpchat.com; many also post on Twitter. You can find me there as @nbougalis.