Securing validators18 May 2018
I just got back from New York, where I spent a lot of time talking to people. A lot of my discussions were about operating validators on the XRP Ledger. And a few people took me to task for not having delivered on my promise to write a blog post about how to operate a validator securely.
First things first. The XRP Ledger is a decentralized peer-to-peer network of computers which, collectively, share a distributed database, processing transactions which alter the state of that database.
Some servers operate as “validators” which means that they participate in the process that is used to decide the order in which transactions are to be executed.
Ripple’s brand-spanking new developers.ripple.com portal is a wonderful resource and it includes a lot of great content; and it does, of course, include a step-by-step introduction to validators and how to go about running one.
So, installing the software and configuring it to run as a validator is not hard. What is hard is running the validator securely.
Validators contain sensitive cryptographic material: material which is used to derive the validator’s secret key; protecting it is extremely important.
This is, without a doubt, one of the most important topics in applied crypto—regardless of whether you think that the word is short for “cryptography” or “cryptocurrency”.
I’ll have more to say on this topic in a future blog post, but for now, I’ll choose to focus only on a set of “best practices” that I believe that validator operators should implement.
One of my primary considerations was the physical security of the machines. This may seem like an odd place to start, but it helps inform all our other decisions.
Let’s be clear from the outset: no set of steps can guarantee security. The idea here isn’t to implement a hack-proof system, for no such thing exists.
The idea is to batten the hatches down as tightly as possible because the Internet is a hostile place.
Rather than choosing to operate my validator from home or an office at a strip mall, I opted to co-locate it at a commercial hosting facility which uses sophisticated electronic monitoring equipment as well as around-the-clock security guards.
The server is located in a secure part of the facility, where access is restricted, and is further segregated from other equipment by a cage. The enclosure itself, while not strictly tamper-evident, is protected with tamper-evident stickers and is rigged so that an attempt to open it will write an event to the BIOS log.
Hosting, co-location and cloud computing providers
There are plenty of hosting and co-location options when it comes to operating a validator. Most larger cities have at least one facility that is suitable. Google is your friend here.
I’d generally recommend avoiding using a cloud computing provider. I have nothing against cloud computing; it is wonderful and I use it. But I wonder how many validators are running on any given cloud?
Picking hardware is fun, even if it ends up expensive. The excellent capacity planning guide offered by Ripple should be the first stop here. The “Network and Hardware” section has more details.
For a production validator that will be sitting at a co-location facility, I strongly recommend purchasing server equipment as opposed to building a server using commodity hardware.
Ripple’s recommendations are good, but I chose to go above and beyond for my server vis-à-vis RAM and CPU; my suggestion is to not use less than 32GB of RAM and, if possible, to go for more than that—one can never have too much RAM. I also opted for a server with a TPM.
When it comes to disk storage, I use solid-state devices for everything
except machines that are used as archiving servers. For a
whether a regular node, a validator or a full history server, I recommend
using only solid-state storage; my personal preference is to use Samsung
SSDs, but there are other other excellent options available. If you have
an option, then you should prefer NVMe over SATA.
Part of the reason for upgrading the specs was that I was planning on running the validator as a VM guest, rather than directly on the bare metal.
This is not something I generally recommend at this time; I have very specific reasons for doing that, which are almost certainly not applicable for most other validator operators. I am only mentioning it here for completeness. If anyone is really interested, I could write a follow-up.
rippled doesn’t need a lot of random data, it does need some and
it should be of good quality. Operating systems typically use an entropy
pool that they seed using several sources of randomness. Rather than rely
solely on the system’s entropy pool,
which provides APIs specifically designed to produce cryptographically
secure random data.
But the more entropy is available to the system, the better off you will be and it is generally, it is a good idea to ensure that the system always has a sufficient amount of entropy available to feed into the entropy pool, especially when generating public-private keypairs.
One way to do that is to use a hardware true random number generator. These are devices which will generate a stream of random data from some physical process—often relying on tapping quantum mechnical physical randomness.
Some newer CPUs have such devices embedded into them, and modern operating systems will automatically use them. But external devices are also available and should, in my opinion, always be used, especially on servers.
These devices come in a wide range of form factors and prices, ranging from tiny to large and from affordable to… well… pricey. There are even several guides on how to make your own for those who like tinkering with electronics.
If you’re interested in purchasing such a device, I know of two affordable
options that work well: the TrueRNG and the
OneRNG. Both come with instructions on how to
integrate them with your operating system; no configuration specific to
rippled is needed.
My validator is running on CentOS, which is one of the platforms that Ripple recommends (the other being CentOS’s enterprise cousin, RedHat Enterprise Linux).
Installing Linux is pretty straight-forward these days; I installed the bare minimum, mostly using the defaults, and made sure that the server was up to date.
The only changes I made was to dedicate an entire disk to
chose to have it mounted it under
/opt/ripple, formatted it using ext4
and tweaked the default mount options
to improve the performance of the filesystem, by using
writeback mode and
commit timeout to 30. I don’t recommend similar tweaks unless
your machine is connected to a UPS and configured to properly shut down
in the event of a power outage.
If my validator was running on bare metal I would have, also, opted to encrypt the disks; on a modern CPU with AES-NI support, the overhead is minimal and the increased security is a small price to pay. Choosing to encrypt does have its drawbacks, however: whenever the machine restarts you would need to enter a passphrase. Unless you have the ability to connect to the machine’s console over the Internet, a reboot might mean a trip down to the colocation facility.
Sidenote: if you’re using Linux, you could rig the system to store the key in the TPM, but that is still unnecessarily hard on Linux, especially when compared to BitLocker on Windows. Better TPM integration is desperately needed.
In addition to some basic administrative housekeeping, I also used some of the publicly available “recipes” to harden the operating system. The guides available for Linux are pretty thorough. Spending some time on this now can save you time and frustration in the future. Harden your operating system and restrict what is running to a bare minimum.
I also chose to configure some very restrictive firewall rules; I do not
allow any incoming connections to the server, and only allow outgoing
connections from the
rippled process to the two servers that were going
to be “fronting” my validator.
How you choose to install
rippled is up to you. I download and install
rpm packages. I probably should be compiling from source and installing,
but I know, first hand, how the packages built and signed by Ripple are
built and signed, so I feel confident that using them is safe.
Your mileage may vary.
Compiling from scratch affords you the opportunity to tweak compilation options to optimize the compiled executable for your specific architecture but you’ll need to make sure that you have a new enough version of the compiler and the various libraries.
I make a few tweaks to the default
As I mentioned earlier, I do not allow incoming connections, so there is
really no point in specifying a port for peers to connect to, so I removed
[port_peer] stanza and the
port_peer entry from the
I also change
huge. The value you have depends on the
amount of RAM your server has available; consult with Ripple’s documentation
on what the value should be.
I also choose to remove the
[debug_logfile] stanza entirely; the
information in those logs is only rarely useful outside of helping
troubleshoot issues during development.
RocksDB is acceptable for a validator. But if you want to maintain
full history, you should probably opt for
I make some changes to the default settings, by changing the
from 2000 to 30000. At a high-level, this value controls how often the server
will remove old information from its databases. Keeping 2000 ledgers means
that the server will perform this housekeeping approximately every two hours
which is just too often. Unless disk space is at a premium, I would suggest
changing this value to at between 30000 and 60000.
By default, the first time it starts,
rippled will randomly generate a node
seed, which is used to derive a unique cryptographic identity for each server.
The seed is generated once, and is stored in
wallet.db so that it can reused
after every restart.
You can manually configure a node seed, but it is almost never required.
In order to operate as a validator,
rippled requires a unique
public-private key pair; it will use the private key to sign validations,
and others will use the public key to verify that the validations are
If you install
rippled using the supplied packages, you will also have
access to the
validator-keys tool which simplifies the processes of
generating a key.
The documentation outlines the steps needed to generate a new validation seed.
For more information on how to use the tool to do things like key revocation you can check out this guide.
Go forth and validate
A diverse group of validators is important for the long term health and stability of the XRP Ledger. No one group or entity should have control of the network, and the more secure validators we have, the less likely it becomes that any one group or entity will have such control.
If you are an exchange, gateway, or business transacting over the XRP Ledger, then it is in your interest to operate a validator, and there is no excuse to not do so.
Other ways to contribute
Operating a validator isn’t the only way to contribute to the XRP Ledger. If you are a cloud or hosting provider, you can contribute significantly by operating a high-capacity hub server.
If you are a developer or work with developers, you can contribute by
developing new tools and applications, or by contributing code to
If you are an enthusiast, you can help educate others about the XRP Ledger and the broader space. There is a thriving community of similarly minded people on xrpchat.com; many also post on Twitter. You can find me there as @nbougalis.