Bitcoin Core 0.14.0 Released with Performance Improvements

We are pleased to release Bitcoin Core 0.14.0, which significantly speeds up the processing of historic blocks by newly started nodes and the validation and relay of new blocks by online nodes. An optional new feature (disabled by default) is also provided to allow wallet users to rebroadcast one of their previously-sent unconfirmed transactions with a higher fee, which may allow it to confirm more quickly.

A short summary of the major new features is provided below, with links to more details later in this post.

• Improved Initial Block Download (IBD) performance: a full node that has been started for the first time can now validate all blocks up until the present faster than before. This is at least the sixth time IBD performance has been significantly improved by developers to help ensure that new users aren’t discouraged from running a full node because synchronizing the ever-growing block chain takes too long.

• Faster new block validation and relay: miners in particular will benefit from improvements made in four existing Bitcoin Core features. The signature cache has been upgraded for machines with many CPU cores—a test on a system with 16 cores shows a 40% speed increase in processing a new block. The BIP152 CompactBlocks implementation will now relay some blocks before they’ve been fully validated, allowing those blocks to propagate across the peer-to-peer (P2P) network faster than before. The P2P network code has also generally been refactored to allow multiple actions to happen at the same time (concurrency) as well as to increase throughput, eliminating many potential delays in processing new blocks. Finally, unconfirmed transactions in each node’s mempool can now be saved to and restored from disk when the node is restarted.

• Optional fee bumping: Bitcoin Core’s wallet now allows users to optionally send transactions that opt-in to fee-based transaction replacement. This allows users to increase the fee their transactions pay even after they’ve broadcast an earlier version of the transaction. This feature is disabled by default.

Improved Initial Block Download (IBD) Performance

Over time, the constantly-increasing size of the block chain has forced new first-time nodes to process larger and large amounts of data before they can be used with a wallet to receive and send payments. Many previous Bitcoin and Bitcoin Core releases have included major improvements designed to eliminate the pain of this Initial Block Download (IBD), also called the initial sync. For example:

This 0.14.0 release includes two more improvements that significantly increase IBD speed:

1. Assumed valid blocks
2. Memory-sharing between mempool and UTXO DB cache

Those two optimizations are described in more detail below. A test of the speed of the previous release (Bitcoin Core 0.13.2) compared to the speed of this Bitcoin Core 0.14.0 release was performed using Amazon EC2 virtual private servers, type t2.xlarge with four cores and 16 GB memory at a cost of $0.188 USD per hour (excluding block storage costs). All Bitcoin Core settings were left at their defaults.

• Bitcoin Core 0.13.2 took 1 day, 12 hours, and 40 minutes to complete IBD for a total cost of $6.89.

• Bitcoin Core 0.14.0 took 0 days, 6 hours, and 24 minutes to complete IBD for a total cost of $1.20.

Under these testing conditions, Bitcoin Core 0.14.0 was 5.7 times faster at IBD than the previous version of Bitcoin Core.

Note: Bitcoin Core’s defaults are chosen to allow it to run on a large variety of hardware, including older machines with small amounts of memory. For users who want a faster sync than six hours, consider starting Bitcoin Core with the -dbcache= option higher than its 300 (MB) default. Many modern computers should be able to sync in about 3 hours using Bitcoin Core 0.14.0 and 8 GB of memory (-dbcache=8000).

Assumed valid blocks

Bitcoin 0.3.2 introduced a mechanism called checkpoints to prevent denial-of-service attacks during IBD by ensuring that new full nodes couldn’t be tricked into spending excessive amounts of effort validating alternative block chains that were different than the best-known chain at certain points in time.

Bitcoin 0.5.0 built upon those checkpoints to speed up IBD by skipping verification of signatures in blocks that were earlier in the block chain than the most recent checkpoint.

Over time, other improvements in Bitcoin Core’s security (such as headers-first sync and minimum chainwork) has reduced the need for checkpoints, while at the same time many Bitcoin developers have expressed the desire to remove checkpoints because they confuse new developers about the security model—checkpoints make it look like Bitcoin developers are deciding which chain is valid even though the intent was always to only accept the chain which the software had already decided was valid.

Assumed valid blocks is a new feature that separates the signature-skipping optimization from the checkpoints anti-denial-of-service mechanism so they can each be dealt with independently.

An assumed valid block is a block that individual users consider to be valid, including being part of a valid block chain. This is easy for anyone to test in a completely repeatable (deterministic) way: take any consensus-compatible Bitcoin implementation and run it on the whole block chain including the block in question. If the software doesn’t reject the block or any preceding block, it’s valid.

If someone who starts a new full node for the first time knows about any valid blocks, they can then provide the highest-height one of those blocks to Bitcoin Core 0.14.0 and the software will skip verifying signatures in the blocks before the assumed valid block. Since verifying signatures consumes a lot of CPU during IBD, using assumed valid blocks can significantly speed up IBD. All blocks after the assumed valid block will still have their signatures checked normally.

A critical difference between checkpoints and assumed valid blocks is that Bitcoin 0.3.2 required all checkpointed blocks to be part of the block chain but Bitcoin Core 0.14.0 does not require any assumed valid blocks to be part of the chain. If no assumed valid block provided by the user (or the system defaults) is part of the block chain, Bitcoin Core will simply verify all signatures for historic blocks. Or, if there’s a block chain fork and the valid block chain with the most proof of work doesn’t include a known assumed valid block, Bitcoin Core will still switch to the new best block chain even though it means abandoning the assumed valid block.

New users to Bitcoin probably won’t know about any valid blocks, but they probably also won’t know all the consensus rules—so they can simply use the copy of the full node software they download. Bitcoin Core 0.14.0 ships with a default assumed valid block that is set during the release process by having multiple well-known developers each confirm that a certain block is known to be valid by them (example).

Anyone who wants to verify all signatures using Bitcoin Core can still do so by starting the program using -assumevalid=0. Anyone who wants to specify an alternative assumed valid block can specify the block identifier as a parameter to assumevalid; for example:

-assumevalid=00000000000000000013176bf8d7dfeab4e1db31dc93bc311b436e82ab226b90

The default assumed valid block in Bitcoin Core 0.14.0 is #453354, 16 Februrary 2017, with hash 00000000000000000013176bf8d7dfeab4e1db31dc93bc311b436e82ab226b90.

Memory-sharing between mempool and UTXO DB cache

During IBD, Bitcoin Core doesn’t use it’s mempool because until you have the most recent blocks there’s no way to verify most recently-created transactions. This means that the memory allocated to the mempool has historically simply been unallocated, meaning Bitcoin Core ran with less memory during IBD than it did normally.

In Bitcoin Core 0.14.0, unused mempool memory is shared with the UTXO database cache, which increases the number of Unspent Transaction Outputs (UTXOs) that can be cached in fast memory rather than needing to be stored and retrieved from a much slower disk drive.

Faster new block validation and relay

Four significant improvements in Bitcoin Core 0.14.0 will be especially interesting to miners and other users who need to receive and process new blocks as fast as possible.

Improved signature cache

The first feature is an update of the signature cache to use cuckoo hashing. The signature cache allows Bitcoin Core to save (cache) the result of verifying signatures in an unconfirmed transaction so that when the same transaction appears in a new block, Bitcoin Core doesn’t need to verify the signatures again. Because signature verification is typically the most computationally-expensive part of processing a new block, using the signature cache significantly improves the speed at which new blocks can be processed by nodes that have been online for a while.

The existing signature cache in Bitcoin Core 0.13.2 and below works well for systems with fewer than eight CPU cores, but for systems with eight or more CPU cores a bottleneck (lock contention) appears that prevents the system from using all available cores as effectively as possible. The upgrade to using cuckoo hashing to provide a “cuckoo cache” eliminates this problem and allows more cores to be used effectively.

In a test on a system with 16 cores, a new block was able to be added (connected) to the block chain 40% faster using 0.14.0 than a previous version of Bitcoin Core. For systems with fewer than 8 cores, there is no major performance increase, although the cuckoo cache does allow caching more signatures than before for the same amount of memory, so there can be a slight performance improvement.

Earlier BIP152 compactblock relay

The second feature improved in Bitcoin Core 0.14.0 is the BIP152 compactblock implementation. The previous implementation supported both of BIP152’s two opt-in modes:

• A low-bandwidth mode that attempts to send the minimum data necessary to relay a new block, including waiting for the receiving node to request that specific new block.

• A high-bandwidth mode that sends new block data without waiting for the receiving node to request that specific block. This risks sending the receiving node the same data that another node sent it—a waste of bandwidth—but helps ensure that blocks are transferred quickly.

The upgraded implementation enhances the high-bandwidth node by starting the relay of a new block before the block has been fully validated. In the best case, removing the validation delay can allow new blocks to propagate across multiple hops on the peer-to-peer network several times faster than they could before. In the worst case, some additional bandwidth is wasted transferring invalid blocks. In either case, the security model remains the same since all nodes will still reject invalid blocks.

P2P code refactor focused on concurrency and throughput

The third feature improved in Bitcoin Core 0.14.0 is the entire set of P2P networking code, which has been refactored to support increased concurrency and throughput. The concurrency improvements help allow newly-received blocks (such as BIP152 compactblocks) to be processed ahead of lower-priority traffic, ensuring that blocks are relayed and validated as fast as possible.

The refactor also now allows network activity to continue in the background during message processing, notably providing an improvement in IBD speed that complements the headers-first sync introduced in Bitcoin Core 0.10.0 (see above for more information).

Mempool saved to disk

A fourth feature which helps support the signature cache and compactblocks implementations is that the memory pool (mempool) of unconfirmed transactions received by each node is now saved to disk during regular shutdown, and is then loaded back into memory when the node starts back up.

Combined with compact blocks, this can save the node from having to re-download all those unconfirmed transactions when they are received in a newly-produced block. Combined with the signature cache, this allows the node to cache the signature verification of those unconfirmed transactions so that new blocks including those transactions can be validated more quickly.

Optional fee bumping

An optional feature in Bitcoin Core 0.14.0 (disabled by default) causes all new transactions produced by the the wallet to opt-in to transaction replacement, specifically BIP125 opt-in Replace By Fee (RBF).

When this feature is enabled by starting Bitcoin Core with the -walletrbf option, an additional RPC (bumpfee) is made available that allows a previously-sent unconfirmed transaction to be resent with a higher fee. Miners who support either opt-in RBF or full RBF will usually replace the lower-fee transaction with the higher-fee transaction in their queues, and the higher fee will encourage miners to mine the new version of the transaction more quickly.

Conclusion

For details on all the changes made in Bitcoin Core 0.14.0, please read the release notes. To download, please visit the download page or the files directory.

The next major planned release will be scheduled for approximately six months after the 0.14.0 release (but it will not be released until fully tested).

If you are interested in contributing to Bitcoin Core, please see our contributing page and the document How to contribute code to Bitcoin Core. If you don’t know where to get started or have any other questions, please stop by either our IRC or Slack chatrooms and we’ll do our best to help you.

Hashes for verification

These are the SHA-256 hashes of the released files:

466adccf7352f06de35afc1627a3ea721764268ceaf08fa3641f9b47e7df091a bitcoin-0.14.0-aarch64-linux-gnu.tar.gz 55957e2c35aa2ba836cbae7cbf945bcf489a46b243551b0f6fd86f60603032a6 bitcoin-0.14.0-arm-linux-gnueabihf.tar.gz e4bb8b52acde07788dfcf024645fe291f0deca2b7172939fb2ddb8789fe56973 bitcoin-0.14.0-i686-pc-linux-gnu.tar.gz e01e3cdd3c4138eccaf0c1267caa3bcdb6949ee63c1e396842b70f102fb4bcaf bitcoin-0.14.0-osx64.tar.gz 50fea43935e93381552b6730444eed6bbe513637a785e1b864b0c5883729228c bitcoin-0.14.0-osx.dmg d743d4866a0d4c1457f81530c45258a8b6383d1cafc458eedcba8d01728a641e bitcoin-0.14.0.tar.gz 95a030be5c1649023e3aa81d1cd9eabd4258f1b00f0fc51066d02126219705af bitcoin-0.14.0-win32-setup.exe 864ef77b9b4812ec59adff04d44213a6039c66970a9ae31e8620997a8c1537bc bitcoin-0.14.0-win32.zip f260d52cf2fe91c4be99ed6fcf8aa0de669ff326c5da920b7ed3a3e2ec981e0a bitcoin-0.14.0-win64-setup.exe 415693ed81cfc4960bbfcb815529003405aefbf839ef8fc901b0a2c4ef5317d0 bitcoin-0.14.0-win64.zip 06e6ceeb687e784e9aaad45e9407c7eed5f7e9c9bbe44083179287f54f0f9f2b bitcoin-0.14.0-x86_64-linux-gnu.tar.gz