IRC meeting summary for 2017-05-11

Overview

Notes / short topics

  • Sipa and Gmaxwell have been working on a proposal to maintain a UTXO commitment hash all the time in a way that does this at a cost of a few microseconds per input and output. This would be useful for making gettxoutsetinfo instantaneous, or for syncing from someone else’s UTXO set, or as the basis for a softfork later. There are 3 ways of implementing this, all with different performance and security tradeoffs. A few days after the meeting a mailinglist post was made explaining the different approaches and options.

Main topics

  • Per-txo utxo database
  • fee targeting/coin selection overhaul

Per-txo utxo database

background

Since Bitcoin Core v0.8, we’ve used a per-tx model for the chainstate database and its cache. This means that the database is effectively a map from txids to the list of unspent outputs for that transaction. PR #10195 changes that to a map from outpoints ((txid,index) pairs) to just the individual unspent output for that outpoint.

The original reason for aggregating the outputs per transaction was to save space: this way, we could avoid duplicating the txid and transaction meta across the multiple outputs. However, LevelDB internally uses an encoding that omits repeated prefix bytes in keys, and because of that, duplicating the txids is not very significant.

There are many advantages for using a per-txout model:

  • Simpler code.
  • Avoiding the CPU overhead of deserializing and serializing the unused outputs.
  • More predictable memory usage.
  • More easily adaptable to various cache flushing strategies.

Downsides:

  • Slightly larger on-disk representation, and sometimes larger in-memory representation (when there are multiple outputs for the same txid in the cache, which becomes optional).

meeting comments

The results of testing are very promising as shown in this graph. Sipa notes the chainstate went from 2.2GB to 2.7GB in the test.

Gmaxwell thinks it should be merged quickly to have more testing in master, but it needs some more testing/review. BlueMatt is halfway through review, cfields made it through review but is not confident enough to ACK it.

There still needs to be a few code cleanups and a better user interface for the upgrade process, as there’s a one-time upgrade from the old database to the new one at startup, which takes a couple of minutes.

meeting conclusion

  • Review PR #10195 (Switch chainstate db and cache to per-txout model)

fee targeting/coin selection overhaul

background

Coin selection is the algorithm that figures out how to spend the coins you have in your wallet. Which ones should be used to fund the transaction. This is a balance between not creating non-standard transactions (by exceeding certain size, or creating dust outputs), trying to find a low fee configuration, give users as much privacy as possible and shrinking the overall UTXO set size.

There have been many improvements made over the years, but there are still issues that pop up once in a while. Instagibbs is working on changing the fee-targeting algorithm to consider “effective value” of considered inputs instead of simply trying to hit an absolute fee, seeing if it failed, then trying again with the estimated total fee at the end of the loop.

meeting comments

Gmaxwell feels a bit uneasy about the changes while there’s no strategy to sweep dust, he’s worried there’s a potential unintended UTXO count blowup.

As conversation goes more into the general approach of fee targeting and coin selection, Morcos mentions the idea of being “fee-smart”, where it could include more inputs when fees are low.

Wumpus likes the idea of cleaning up UTXOs with negative value during low fee-rates, but adds it would be better if those where not created in the first place. Morcos mentions his PR #9343 which tries to do that.

The design goal for Bitcoin Core is to never create change outputs smaller than 0.01BTC unless the wallet is being almost depleted with the transaction, however that goal is not achieved at all.

A good step forwards would also be to dissect and modularize the coin selection out of wallet.cpp.

meeting conclusion

  • a lot of ideas have come up for approaches to fee targeting and coin selection, to be discussed further outside the meeting.

High priority review

Instagibbs would like to add PR #10333 which fixes some of the fee issues reported by users, but is less extreme than the coin selection overhaul.

There where some review comments on multiwallet, however the PR is pending on PR #9494 which has been added to the high priority review last week.

Jonasschnelli would like to add PR #10240 (HD wallet auto-restore functionality) which should be included in 0.15.

Participants

IRC nick Name/Nym
jonasschnelli Jonas Schnelli
sipa Pieter Wuille
cfields Cory Fields
luke-jr Luke Dashjr
kanzure Bryan Bishop
gmaxwell Gregory Maxwell
BlueMatt Matt Corallo
instagibbs Gregory Sanders
wumpus Wladimir van der Laan
murchandamus Mark Erhardt
morcos Alex Morcos
sdaftuar Suhas Daftuar

Disclaimer

This summary was compiled without input from any of the participants in the discussion, so any errors are the fault of the summary author and not the discussion participants.