payout: key obligations on block hash, and wait for coin instead of expiring

A 280-block reorg (heights 3075-3354) orphaned 292 of the pool's 707 blocks
and destroyed 14,600 BTF of coinbase. The payout ledger did not notice: it
records only matureAtHeight, and a reorg replaces the block at a height
without changing the height, so ~14.4k BTF of obligations outlived the coin
that was supposed to fund them. The pool then paid fork-era miners out of
real-chain coin until the wallet was empty -- 17,827 BTF paid against 17,900
BTF of matured supply, leaving under 1.1 BTF spendable and every subsequent
payment failing.

Three changes, all in the payout path:

- Record the funding block's hash and height, persist both, and void an
  obligation whose block is no longer in the main chain. Checked both at
  maturity and again in the exec queue, since an entry can sit there for
  hours across retries and the reorg can land at any point in that window.
  Entries loaded from a file written before this are grandfathered rather
  than voided on a missing field.

- Stop spending the retry budget on maturity. SelectCoins failing for want
  of spendable coin is not an error, but it consumed one of 1440 retries
  every 60s, so a payment the pool fully intended to honour would hit
  GIVING UP 24h later and the miner would silently never be paid. Those
  attempts now defer without incrementing retries, logged hourly rather
  than every minute. GIVING UP now means what it says: 1440 real failures
  against a block still on-chain with the money available.

- Schedule at COINBASE_SPENDABLE_DEPTH instead of COINBASE_MATURITY. At
  depth 100 a block's own coinbase is 20 blocks short of spendable, so its
  payout had to come from unrelated mature coin and the pool carried a
  permanent ~19-block float. The "96% of 934 payments went through at depth
  100" note that justified the shorter wait was measured while this node was
  alone on a private fork paying itself. The extra 20 blocks also put the
  maturity gate past the reorg that caused all of the above.

(cherry picked from commit 696386220a3d2fac88cb8045931392f4909e0985)

1 files changed, 126 insertions, 12 deletions

diff --git a/src/rpc.cpp b/src/rpc.cpp
index 24b1c88..b12dc2f 100644
--- a/src/rpc.cpp
+++ b/src/rpc.cpp
@@ -230,9 +230,18 @@ static bool RebuildJob()
 // Payouts -- event loop thread only
 // ---------------------------------------------------------------------------
 struct PendingPayout {
+    // The block that funds this payout, identified by hash and not by height
+    // alone. A reorg replaces the block at a height without changing the
+    // height, so a height-keyed obligation outlives the coinbase that was
+    // supposed to pay it: a 280-block reorg once left ~14.4k BTF of debt
+    // owed against blocks that no longer existed, which then drained the
+    // wallet paying fork-era miners out of real-chain coin.
+    int                          blockHeight = 0;
+    uint256                      blockHash;      // zero = entry predates hash keying
     int                          matureAtHeight;
     std::map<std::string, int64> amounts; // address -> satoshis
     int                          retries = 0;    // exec-thread only: SendMoney attempts so far
+    int                          deferrals = 0;  // exec-thread only: waits for spendable coin
     int64                        retryAfter = 0; // exec-thread only: unix time, 0 = ready now
 };
 static std::vector<PendingPayout> gPendingPayouts;
@@ -247,6 +256,8 @@ static void SavePendingPayouts()
     nlohmann::json arr = nlohmann::json::array();
     for (const auto& pp : gPendingPayouts) {
         nlohmann::json obj;
+        obj["blockHeight"]    = pp.blockHeight;
+        obj["blockHash"]      = pp.blockHash.ToString();
         obj["matureAtHeight"] = pp.matureAtHeight;
         nlohmann::json amounts = nlohmann::json::object();
         for (const auto& kv : pp.amounts)
@@ -281,6 +292,13 @@ static void LoadPendingPayouts()
         for (const auto& obj : arr) {
             PendingPayout pp;
             pp.matureAtHeight = obj["matureAtHeight"].get<int>();
+            // Entries written before payouts were keyed by hash carry neither
+            // field. Leave blockHash zero and they are grandfathered past the
+            // orphan check rather than voided on a guess.
+            if (obj.contains("blockHash") && obj["blockHash"].is_string())
+                pp.blockHash.SetHex(obj["blockHash"].get<std::string>());
+            if (obj.contains("blockHeight") && obj["blockHeight"].is_number())
+                pp.blockHeight = obj["blockHeight"].get<int>();
             for (auto it = obj["amounts"].begin(); it != obj["amounts"].end(); ++it)
                 pp.amounts[it.key()] = it.value().get<int64>();
             gPendingPayouts.push_back(pp);
@@ -300,7 +318,27 @@ static std::vector<PendingPayout> gPayoutExecQueue;
 static std::mutex                 gPayoutExecMutex;
 static std::atomic<bool>          gPayoutThreadRunning{false};
 
+// Is the block that funds this payout still on the main chain?
+//
+// A zero hash means the entry was written before payouts recorded one, so
+// there is nothing to check and we assume it is good -- voiding those would
+// mean refusing to pay miners on the strength of a missing field.
+static bool PayoutBlockStillOnChain(const PendingPayout& pp)
+{
+    if (pp.blockHash == 0)
+        return true;
+    CRITICAL_BLOCK(cs_main)
+    {
+        std::map<uint256, CBlockIndex*>::iterator mi = mapBlockIndex.find(pp.blockHash);
+        if (mi == mapBlockIndex.end() || mi->second == NULL)
+            return false;
+        return mi->second->IsInMainChain();
+    }
+    return false;
+}
+
 static void QueuePayouts(int blockHeight,
+                         const uint256& blockHash,
                          const std::map<std::string, uint64>& shareCount,
                          uint64 total)
 {
@@ -322,12 +360,19 @@ static void QueuePayouts(int blockHeight,
     int64 minerReward = (int64)((double)nSubsidy * (1.0 - feeFrac));
 
     PendingPayout pp;
-    // Deliberately COINBASE_MATURITY, not COINBASE_SPENDABLE_DEPTH: a payout is
-    // funded from whatever coin is already mature, not from its own block's
-    // coinbase, so waiting the extra 20 blocks delays every payment without
-    // making one more likely to succeed. Measured over 934 payments: 96% went
-    // through at depth 100, the moment they were queued.
-    pp.matureAtHeight = blockHeight + COINBASE_MATURITY;
+    pp.blockHeight = blockHeight;
+    pp.blockHash   = blockHash;
+    // COINBASE_SPENDABLE_DEPTH, not COINBASE_MATURITY. Paying at depth 100
+    // means the block's own coinbase is still 20 blocks short of spendable, so
+    // the payment has to come out of some *other* mature coin -- the pool runs
+    // a permanent ~19-block float and every payout fails until unrelated coin
+    // matures. The earlier "96% of 934 payments went through at depth 100"
+    // measurement that justified the shorter wait was taken while this node was
+    // alone on a private fork paying itself, so it says nothing about a pool
+    // competing on the real chain. Waiting the extra 20 blocks makes each
+    // payout funded by construction, and puts the maturity gate past the
+    // reorg depth that orphaned 280 blocks of obligations.
+    pp.matureAtHeight = blockHeight + COINBASE_SPENDABLE_DEPTH;
 
     int skipped = 0;
     int64 burned = 0; // amount owed to miners with bad/empty addresses
@@ -384,6 +429,23 @@ static void QueuePayouts(int blockHeight,
 // re-queueing blindly would double-pay whoever was already sent to.
 static void FlushMaturePayouts()
 {
+    // Void obligations whose funding block lost a reorg. The coin that was
+    // meant to pay them does not exist, so paying anyway spends real-chain
+    // coin the pool owes to somebody else.
+    bool voided = false;
+    for (auto it = gPendingPayouts.begin(); it != gPendingPayouts.end(); ) {
+        if (PayoutBlockStillOnChain(*it)) { ++it; continue; }
+        int64 total = 0;
+        for (auto& kv : it->amounts) total += kv.second;
+        LogPrint("payout", "[payout] block %d (%s) was orphaned -- voiding %zu"
+                 " payment(s) totalling %s\n",
+                 it->blockHeight, it->blockHash.ToString().substr(0, 14).c_str(),
+                 it->amounts.size(), FormatMoney(total).c_str());
+        it = gPendingPayouts.erase(it);
+        voided = true;
+    }
+    if (voided) SavePendingPayouts();
+
     std::vector<PendingPayout> ready;
     for (auto it = gPendingPayouts.begin(); it != gPendingPayouts.end(); ) {
         if (nBestHeight < it->matureAtHeight) { ++it; continue; }
@@ -429,6 +491,18 @@ static void PayoutThreadFn(void*)
                 gPayoutExecQueue.push_back(pp);
                 continue;
             }
+            // Re-check here and not only at maturity: an entry can sit in this
+            // queue for hours across retries, and the reorg that kills its
+            // funding block can land at any point in that window.
+            if (!PayoutBlockStillOnChain(pp)) {
+                int64 total = 0;
+                for (auto& kv : pp.amounts) total += kv.second;
+                LogPrint("payout", "[payout] block %d (%s) orphaned while queued"
+                         " -- dropping %zu payment(s) totalling %s\n",
+                         pp.blockHeight, pp.blockHash.ToString().substr(0, 14).c_str(),
+                         pp.amounts.size(), FormatMoney(total).c_str());
+                continue;
+            }
             LogPrint("payout", "[payout] executing payments for block matured at %d"
                      " (%zu miners)\n", pp.matureAtHeight, pp.amounts.size());
             for (auto& kv : pp.amounts) {
@@ -443,7 +517,35 @@ static void PayoutThreadFn(void*)
                 sc << OP_DUP << OP_HASH160 << h160 << OP_EQUALVERIFY << OP_CHECKSIG;
                 CWalletTx wtx;
                 wtx.mapValue["comment"] = strprintf("Pool payout height %d",
-                                                     pp.matureAtHeight);
+                                                     pp.blockHeight);
+
+                // Waiting for coin to mature is not a failure, so it must not
+                // consume the retry budget. Burning a retry every 60s means a
+                // payment the pool fully intends to honour can hit GIVING UP
+                // 24h later purely because the subsidy funding it had not
+                // matured yet -- the miner is then silently never paid.
+                int64 nSpendable = GetBalance();
+                if (nSpendable < kv.second) {
+                    PendingPayout dp;
+                    dp.blockHeight    = pp.blockHeight;
+                    dp.blockHash      = pp.blockHash;
+                    dp.matureAtHeight = pp.matureAtHeight;
+                    dp.amounts[kv.first] = kv.second;
+                    dp.retries    = pp.retries;          // unchanged on purpose
+                    dp.deferrals  = pp.deferrals + 1;
+                    dp.retryAfter = GetTime() + 60;
+                    // Hourly, not every minute: a dry pool would otherwise
+                    // write one line per payment per minute forever.
+                    if (dp.deferrals == 1 || dp.deferrals % 60 == 0)
+                        LogPrint("payout", "[payout] deferring %s to %s -- only %s"
+                                 " spendable (waiting %d min so far)\n",
+                                 FormatMoney(kv.second).c_str(), kv.first.c_str(),
+                                 FormatMoney(nSpendable).c_str(), dp.deferrals);
+                    std::lock_guard<std::mutex> lk(gPayoutExecMutex);
+                    gPayoutExecQueue.push_back(std::move(dp));
+                    continue;
+                }
+
                 if (SendMoney(sc, kv.second, wtx)) {
                     LogPrint("payout", "[payout] paid %s to %s\n",
                              FormatMoney(kv.second).c_str(), kv.first.c_str());
@@ -453,24 +555,34 @@ static void PayoutThreadFn(void*)
                     // with backoff. Permanent cases (orphaned coinbase) give
                     // up loudly after 1440 tries (~24 h).
                     PendingPayout rp;
+                    rp.blockHeight    = pp.blockHeight;
+                    rp.blockHash      = pp.blockHash;
                     rp.matureAtHeight = pp.matureAtHeight;
                     rp.amounts[kv.first] = kv.second;
                     rp.retries = pp.retries + 1;
+                    rp.deferrals = pp.deferrals;
                     rp.retryAfter = GetTime() + 60;
                     {
                         std::lock_guard<std::mutex> lk(gPayoutExecMutex);
                         gPayoutExecQueue.push_back(std::move(rp));
                     }
                     LogPrint("payout", "[payout] FAILED to pay %s to %s"
-                             " -- retry %d/1440 in 60s\n",
+                             " (%s spendable at the time) -- retry %d/1440 in 60s\n",
                              FormatMoney(kv.second).c_str(), kv.first.c_str(),
+                             FormatMoney(nSpendable).c_str(),
                              pp.retries + 1);
                 } else {
+                    // Reaching here now means 1440 attempts that were NOT
+                    // insufficient-funds waits (those defer instead) against a
+                    // block still on the main chain -- i.e. a real wallet fault,
+                    // not a maturity or reorg question.
                     LogPrint("payout", "[payout] GIVING UP on %s to %s after %d"
-                             " retries (orphaned coinbase or persistent wallet"
-                             " error?) -- payment forfeited\n",
+                             " retries -- block %d is still on-chain and %s was"
+                             " spendable, so this is a wallet fault, not maturity"
+                             " -- payment forfeited\n",
                              FormatMoney(kv.second).c_str(), kv.first.c_str(),
-                             pp.retries);
+                             pp.retries, pp.blockHeight,
+                             FormatMoney(nSpendable).c_str());
                 }
             }
         }
@@ -1065,7 +1177,9 @@ static bool HandleLine(Miner* m, const std::string& rawLine,
         if (accepted) {
             // Snapshot height before RebuildJob() (which increments it)
             int foundHeight = gCurrentJob.height;
-            QueuePayouts(foundHeight, roundShareCount, roundShareTotal);
+            // b is the block we submitted, untouched by ProcessBlock (which
+            // took a copy), so its hash is what the payout must be keyed on.
+            QueuePayouts(foundHeight, b.GetHash(), roundShareCount, roundShareTotal);
             // Reset round shares
             roundShareCount.clear();
             roundShareTotal = 0;