payout: do not write off a debt the wallet is only temporarily unable to pay

ForfeitPayoutRecipient permanently erases a miner's money: the amount leaves
the ledger, the operator keeps the coin, and nothing on disk records that it
was ever owed. It was reachable from two entirely ordinary conditions.

The deferral gate compared GetBalance() against r.amount, ignoring the fee.
A balance anywhere between amount and amount+fee therefore passed the gate,
failed inside SendMoney, and was counted as a send FAILURE -- a retry. At one
a minute, 1440 of those is 24 hours, and then the recipient is gone. A wallet
left locked overnight did exactly the same thing: every attempt fails because
nothing can be signed, and each failure spent budget.

Three changes.

The gate now requires amount + nTransactionFee, which is the part of the fee
knowable before the transaction exists. It deliberately does not try to
reserve GetMinFee(), which depends on the serialised size and so on how many
inputs SelectCoins happens to pick: refusing to attempt a payment the wallet
could actually have made is its own way of not paying a miner. A locked wallet
is now a deferral at the same gate, for the same reason maturity is.

Failures are classified after the fact instead of all counting as faults. The
balance and lock state are re-read after SendMoney returns -- it holds cs_main
and can block for a long time, so both may have moved -- and if the wallet is
locked, or the balance no longer covers amount + fee + one fee tier of
headroom, the attempt is recorded as a deferral. That headroom is where the
size-dependent part of the fee lands, so a fee shortfall is a wait rather than
a fault, which is what it is.

The retry budget therefore now only counts failures that happened while the
payment was funded, the wallet was unlocked, and the funding block was on the
main chain. Reaching 1440 of those means the wallet refuses a payment it has
every means to make, which is the only state in which the payment can be
called impossible rather than merely blocked. The give-up log says which
conditions held, rather than implying it.

Finally, the forfeit is reconciled with the public record. Dropping the amount
from the ledger while pool_rounds.json goes on advertising the address as owed
with "txid": null left the pool's two accounts of the same round permanently
disagreeing, with the operator holding the difference. PoolRoundProof gains a
per-address forfeited/forfeitReason, written by MarkPoolRoundForfeited when
the ledger writes the debt off, and pool_status.json counts those separately
from genuinely unpaid ones instead of reporting them as still pending.

On-disk: pool_rounds.json gains optional "forfeited" and "forfeitReason" keys
inside a payout object, and pool_status.json gains "forfeitedPayouts" per
round. Both are emitted only when a forfeit has actually happened, both are
read back with defaults, and files written by any earlier build load unchanged.

1 files changed, 175 insertions, 39 deletions

diff --git a/src/rpc.cpp b/src/rpc.cpp
index b71457c..6f6d101 100644
--- a/src/rpc.cpp
+++ b/src/rpc.cpp
@@ -386,6 +386,12 @@ struct PoolRoundProof {
     std::map<std::string, uint64> shares;
     std::map<std::string, int64>  payouts;
     std::map<std::string, std::string> paidTxids;
+    // Recipients the ledger has written off, and why. Without this the public
+    // round proof goes on advertising an address as owed with "txid": null
+    // forever, while the ledger has already dropped the obligation -- the two
+    // records of the same round permanently disagreeing, with the operator
+    // holding the money.
+    std::map<std::string, std::string> forfeited; // address -> reason
 
     PoolRoundProof()
         : blockHeight(0), matureAtHeight(0), createdAt(0), feePercent(0.0),
@@ -890,6 +896,13 @@ static nlohmann::json PoolRoundToJson(const PoolRoundProof& r)
             p["txid"] = tx->second;
         else
             p["txid"] = nullptr;
+        // Only emitted when it happened, so the common shape of the file is
+        // unchanged for anything already parsing it.
+        std::map<std::string, std::string>::const_iterator fo = r.forfeited.find(kv.first);
+        if (fo != r.forfeited.end()) {
+            p["forfeited"] = true;
+            p["forfeitReason"] = fo->second;
+        }
         j["payouts"][kv.first] = p;
     }
     return j;
@@ -966,6 +979,9 @@ static void LoadPoolRounds()
                         r.payouts[it.key()] = it.value().value("amountSatoshis", (int64)0);
                         if (it.value().contains("txid") && it.value()["txid"].is_string())
                             r.paidTxids[it.key()] = it.value()["txid"].get<std::string>();
+                        if (it.value().value("forfeited", false))
+                            r.forfeited[it.key()] =
+                                it.value().value("forfeitReason", std::string("unspecified"));
                     } else if (it.value().is_number_integer()) {
                         r.payouts[it.key()] = it.value().get<int64>();
                     }
@@ -1132,6 +1148,33 @@ static void MarkPoolRoundPaid(const std::string& id,
     PublishPoolRounds(snap);
 }
 
+// Publish the fact that a recipient was written off, so the round proof and
+// the ledger agree. The ledger erases the amount (it is not owed any more and
+// must not be reported as sent); the proof keeps the amount visible and marks
+// it forfeited with a reason, because a miner who was promised a share is
+// entitled to see what happened to it.
+static void MarkPoolRoundForfeited(const std::string& id,
+                                   const std::string& address,
+                                   const std::string& reason)
+{
+    VersionedBlob snap;
+    {
+        std::lock_guard<std::mutex> lk(gPoolRoundsMutex);
+        bool found = false;
+        for (PoolRoundProof& r : gPoolRounds) {
+            if (r.id != id)
+                continue;
+            r.forfeited[address] = reason;
+            found = true;
+            break;
+        }
+        if (!found)
+            return;
+        snap = SerializePoolRoundsLocked();
+    }
+    PublishPoolRounds(snap);
+}
+
 // Which wallet transaction, if any, already paid this?
 //
 // Two kinds of evidence, in order of quality.
@@ -1634,14 +1677,21 @@ static void RecordPayoutTxid(const PayingRecord& r, const std::string& txid)
         LogPrint("payout", "[payout] block %d fully settled\n", r.blockHeight);
 }
 
-// Write a recipient off. Only reached after the full retry budget against a
-// block that is still on the main chain with coin available to spend, i.e. a
-// wallet fault rather than a maturity or reorg question.
+// Write a recipient off. Only reached after a full budget of failures that
+// were NOT the wallet being locked or short of coin -- see the classification
+// in PayoutThreadFn -- against a block still on the main chain.
 //
 // The amount is removed from the entry rather than marked paid, so the money
-// is not reported as sent. Once nothing is left owed the entry retires.
-static void ForfeitPayoutRecipient(const PayingRecord& r)
+// is not reported as sent. Once nothing is left owed the entry retires. The
+// round proof is told separately, because dropping the obligation from the
+// ledger while pool_rounds.json goes on publishing the address as owed with
+// "txid": null leaves the pool's two records of the same round permanently
+// disagreeing.
+static void ForfeitPayoutRecipient(const PayingRecord& r, const std::string& reason)
 {
+    LogPrint("payout", "[payout] writing off %s owed to %s for block %d: %s\n",
+             FormatMoney(r.amount).c_str(), r.address.c_str(), r.blockHeight,
+             reason.c_str());
     VersionedBlob blob;
     {
         std::lock_guard<std::mutex> lk(gPayoutLedgerMutex);
@@ -1657,6 +1707,8 @@ static void ForfeitPayoutRecipient(const PayingRecord& r)
         blob = SerializePendingPayoutsLocked();
     }
     PublishPendingPayouts(blob);
+    MarkPoolRoundForfeited(PayoutIdForBlock(r.blockHeight, r.blockHash),
+                           r.address, reason);
 }
 
 // Payout execution thread: works through gPaying, calls SendMoney.
@@ -1766,17 +1818,51 @@ static void PayoutThreadFn(void*)
             // payment the pool fully intends to honour can hit GIVING UP
             // 24h later purely because the subsidy funding it had not
             // matured yet -- the miner is then silently never paid.
+            //
+            // The comparison includes the fee. It used to be against the
+            // amount alone, so a balance anywhere between amount and
+            // amount+fee sailed past this gate, failed in SendMoney, and was
+            // counted as a send FAILURE rather than a wait -- 1440 of those is
+            // 24 hours, and the recipient was erased with the money left in
+            // the operator's wallet and nothing on disk saying it was owed.
+            //
+            // nTransactionFee is the part of the fee that can be known before
+            // the transaction exists. The rest is GetMinFee(), a function of
+            // the serialised size and therefore of how many inputs SelectCoins
+            // ends up picking, which cannot be predicted from here -- so this
+            // gate deliberately does NOT try to reserve it. Refusing to attempt
+            // a payment the wallet could actually have made would be its own
+            // way of not paying a miner. That residual is handled after the
+            // fact, by classifying the failure below.
             int64 nSpendable = GetBalance();
-            if (nSpendable < r.amount) {
+            if (nSpendable < r.amount + nTransactionFee) {
                 r.deferrals++;
                 r.retryAfter = GetTime() + 60;
                 // Hourly, not every minute: a dry pool would otherwise
                 // write one line per payment per minute forever.
                 if (r.deferrals == 1 || r.deferrals % 60 == 0)
                     LogPrint("payout", "[payout] deferring %s to %s -- only %s"
-                             " spendable (waiting %d min so far)\n",
+                             " spendable, need %s with fee (waiting %d min so far)\n",
+                             FormatMoney(r.amount).c_str(), r.address.c_str(),
+                             FormatMoney(nSpendable).c_str(),
+                             FormatMoney(r.amount + nTransactionFee).c_str(),
+                             r.deferrals);
+                ReschedulePayment(r);
+                continue;
+            }
+
+            // Likewise a locked wallet: it cannot sign, so every attempt fails
+            // for a reason that has nothing to do with this payment being
+            // payable. Left to the retry budget, an operator who locks the
+            // wallet overnight comes back to erased debts.
+            if (IsWalletLocked()) {
+                r.deferrals++;
+                r.retryAfter = GetTime() + 60;
+                if (r.deferrals == 1 || r.deferrals % 60 == 0)
+                    LogPrint("payout", "[payout] deferring %s to %s -- wallet is"
+                             " locked (waiting %d min so far)\n",
                              FormatMoney(r.amount).c_str(), r.address.c_str(),
-                             FormatMoney(nSpendable).c_str(), r.deferrals);
+                             r.deferrals);
                 ReschedulePayment(r);
                 continue;
             }
@@ -1803,36 +1889,76 @@ static void PayoutThreadFn(void*)
                 LogPrint("payout", "[payout] paid %s to %s (tx %s)\n",
                          FormatMoney(r.amount).c_str(), r.address.c_str(),
                          wtx.GetHash().ToString().substr(0, 12).c_str());
-            } else if (r.retries < 1440) {
-                // Transient failures (reorg, locked wallet) must not silently
-                // burn the payment -- retry with backoff. Insufficient funds
-                // never lands here; it defers above.
-                r.retries++;
-                r.retryAfter = GetTime() + 60;
-                ReschedulePayment(r);
-                LogPrint("payout", "[payout] FAILED to pay %s to %s"
-                         " (%s spendable at the time) -- retry %d/1440 in 60s\n",
-                         FormatMoney(r.amount).c_str(), r.address.c_str(),
-                         FormatMoney(nSpendable).c_str(), r.retries);
             } else {
-                // Reaching here means 1440 attempts that were NOT
-                // insufficient-funds waits (those defer instead) against a
-                // block still on the main chain -- i.e. a real wallet fault,
-                // not a maturity or reorg question.
-                LogPrint("payout", "[payout] GIVING UP on %s to %s after %d"
-                         " retries -- block %d is still on-chain and %s was"
-                         " spendable, so this is a wallet fault, not maturity"
-                         " -- payment forfeited\n",
-                         FormatMoney(r.amount).c_str(), r.address.c_str(),
-                         r.retries, r.blockHeight,
-                         FormatMoney(nSpendable).c_str());
-                // Write the debt off in the ledger as well as dropping the
-                // in-flight record. Under the survives-until-fully-paid model
-                // the entry would otherwise never complete, and every later
-                // flush would explode it again -- a forfeit that silently
-                // becomes an infinite retry.
-                ForfeitPayoutRecipient(r);
-                ForgetPayment(r);
+                // Why did it fail? The answer decides whether this costs a
+                // retry, and the retry budget is what eventually erases a
+                // miner's money -- so guessing here is how a pool keeps coin
+                // it owes.
+                //
+                // Re-read the conditions rather than trusting the ones checked
+                // before the attempt: SendMoney takes cs_main and can block for
+                // a long time, so the balance and the lock state may both have
+                // moved underneath it.
+                //
+                // Short of coin covers the size-dependent part of the fee that
+                // the gate above deliberately does not reserve: if the balance
+                // no longer covers amount + configured fee + one fee tier, the
+                // failure is a funding shortfall, not a fault.
+                int64 nAfter    = GetBalance();
+                bool  fLocked   = IsWalletLocked();
+                bool  fUnfunded = nAfter < r.amount + nTransactionFee + CENT;
+
+                if (fLocked || fUnfunded) {
+                    // Conditions are temporarily wrong. Wait; do not spend the
+                    // budget that leads to a permanent write-off.
+                    r.deferrals++;
+                    r.retryAfter = GetTime() + 60;
+                    if (r.deferrals == 1 || r.deferrals % 60 == 0)
+                        LogPrint("payout", "[payout] deferring %s to %s -- send"
+                                 " failed with %s (%s spendable, %s needed with"
+                                 " fee headroom); waiting %d min so far\n",
+                                 FormatMoney(r.amount).c_str(), r.address.c_str(),
+                                 fLocked ? "the wallet locked" : "too little coin",
+                                 FormatMoney(nAfter).c_str(),
+                                 FormatMoney(r.amount + nTransactionFee + CENT).c_str(),
+                                 r.deferrals);
+                    ReschedulePayment(r);
+                } else if (r.retries < 1440) {
+                    // A genuine fault: funded, unlocked, block on the main
+                    // chain, and the wallet still would not build or accept
+                    // the transaction. Retry with backoff.
+                    r.retries++;
+                    r.retryAfter = GetTime() + 60;
+                    ReschedulePayment(r);
+                    LogPrint("payout", "[payout] FAILED to pay %s to %s"
+                             " (%s spendable, wallet unlocked) -- retry %d/1440"
+                             " in 60s\n",
+                             FormatMoney(r.amount).c_str(), r.address.c_str(),
+                             FormatMoney(nAfter).c_str(), r.retries);
+                } else {
+                    // 1440 failures that were each individually impossible to
+                    // explain by maturity, funding, the wallet being locked, or
+                    // a reorg. That is the only state in which this payment can
+                    // be called impossible rather than merely blocked.
+                    const std::string reason = strprintf(
+                        "wallet refused this payment %d times while funded"
+                        " (%s spendable), unlocked, and block %d on the main"
+                        " chain", r.retries, FormatMoney(nAfter).c_str(),
+                        r.blockHeight);
+                    LogPrint("payout", "[payout] GIVING UP on %s to %s: %s"
+                             " -- payment forfeited\n",
+                             FormatMoney(r.amount).c_str(), r.address.c_str(),
+                             reason.c_str());
+                    // Write the debt off in the ledger as well as dropping the
+                    // in-flight record. Under the survives-until-fully-paid
+                    // model the entry would otherwise never complete, and every
+                    // later flush would explode it again -- a forfeit that
+                    // silently becomes an infinite retry. The round proof is
+                    // told too, so the public record does not keep claiming the
+                    // address is owed.
+                    ForfeitPayoutRecipient(r, reason);
+                    ForgetPayment(r);
+                }
             }
         }
 
@@ -1942,10 +2068,19 @@ static void WritePoolStatusJsonLocked()
         size_t start = gPoolRounds.size() > 10 ? gPoolRounds.size() - 10 : 0;
         for (size_t i = start; i < gPoolRounds.size(); i++) {
             const PoolRoundProof& r = gPoolRounds[i];
+            // A forfeited recipient is not "unpaid and pending" -- the ledger
+            // has already dropped the obligation. Counting it as unpaid is
+            // what made the round proof and the ledger disagree for good.
             int unpaid = 0;
-            for (const auto& kv : r.payouts)
-                if (r.paidTxids.count(kv.first) == 0)
+            int forfeited = 0;
+            for (const auto& kv : r.payouts) {
+                if (r.paidTxids.count(kv.first))
+                    continue;
+                if (r.forfeited.count(kv.first))
+                    forfeited++;
+                else
                     unpaid++;
+            }
             j["recentRounds"].push_back({
                 {"id", r.id},
                 {"blockHeight", r.blockHeight},
@@ -1954,6 +2089,7 @@ static void WritePoolStatusJsonLocked()
                 {"totalShares", r.totalShares},
                 {"payouts", (int)r.payouts.size()},
                 {"unpaidPayouts", unpaid},
+                {"forfeitedPayouts", forfeited},
                 {"coinbaseValueBtf", FormatMoney(r.coinbaseValue)},
                 {"minerRewardBtf", FormatMoney(r.minerReward)}
             });