| Age | Commit message (Collapse) | Author |
|---|
|
std::thread::hardware_concurrency() reports the MACHINE's core count. Inside a
container that is the wrong number: with --cpuset-cpus 0-5 on a 12-core host it
still answers 12, and every caller that sized a pool from it oversubscribed 2x.
Measured on the pool box:
- the pool node logged "Mining with 11 thread(s) of 12 core(s)" while
confined to 6 CPUs
- a burst miner launched with /hashthreads=2 filled the ~2 GB RandomX
dataset with 12 threads, alongside the pool node in the same 8 GB container
AvailableCoreCount() asks sched_getaffinity what the cpuset actually permits,
which is the number all of these meant. It cannot make the dataset build slower:
the CPU budget is identical either way, so fewer threads on the same cores is
the same work without the thrash. Falls back to hardware_concurrency where the
call is unavailable, and never returns 0 because callers subtract from it.
Not the cause of the burst miner dying -- that was a corrupt Berkeley DB
environment in the worker datadir -- but it is why the failure landed on the
dataset build, and it is wrong on its own terms.
(cherry picked from commit b3bee16f3eef8982bff1174bd9d0a9c21473732c)
|
|
Upstream #109's message-stall rework, hand-merged so it keeps the progress
timing this tree already had.
The old path consumed the header, found the payload short, inserted the header
back at the front of vRecv, then Sleep(100) inside the shared message thread.
A peer that announces a large payload and drips bytes made every pass pay an
O(n) insert and a 100 ms stall -- one peer slowing the thread that serves all
of them. Now the header is read from a copy and only skipped once the whole
body has arrived, so there is nothing to put back and nothing to sleep for.
Kept OURS, not upstream's, on the timeout: upstream restarts its clock only
when the (size, command) pair changes, which drops a peer sending one big
message slowly but honestly. This times the STALL -- the clock restarts on any
progress -- which is the distinction that matters on a slow link.
Also from #109: a header could announce 256 MB while serialize.h refuses
anything over MAX_SIZE (32 MB), so the check admitted an allocation the parser
would reject anyway and burned a peer slot for two minutes doing it.
MAX_PROTOCOL_MESSAGE_SIZE ties the two together. 8x less a peer can make us
hold, and this node takes public Stratum.
And #103's other half: ~CNode closed its socket with a raw closesocket(),
invisible to the accounting, and left hSocket set afterwards -- which is how a
descriptor number that now belongs to somebody else gets closed a second time.
(cherry picked from commit 2f25e2e07985e9bc732cadfbb3c9ac233895a814)
|
|
CheckBlock() validated block size against MAX_SIZE, the serializer's generic
32 MB guard, not a block rule. MAX_BLOCK_SIZE did not exist in this tree at
all: Bitcoin's 1 MB cap arrived in 0.3.x, after the 0.1.0 this fork came from.
A miner could produce a 32 MB block and every node would download, verify and
store it. Upstream #111 (from #110 by mtg0001).
The cap is a tightening, so a node that does not upgrade is the permissive one.
Measured upstream over 5709 blocks: largest 34,290 bytes, median 216, so 1 MB
rejects nothing that exists.
Upstream shipped the validation half only. Both places that ASSEMBLE a block
still filled to MAX_SIZE/2 (16 MB), which was consistent against a 32 MB
validation cap and is inverted by a 1 MB one -- the node would solve a block
and then reject it itself. Fixed both:
main.cpp CreateNewBlock, the node's own miner
rpc.cpp RebuildJob, the POOL's job builder -- the one that matters here,
because every miner on the pool would be working a template the
network rejects, and this pool is ~90% of hashrate
Half the cap in both, preserving the original margin: each test runs against a
transaction that is already counted, so the loop can overshoot by one.
(cherry picked from commit a3be9f04903447fa2741aff4eb529fe45f916877)
|
|
From the first of two wedges captured on the pool. It is a different
proximate fault from the second one, and this is the part that turns a
single bad socket into a node-wide outage.
A failed select() zeroes both descriptor sets, so no peer is serviced on
that pass. That is correct as far as it goes. What was missing is that
nothing stopped the offending descriptor coming straight back:
- the invalid socket was only marked fDisconnect, and a node is not
removed from vNodes until it has drained or DISCONNECT_DRAIN_SECS has
elapsed -- and it cannot drain, because its socket is gone;
- the select set was rebuilt from vNodes with no check, so the same dead
descriptor went back in on the very next pass and failed select()
again.
Measured in the capture: 1,027 consecutive 'its socket is no longer valid'
passes across 61 seconds -- the full drain timeout -- one bad socket, and
for the whole of it the node read nothing from anybody. The peer that
caused it had 57 bytes unsent, so it took the slow path out.
Now the descriptor is forgotten as soon as getsockopt says it is not ours:
select() succeeds again on the next pass and every other peer is served
normally while the dead node ages out as before.
Deliberately forgotten rather than closed. getsockopt has just said this
descriptor is no longer ours; calling close() on it is precisely how an
unrelated socket gets closed instead, which is the fault fixed in e742d0b.
(cherry picked from commit f14c513b4e39d6203b73584f6dc01acbb5a9546e)
|
|
Found auditing the previous commit for the push. Timing from the first
rewind would drop a peer whose message is large and merely slow, which
would be a fault of our own making rather than one we fixed.
Restart the clock whenever the buffer has grown by even one byte since the
last look. A transfer still making progress is never dropped however long
it takes; only one that has stopped growing while the node goes on
expecting more is, which is the actual failure.
(cherry picked from commit 5dddb26bae02013ce87719b11953aadf0bfa2594)
|
|
THIS IS THE WEDGE, and the note asking for this fix has been in the file
since Bitcoin 0.1: 'need a mechanism to give up waiting for overlong
message size error'.
ProcessMessages reads a header, finds the body has not all arrived, pushes
the header back onto the front of vRecv and breaks. Correct for a message
still in flight. For one that never completes it is permanent: every later
pass re-reads the same header and breaks in the same place, so that peer is
never parsed again for the life of the process -- no blocks, no inv, not
even its version message.
Nothing rescues it:
- a header may claim up to 256 MB (0x10000000) and still pass
CMessageHeader::IsValid(), so a garbage or truncated header is accepted
and then waited on forever;
- nLastRecv is refreshed by any byte that arrives (net.cpp), whether or
not it parses, so the inactivity timeout that drops a silent peer never
fires on a peer that is merely unreadable;
- the socket stays ESTABLISHED and select() keeps offering it, so nothing
at the socket layer notices either.
It also explains the signature that made no sense. A peer stuck before its
version message is parsed keeps nStartingHeight == -1, so a node with a full
vNodes still reports GetPeerMedianHeight() == -1 -- which is exactly what
was observed, and exactly what disabled the pool's behind-the-network gate
and let it fork.
Evidence, from two wedges captured on the pool by the watchdog:
MESSAGE-BREAK 2 ran 2014 times in the deaf window against 607 in the healthy
window before it, while blocks received went to zero and the node kept
processing miner traffic normally throughout.
Now: a peer that announces a body and does not deliver it within two minutes
is dropped, and says so in the log unconditionally. Two minutes is far
beyond any legitimate header-to-body gap and well inside
BTF_RECV_TIMEOUT_SECS.
(cherry picked from commit ca1e99288e03ac7beb2b97cf615a3255b4635d95)
|
|
THE WEDGE.
CNode::Disconnect() closed hSocket and left the member holding the stale
number. ~CNode's only guard is 'hSocket != INVALID_SOCKET' and nothing in
the tree ever assigned that sentinel, so the destructor closed the same
number a second time.
The two closes are five minutes apart by design: ThreadSocketHandler2
disconnects a node, sets nReleaseTime to GetTime() + 5*60, and parks it in
vNodesDisconnected; GetRefCount() reports it busy until that expires, so
'delete pnode' -- and the second close -- cannot run before then.
Linux hands out the lowest free descriptor. Measured on the pool: 202 open
descriptors with a high-water mark of 216, so a freed number is reissued
within seconds -- to a Nostr relay TLS socket, a rendezvous socket, a
tunnel end, or a miner connection. Five minutes later the destructor
closes it. The victim gets no error and no log line. It simply stops.
That is the whole signature, including the part that never made sense:
.btf block delivery and Nostr descriptor lookup share no lock, no queue
and no thread, so they should not fail together. They share the file
descriptor table, and a stray close is drawn from all of it.
It also explains why #78 did not help and may have hurt. #78 drops stuck
nodes -- it fired 41 times in one window -- and every drop is another
Disconnect(), which is another delayed close armed against an innocent
socket.
Second bug, same symptom, different subsystem: the pooled relay sockets.
ResolveDescriptor returned the instant a descriptor verified, leaving the
relay's EOSE unread on a socket that goes straight back into g_relayPool,
so the next caller read our leftovers as its own answer -- permanently one
message out of step, and no code path anywhere inspected the subscription
id that would have caught it. Worse, BtfResolve set lease.fOk = true on
every failure path including a dead RecvText, labelling it 'exchange
completed, just no match'. fOk = false is the only thing that ever closes
a pooled socket or clears fConnected, so a relay connection that died was
pinned in the pool, reported healthy, and reused for the life of the
process. That is 'no Nostr descriptor found' for a descriptor that nak can
see, cleared only by a restart.
Now: read to EOSE before returning, match the subscription id, and keep a
pooled socket only when the relay actually finished the conversation.
(cherry picked from commit ed4640c306cc46dbdbd92cb6dc55e7528651269c)
|
|
The peer-height gate replaced a tip-age check that was wrong in both
directions. It is a strict improvement and it is still blind in the one
case that matters most.
When the node goes deaf -- peers still in vNodes, nothing arriving -- no
peer announces anything, GetPeerMedianHeight() returns -1, and the
fallback mines our best chain. That fallback is right for a node that has
not been told yet and wrong for a node that has stopped being able to
hear, and a height nobody sent cannot tell those apart. A wedged node is
exactly a node with no peer height information, so the gate was silent
for the whole of heights 3966-4085: 120 blocks, 5890.50 BTF voided.
Time can tell them apart. GetPeerHeightAgeSecs() measures how long since
any peer reported a height, counting from first use rather than the epoch
so a starting node does not refuse jobs before its first handshake. Past
20 minutes the pool stops issuing jobs and says so unconditionally in the
log. /poolmineblind overrides it: without an escape hatch a genuinely
isolated pool that is 90% of the hashrate could never restart its own
chain, which is the deadlock the tip-age check used to cause.
Also stop nStartingHeight from going stale. It was a snapshot taken once
during the version handshake and never touched again, so on a long-lived
connection the median is built from heights the peers had hours ago, and
the behind-the-network check drifts toward 'everyone is behind us' and
quietly stops firing. A peer that hands us a block is demonstrably at
least at that block's height.
(cherry picked from commit 0800c18faf703b4acde1b131d0b742d37a86d92b)
|
|
The sync gate compared the tip timestamp against 30 minutes. That cannot
distinguish "we are not synced" from "the network is slow", and it failed
in both directions on the same day.
A pool mining its own fork keeps its tip fresh, so the check passed for
the whole of heights 3966-4085 -- 120 consecutive blocks, all orphaned,
5890.50 BTF of payouts voided, and nothing in the log to say so. Then,
once the pool stopped finding blocks, the same check refused to issue
jobs against a tip that was the network's own best. The pool is ~90% of
the hashrate, so nobody else could move the chain either: stale tip, no
jobs; no jobs, stale tip.
Gate on the height peers announce in their version message instead.
Median, so one peer cannot stall us with an absurd claim, and a block of
slack to match WarnIfBehind. Peers predating the announcement report -1
and are ignored; with no reports at all we mine our best chain, which is
what the node does anyway without a pool attached.
(cherry picked from commit 4e76e34f37a82c0885f75db7dfb9e49db410171d)
|
|
The exec queue was memory-only, and FlushMaturePayouts erased a matured entry
from pending_payouts.json *before* handing it over. Any restart in that window
lost those payments with no record that anyone was owed anything. It was not
hypothetical: a container restart on 2026-07-30 silently dropped a backlog of
~1,156 BTF, and there was nothing left to reconstruct it from.
Re-queueing blindly would have double-paid whoever had already been sent to,
so the payment needs to be idempotent rather than merely persistent.
In-flight work is now a persisted list of per-payment records
(paying_payouts.json), one per (block, address) -- the granularity SendMoney
works at, and therefore the granularity at which a payment can be half-done
when the process dies. Records are written, via a temp file and rename so a
crash cannot truncate them, *before* the ledger entry is erased, so the debt
is always described by at least one of the two files. Each attempt updates the
record (retries, deferrals, backoff); settling it removes the record and
rewrites the file, which is what stops a restart from paying twice.
On startup ReconcilePayouts runs before the payout thread:
- a record whose payment the wallet already shows -- matched on the payout
comment plus the exact amount and destination -- is dropped, not resent.
SendMoney commits the transaction to the wallet before it returns, so a
crash immediately after sending is recoverable.
- a ledger entry that has already been exploded into records is dropped,
which is the crash-in-the-window case where both files describe one debt.
The wallet check is deliberately biased towards "already paid": skipping a
payment is visible in the log and recoverable, paying twice is neither.
Verified live: sampling paying_payouts.json during a payout burst caught 3
records / 18.56 BTF mid-flight, cleared once paid, and the paid lines now
carry the txid.
(cherry picked from commit d0248f2e65dbe745f2e9b1d09e1f53a4354d3640)
|
|
Two fixes.
uint256's default constructor is empty (uint256.h:514), so the blockHash
field added in 6963862 was uninitialised garbage on every PendingPayout
built without an explicit hash -- which is every entry loaded from a ledger
written before that commit. Those entries were meant to be grandfathered
past the orphan check on a zero hash; instead each carried a random value
that matched no block in mapBlockIndex, and the first maturity sweep after
deploy voided the whole loaded ledger: 100 entries, 4950 BTF. Only one
"was orphaned -- voiding" line reached the log because LogPrint collapsed
the other 99 as duplicates, so the destruction was nearly silent. Nothing
had been paid at that point and the file was restored from a pre-deploy
backup, so no obligation was lost. Assigning 0 explicitly is all that was
missing.
Separately, four of the twelve relays are dead. Probed all twelve from two
independent networks (a residential line and the pool's VPS) over v4 and v6,
with DNS confirmed against 1.1.1.1:
relay.current.fyi, relay.nostr.bg -- NXDOMAIN, no A and no AAAA. The
domains are gone. (A getaddrinfo call on the VPS reports an address for
them, but that is the VPS's own IPv6 handed back by its local resolver,
not a real host.)
relay.nostr.band, nostr.fmt.wiz.biz -- DNS resolves and TCP connects, but
the TLS handshake never completes, from either network. Whether they are
down or refusing non-browser clients, this node cannot use them.
The remaining eight all complete a WebSocket upgrade. A dead relay is not
free: each holds a connect/retry slot and delays discovery for no added
redundancy.
(cherry picked from commit 144bc96f150d23507d57d29cb339f177dcd6b092)
|
|
A 280-block reorg (heights 3075-3354) orphaned 292 of the pool's 707 blocks
and destroyed 14,600 BTF of coinbase. The payout ledger did not notice: it
records only matureAtHeight, and a reorg replaces the block at a height
without changing the height, so ~14.4k BTF of obligations outlived the coin
that was supposed to fund them. The pool then paid fork-era miners out of
real-chain coin until the wallet was empty -- 17,827 BTF paid against 17,900
BTF of matured supply, leaving under 1.1 BTF spendable and every subsequent
payment failing.
Three changes, all in the payout path:
- Record the funding block's hash and height, persist both, and void an
obligation whose block is no longer in the main chain. Checked both at
maturity and again in the exec queue, since an entry can sit there for
hours across retries and the reorg can land at any point in that window.
Entries loaded from a file written before this are grandfathered rather
than voided on a missing field.
- Stop spending the retry budget on maturity. SelectCoins failing for want
of spendable coin is not an error, but it consumed one of 1440 retries
every 60s, so a payment the pool fully intended to honour would hit
GIVING UP 24h later and the miner would silently never be paid. Those
attempts now defer without incrementing retries, logged hourly rather
than every minute. GIVING UP now means what it says: 1440 real failures
against a block still on-chain with the money available.
- Schedule at COINBASE_SPENDABLE_DEPTH instead of COINBASE_MATURITY. At
depth 100 a block's own coinbase is 20 blocks short of spendable, so its
payout had to come from unrelated mature coin and the pool carried a
permanent ~19-block float. The "96% of 934 payments went through at depth
100" note that justified the shorter wait was measured while this node was
alone on a private fork paying itself. The extra 20 blocks also put the
maturity gate past the reorg that caused all of the above.
(cherry picked from commit 696386220a3d2fac88cb8045931392f4909e0985)
|
|
SendMoney's "oversized transaction that requires a transaction fee of
0.00" is not an oversized transaction and there is no fee. That branch is
taken when nValue + nFeeRequired > GetBalance() -- an empty spendable
balance -- and the fee reads 0.00 because CreateTransaction failed before
setting one. 3,267 of these in debug.log sent me looking for a
transaction-size problem that does not exist. Report what is short, by how
much, and against what balance.
Name COINBASE_SPENDABLE_DEPTH in main.h and derive GetBlocksToMaturity
from it, because the distinction it captures is genuinely surprising:
CMerkleTx::GetCredit values a mined coinbase at zero until
COINBASE_MATURITY+20 deep, not COINBASE_MATURITY.
I first changed the payout gate to that deeper value, on the theory that
each payout was firing 20 blocks before its own coinbase could be spent.
Measuring it refuted that: across 934 payments, 96% succeeded at depth
100, the moment they were queued, which is only possible because a payout
is funded from whatever coin is already mature rather than from its own
block. The gate stays at COINBASE_MATURITY, with a comment recording the
measurement so the next person does not re-make the same change.
The real shortfall is structural and not addressed here: the pool pays out
~99% of each block's subsidy while only matured coinbases are spendable,
so it runs with almost no buffer, and any orphaned block it queued a
payout for drains that buffer permanently. The 24h retry loop absorbs this
-- 0 payouts forfeited -- but the buffer is what needs fixing.
(cherry picked from commit e28df01f90d621b0cd818f28d70a20b8ab5d1a8d)
|
|
Upstream #30 split rendezvous registration from waiting for a dial, so
RvServiceRegister now returns as soon as the relay lists the service and
no longer consumes the relay's 0x01 pairing byte. It updated its own
caller in net.cpp and left this one behind.
AcceptOneFn went straight from RvServiceRegister to BtfServiceWrap, so
BtfServiceWrap read that 0x01 as the first byte of the miner's 32-byte
ephemeral key and every byte after it was shifted by one. The X25519
handshake has no confirmation step, so it completed against a garbage
shared secret, the tunnel was handed to the Stratum event loop, and
nothing failed loudly: the pool logged a miner arriving and then dropped
it as "never authorized", while the miner logged a pool that closed on it
with gotSubscribeAck=0. Loopback miners were unaffected, which is exactly
what made it look like a relay problem.
Block in RvServiceWaitPaired before wrapping, and close and re-register if
the relay drops us first.
test_rvaccept.cpp covers both shapes against a real bitflashd on loopback:
the pre-fix order establishes a tunnel that carries no bytes, the fixed
order round-trips. Also corrects the btftunnel.h contract comment, which
still described the pre-#30 behaviour and is what the old code was written
against -- upstream's own test_relayproc.cpp, test_btftunnel.cpp and
test_btfrv.cpp are still on that stale contract and fail the same way.
(cherry picked from commit 3e3a36b2bf1e6ffb4861507089b743688619377f)
|
|
SSL_CTX_new leaves verification at SSL_VERIFY_NONE and loads no trust
store, and nothing here ever changed that. SNI was set, a certificate was
received, and it was then ignored: every wss:// relay connection this node
has ever made would have completed against an attacker's certificate.
Signed events limit the damage -- a MITM cannot forge an announcement
attributable to our key -- but it can drop the rendezvous registration so
miners never find the pool, censor subscriptions, or answer them with
arbitrary content, which is the input the allocation bounds exist to
contain.
Load the system trust store, require SSL_VERIFY_PEER, and bind the check
to the host we dialed with SSL_set1_host, since chain validation alone
would accept any certificate a trusted CA ever issued for any name. Floor
the protocol at TLS 1.2. Verification failures report the specific X509
reason rather than a generic handshake error, and a missing CA store is a
hard failure instead of a silent downgrade to trusting everything.
(cherry picked from commit 6020740e68463deb1cc3c5b477372fcd1a245adb)
|
|
The min-2-threads floor applied even when the user explicitly asked
for 1 thread, so /hashthreads=1 silently spawned 2.
(cherry picked from commit a67b9e84f4f7607d307081b4a9873f5d9e78cf8a)
|
|
Descriptor (38501) still published so miners can resolve the pool;
only the public pool listing event (38503) is suppressed.
(cherry picked from commit 17bbdae1ec97945c37c1d5b2486e5c01a4d7ad87)
|
|
A pool that just found its first blocks has zero mature balance until
100 blocks later; orphaned coinbases (observed: reorgs orphaned our
blocks 2651-2655) make that worse. 30 one-minute retries forfeited
payments that would have succeeded hours later.
(cherry picked from commit 49e80023ca23baee8a4b80cfb891e7a358dc39ba)
|
|
A pool started with /rvrelay=ONE still spawned an accept thread for every
Nostr-discovered relay (BtfAllRelays always merged g_discoveredRelays), and
each thread optimistically overwrote the advertised meeting node before
registering -- so a discovered Tailscale-only relay (a mesh host)
flapped the published .btf descriptor and public miners could not connect.
fBtfRelayPinned (set only when /rvrelay parses) makes BtfAllRelays return
exactly the pinned list. Also validate the /rvrelay value at parse time:
with the pin in place no discovered relay can mask a typo.
(cherry picked from commit 0a4d2b02fa5382c5e788b5f0e2f8a844695dec47)
|
|
- RebuildJob: no jobs with zero peers or a tip older than 30min;
mining unsynced produced orphaned blocks and payouts that can
never be paid (observed in production: 8 orphan 'found blocks')
- payout exec: on SendMoney failure requeue with 60s backoff, up to
30 attempts, instead of silently forfeiting the payment
- nostr: add oxtr.dev, nos.social, fmt.wiz.biz, current.fyi, nostr.bg
(12 relays total) for announcement redundancy
(cherry picked from commit f1986cd7e4a6f69469a2256eb96ff03f3f146792)
|
|
|