mine: hash on every core instead of one (#52)

Satoshi left a note above ThreadBitcoinMiner in 2009:

    //// todo: start one thread per processor,
    ////       use getenv("NUMBER_OF_PROCESSORS")

Nobody did, and it stopped being a nicety when the proof of work
became RandomX. SHA-256d on a 2009 desktop lost little to a
single thread. RandomX is built to be fed by every core at once,
so one thread on a 32-core machine mines at a thirty-second of
what the box can do -- and the only outward sign is a CPU that
looks suspiciously idle. A user reported exactly that: 30 minutes
of "Mining", no blocks, Bitflash sitting at 3.3% CPU while the
2 GB dataset was allocated and fast mode was active. Nothing was
broken. It was hashing on one core.

  bitflash -gen                     # every core but one
  bitflash -gen -genproclimit=8     # exactly eight

Automatic leaves one core to the rest of the node -- accepting
blocks, talking to peers, drawing the window -- because a node
that mines beautifully while falling behind the chain has won
nothing. /genproclimit overrides it, and a count several times
the core count is refused rather than quietly thrashing.

The wrapper was already built for this: RandomXCreateMinerVM says
"own VM, not shared", and the ~2 GB dataset is one allocation
every thread reads. Only the 2 MB scratchpad multiplies, so
sixteen threads cost about 32 MB more than one. The startup line
says so, because "why is it using 2 GB" is the first question
anyone asks.

Two things had to be right for this to be safe:

  - Shutdown. Miners share slot 3 of vfThreadRunning and StopNode
    waits for every slot before DBFlush tears the database down.
    With several, the first one out announced that everyone had
    finished while the rest were still calling ProcessBlock, so
    they are counted now and only the last one clears the flag.
    The three CheckForShutdown(3) calls inside the miner became
    plain returns: _endthread() does not unwind on Windows, so
    the bookkeeping would have been skipped there.

  - Duplicated work. Each thread makes its own key, and the
    coinbase pays it, so every thread has a different merkle root
    and searches a different space. Block assembly was already
    under cs_main and cs_mapTransactions; AddKey and ProcessBlock
    take their own locks.

Verified on both platforms: four distinct thread ids reach the
hashing loop and all report back on shutdown, which exits 0 with
DBFlush running. Auto picks cores-1 (1 on a 2-core box, correctly
reported). Speedup itself needs a many-core machine to show and
has not been measured here -- the claim in this message is about
threads doing work, not a benchmark.

Not touched: RandomX's memory. RANDOMX_DATASET_BASE_SIZE and
SCRATCHPAD_L3 are part of the hash function, so a knob for "use
more RAM" would either do nothing or fork the chain. Threads are
the only real dial.

6 files changed, 131 insertions, 16 deletions

diff --git a/src/gui.cpp b/src/gui.cpp
index fc6f1e1..825202b 100644
--- a/src/gui.cpp
+++ b/src/gui.cpp
@@ -32,6 +32,7 @@ int64  GetBalance();
 string PubKeyToAddress(const std::vector<unsigned char>& vchPubKey);
 bool   AddressToHash160(const std::string& str, uint160& hash160Ret);
 void   ThreadBitcoinMiner(void*);
+int    StartMinerThreads();
 void   ThreadRPCServer(void*);
 
 // ---------------------------------------------------------------------------
@@ -762,7 +763,7 @@ static void DrawOptionsDialog()
                 fGenerateBitcoins = 0;
             } else {
                 fGenerateBitcoins = 1;
-                if (!vfThreadRunning[3]) _beginthread(ThreadBitcoinMiner, 0, NULL);
+                if (!vfThreadRunning[3]) StartMinerThreads();
             }
 
             if (nMineMode == MINE_OPERATOR) {
diff --git a/src/main.cpp b/src/main.cpp
index c882b0b..d4fc6d0 100644
--- a/src/main.cpp
+++ b/src/main.cpp
@@ -72,6 +72,13 @@ bool fSoloMineTest = false; // /solomine: mine without requiring a peer (local t
 
 // Settings
 int fGenerateBitcoins;
+// How many threads to hash with. 0 means "decide from the hardware" -- see
+// MinerThreadCount(). This was effectively 1 forever, because Bitcoin 0.1.0
+// started exactly one miner thread and nothing here ever changed that. It made
+// sense for SHA-256d in 2009; RandomX is built to be fed by every core at once,
+// so a 32-core machine was mining at a thirty-second of its capacity and the
+// only visible symptom was a suspiciously idle CPU.
+int nMinerThreads = 0;
 int64 nTransactionFee = 0;
 CAddress addrIncoming;
 int    nMineMode        = MINE_RELAY;
@@ -119,6 +126,26 @@ void GetParticipantMiningStats(uint64& sharesSent, uint64& sharesAccepted, doubl
 // mapKeys
 //
 
+// Threads to hash with, resolving the automatic case.
+//
+// Automatic leaves one core to the rest of the node -- accepting blocks,
+// talking to peers, serving the GUI -- because a machine that mines perfectly
+// while falling behind the chain has won nothing. Anyone who disagrees can say
+// so with /genproclimit and get exactly what they ask for.
+//
+// Memory is not the reason to hold back: the ~2 GB RandomX dataset is shared by
+// every thread, and each one adds only its own 2 MB scratchpad. Sixteen threads
+// cost about 32 MB more than one.
+int MinerThreadCount()
+{
+    if (nMinerThreads > 0)
+        return nMinerThreads;
+    unsigned int nCores = std::thread::hardware_concurrency();
+    if (nCores <= 1)
+        return 1;
+    return (int)(nCores - 1);
+}
+
 bool AddKey(const CKey& key)
 {
     CRITICAL_BLOCK(cs_mapKeys)
@@ -2769,7 +2796,7 @@ static bool PoolParticipantMiner()
     return true;
 }
 
-bool BitcoinMiner()
+bool BitcoinMiner(int nThreadId)
 {
     // Relay mode: never mines, just relays/syncs. (Shouldn't normally get here
     // since the GUI hides Start Mining for Relay, but guard it regardless.)
@@ -2809,11 +2836,17 @@ bool BitcoinMiner()
     while (fGenerateBitcoins)
     {
         Sleep(50);
-        CheckForShutdown(3);
+        // Return instead of _endthread()ing. With more than one miner thread the
+        // bookkeeping in ThreadBitcoinMiner has to run on the way out, and
+        // _endthread() does not unwind on Windows, so a normal return is the
+        // only exit that works the same on both platforms.
+        if (fShutdown)
+            return true;
         while (vNodes.empty() && !fSoloMineTest)
         {
             Sleep(1000);
-            CheckForShutdown(3);
+            if (fShutdown)
+                return true;
         }
 
         unsigned int nTransactionsUpdatedLast = nTransactionsUpdated;
@@ -2908,8 +2941,11 @@ bool BitcoinMiner()
             Sleep(1000);
             continue;
         }
-        LogPrint("net", "BitcoinMiner: mining with RandomX (%s)\n",
-               RandomXFastReady() ? "fast 2GB" : "light 256MB");
+        // The thread number is in the line on purpose: every miner would
+        // otherwise print the same text, and the dedup filter would fold them
+        // into one, leaving no way to tell four threads from one.
+        LogPrint("net", "BitcoinMiner: thread %d hashing with RandomX (%s)\n",
+               nThreadId, RandomXFastReady() ? "fast 2GB" : "light 256MB");
 
         unsigned int nStart = GetTime();
         uint256 hashTarget = CBigNum().SetCompact(pblock->nBits).getuint256();
@@ -2950,7 +2986,8 @@ bool BitcoinMiner()
             // ~256 hashes is already plenty of time)
             if ((++pblock->nNonce & 0xff) == 0)
             {
-                CheckForShutdown(3);
+                if (fShutdown)
+                    break;          // falls through to RandomXDestroyMinerVM below
                 if (pblock->nNonce == 0)
                     break;
                 if (pindexPrev != pindexBest)
@@ -2963,6 +3000,8 @@ bool BitcoinMiner()
             }
         }
         RandomXDestroyMinerVM(rxvm);
+        if (fShutdown)
+            break;
     }
 
     return true;
diff --git a/src/main.h b/src/main.h
index 7e5bdf2..750d3f1 100644
--- a/src/main.h
+++ b/src/main.h
@@ -73,6 +73,10 @@ extern string strPoolName;        // operator: announced pool name
 extern string strPoolDashboardUrl; // operator: optional dashboard URL
 extern double dPoolFeePercent;     // operator: announced fee percent
 extern bool fSoloMineTest; // /solomine: mine without requiring a peer (local test)
+// Threads to hash with; 0 = decide from the hardware. Set by /genproclimit.
+extern int nMinerThreads;
+// Resolves the automatic case into a real count.
+int MinerThreadCount();
 extern bool gPoolServerRunning;    // GUI/startup state
 extern volatile bool gPoolRunning; // runtime pool server loop flag (rpc.cpp)
 
@@ -117,7 +121,9 @@ void PrintBlockTree();
 void AddOrphanTx(const CDataStream& vMsg);
 void EraseOrphanTx(uint256 hash);
 void LimitOrphanTx(unsigned int nMaxOrphans);
-bool BitcoinMiner();
+// nThreadId is 1..N, only for logging: it makes each miner's lines
+// distinguishable, which the dedup filter would otherwise collapse into one.
+bool BitcoinMiner(int nThreadId = 1);
 void ThreadRPCServer(void* parg);  // rpc.cpp -- .btf pool server
 void GetParticipantMiningStats(uint64& sharesSent, uint64& sharesAccepted, double& hashRate);
 void SetParticipantMiningStatus(const std::string& status);
diff --git a/src/main_gui.cpp b/src/main_gui.cpp
index 2484592..852ad4d 100644
--- a/src/main_gui.cpp
+++ b/src/main_gui.cpp
@@ -1,6 +1,7 @@
 // Bitflash entry point -- starts node threads then runs GUI (or headless).
 
 #include "headers_core.h"
+#include <thread>          // hardware_concurrency, to sanity-check /genproclimit
 #ifndef _WIN32
 #include <csignal>
 #endif
@@ -52,6 +53,7 @@ static void PrintUsage()
     printf("  /operator\n");
     printf("  /participant=POOL_BTF_ADDRESS\n");
     printf("  /solomine\n");
+    printf("  /genproclimit=N            (mining threads; 0 or absent = every core but one)\n");
     printf("\n");
     printf("Pool operator announcement:\n");
     printf("  /poolname=NAME\n");
@@ -99,6 +101,26 @@ static void ParseStartupArguments(int argc, char* argv[])
     if (arg(argc,argv,"/solomine") || arg(argc,argv,"-solomine"))
         fSoloMineTest = true;
 
+    // /genproclimit=N -- threads to hash with. 0 or absent means automatic,
+    // which is every core but one. Named after Bitcoin's own option so it reads
+    // familiarly to anyone who has run one of these before.
+    string strProcLimit = argval2(argc, argv, "/genproclimit", "-genproclimit");
+    if (!strProcLimit.empty())
+    {
+        int n = atoi(strProcLimit.c_str());
+        unsigned int nCores = std::thread::hardware_concurrency();
+        if (n < 0)
+            fprintf(stderr, "Ignoring /genproclimit=%s: not a thread count\n",
+                    strProcLimit.c_str());
+        else if (nCores > 0 && n > (int)nCores * 4)
+            // Well past diminishing returns, and every thread still costs a
+            // scratchpad. Refuse rather than quietly thrash the machine.
+            fprintf(stderr, "Ignoring /genproclimit=%d: this machine has %u core(s)\n",
+                    n, nCores);
+        else
+            nMinerThreads = n;   // 0 stays automatic
+    }
+
     if (arg(argc,argv,"/operator") || arg(argc,argv,"-operator"))
         nMineMode = MINE_OPERATOR;
 
@@ -289,8 +311,7 @@ int main(int argc, char* argv[])
             printf("Error: _beginthread(ThreadRPCServer) failed\n");
     }
     if (fGenerateBitcoins)
-        if (_beginthread(ThreadBitcoinMiner, 0, NULL) == (uintptr_t)-1)
-            printf("Error: _beginthread(ThreadBitcoinMiner) failed\n");
+        StartMinerThreads();
 
 #ifdef BITFLASH_NO_GUI
     // Nothing else this binary can do; /nogui is accepted and redundant.
diff --git a/src/net.cpp b/src/net.cpp
index 2581578..9ee056f 100644
--- a/src/net.cpp
+++ b/src/net.cpp
@@ -10,6 +10,7 @@
 #undef snprintf
 #endif
 #include "headers.h"
+#include <thread>          // hardware_concurrency, for the miner thread count
 #ifdef _WIN32
 #include <winsock2.h>
 #endif
@@ -1224,18 +1225,63 @@ void ThreadMessageHandler2(void* parg)
 
 
 
-//// todo: start one thread per processor, use getenv("NUMBER_OF_PROCESSORS")
+// Satoshi's "todo: start one thread per processor" stood here since 2009 and is
+// what MinerThreadCount() and StartMinerThreads() below finally do -- by way of
+// std::thread::hardware_concurrency rather than NUMBER_OF_PROCESSORS, which only
+// ever existed on Windows.
+//
+// Miners share slot 3 of vfThreadRunning, and StopNode waits for every slot to
+// go false before the node tears the database down. With one miner that was the
+// same thing as "the miner finished"; with several, the first one out would
+// announce that everybody had finished while the rest were still hashing and
+// still calling ProcessBlock. So count them, and let only the last one leaving
+// clear the flag.
+static CCriticalSection cs_nMinersRunning;
+static int nMinersRunning = 0;
+
 void ThreadBitcoinMiner(void* parg)
 {
-    vfThreadRunning[3] = true;
-    CheckForShutdown(3);
+    int nThreadId = (int)(intptr_t)parg;
+    CRITICAL_BLOCK(cs_nMinersRunning)
+    {
+        nMinersRunning++;
+        vfThreadRunning[3] = true;
+    }
     try
     {
-        bool fRet = BitcoinMiner();
-        printf("BitcoinMiner returned %s\n\n\n", fRet ? "true" : "false");
+        bool fRet = BitcoinMiner(nThreadId);
+        printf("BitcoinMiner thread %d returned %s\n", nThreadId, fRet ? "true" : "false");
     }
     CATCH_PRINT_EXCEPTION("BitcoinMiner()")
-    vfThreadRunning[3] = false;
+    CRITICAL_BLOCK(cs_nMinersRunning)
+    {
+        if (--nMinersRunning <= 0)
+        {
+            nMinersRunning = 0;
+            vfThreadRunning[3] = false;
+        }
+    }
+}
+
+// Start the miners. Returns how many actually got off the ground.
+int StartMinerThreads()
+{
+    int nThreads = MinerThreadCount();
+    int nStarted = 0;
+    for (int i = 0; i < nThreads; i++)
+    {
+        if (_beginthread(ThreadBitcoinMiner, 0, (void*)(intptr_t)(i + 1)) == (uintptr_t)-1)
+            printf("Error: _beginthread(ThreadBitcoinMiner) failed on thread %d\n", i + 1);
+        else
+            nStarted++;
+    }
+    // The dataset is one allocation shared by all of them; only the 2 MB
+    // scratchpads multiply. Printed because "why is it using 2 GB" is the first
+    // question anyone asks.
+    printf("Mining with %d thread(s) of %u core(s) -- about %d MB "
+           "(~2080 MB shared RandomX dataset + ~2 MB per thread)\n",
+           nStarted, std::thread::hardware_concurrency(), 2080 + 2 * nStarted);
+    return nStarted;
 }
 
 
diff --git a/src/net.h b/src/net.h
index 4379d04..11c023e 100644
--- a/src/net.h
+++ b/src/net.h
@@ -58,6 +58,8 @@ int  BtfPexAccept(const std::vector<std::string>& vDesc);
 void AbandonRequests(void (*fn)(void*, CDataStream&), void* param1);
 bool AnySubscribed(unsigned int nChannel);
 void ThreadBitcoinMiner(void* parg);
+// Starts MinerThreadCount() miners; returns how many came up.
+int  StartMinerThreads();
 bool StartNode(string& strError=REF(string()));
 bool StopNode();
 void CheckForShutdown(int n);