Merge upstream v1.2.20 into the fork

Merges upstream tag v1.2.20 (6aa3dade) into our 1.2.19+sec tree. The merge
base is bc3de30's descendant 9e39a50 -- upstream's own v1.2.19 -- so this is
an ordinary three-way merge, not a replay. Five conflict blocks in four
files, all resolved below.

No consensus change. CheckBlock, ConnectBlock, DisconnectBlock, ConnectInputs,
AcceptBlock, AcceptTransaction, GetNextWorkRequired and GetBlockValue hash
identical to v1.2.19; script.cpp and script.h are byte-identical on both
sides. main.cpp is +3 lines and main.h +2, all wallet-backend plumbing.
rpc.cpp is untouched, so the payout ledger arrives unchanged.

Makefile, VERSION. OURS, renumbered. 1.2.20+sec, for the same reason 1.2.19
carried +sec: this tree is not the upstream release of that number and the
string names every release artifact.

Makefile, AppImage recipe. OURS for the AppDir path, THEIRS for the bundled
library. Upstream's hunk reverts the icon paths to a hardcoded
/tmp/bitflash.AppDir, which is the local-user pre-placement hazard the fork
moved out of /tmp in the first place; taking their side wholesale would have
reintroduced it silently, inside a conflict about something else. The AppDir
stays $(APPDIR), and libsqlite3.so.0 joins the bundled list, which the new
backend genuinely needs.

src/Makefile, src/makefile.mingw, HEADERS. BOTH. Ours drops market.h,
theirs adds wallet_sqlite.h; the resolution does both. NODE_OBJS auto-merged
correctly in both files -- obj/wallet_sqlite.o present, obj/market.o still
gone -- and was verified rather than assumed, because that line has trapped
this fork before.

src/selftest.cpp, known-tests list. BOTH, union. Upstream adds eight
wallet-storage tests, we keep payout-ledger. The dispatch chain itself
auto-merged and still routes payout-ledger to RunPayoutLedgerSelfTest.

Hardening verified intact after merge: the TLS 1.2 floor in nostr.cpp, the
/rvrelay pin, opt-in managed Tor, strict sig3-on-onion descriptor
verification, and the marketplace removal. Upstream shipped no build logs or
__pycache__ this time; the three added files are docs/storage.md and
src/wallet_sqlite.{cpp,h}.

The SQLite backend this release introduces is NOT enabled here. An existing
wallet.dat resolves to Berkeley DB on every branch of the new selection
logic, and the conversion wizard is GUI-only and not compiled into the
headless node. The review findings that gate ever turning it on -- the
non-atomic backend marker, the connection-wide WriteHDMaster transaction,
and the encrypt-swap crash window -- are recorded on the tracker, not fixed
here.

1 files changed, 297 insertions, 9 deletions

diff --git a/src/main_gui.cpp b/src/main_gui.cpp
index 211e1e2..2743429 100644
--- a/src/main_gui.cpp
+++ b/src/main_gui.cpp
@@ -113,8 +113,12 @@ static void PrintUsage()
     printf("  /debug\n");
     printf("  /gen\n");
     printf("  /nogui or /daemon\n");
-    printf("  /selftest=wallet-keypool, wallet-hd, wallet-format, wallet-crypto, wallet-encrypt,\n");
-    printf("            net-message, consensus-limits, pool-stratum, parse-money, socks5-proxy, or managed-tor\n");
+    printf("  /selftest=wallet-keypool, wallet-hd, wallet-format, wallet-storage-sanity,\n");
+    printf("            db-env-reopen, wallet-sqlite, wallet-sqlite-migration,\n");
+    printf("            wallet-crypto, wallet-encrypt, wallet-portability,\n");
+    printf("            net-message,\n");
+    printf("            consensus-limits, pool-stratum,\n");
+    printf("            parse-money, socks5-proxy, or managed-tor\n");
     printf("\n");
     printf("Mining mode:\n");
     printf("  /operator\n");
@@ -175,6 +179,23 @@ static void PrintUsage()
     printf("  /recoveryaudit             (show how much spendable balance is\n");
     printf("                              covered by the recovery phrase;\n");
     printf("                              exits 2 when wallet.dat is still needed)\n");
+    printf("  /walletstorageaudit        (count wallet.dat record types without\n");
+    printf("                              printing wallet values, then exit)\n");
+    printf("  /walletstorageauditjson=FILE\n");
+    printf("                              (write the same audit as deterministic JSON)\n");
+    printf("  /walletstoragecheck        (fail closed on inconsistent or unsafe\n");
+    printf("                              wallet.dat storage state)\n");
+    printf("  /walletsqliteexport=FILE   (copy raw wallet.dat records into an\n");
+    printf("                              SQLite store, then exit)\n");
+    printf("  /walletsqliteverify=FILE   (compare wallet.dat with a SQLite export,\n");
+    printf("                              printing counts only, then exit)\n");
+    printf("  /walletsqliterestore=FILE  (rebuild wallet.dat from a SQLite export in\n");
+    printf("                              an empty data directory, then exit)\n");
+    printf("  /walletsqliteloadcheck=FILE\n");
+    printf("                              (parse a SQLite export like the wallet loader)\n");
+    printf("  /walletsqlite=FILE         (stage-load a SQLite wallet export, then exit)\n");
+    printf("  /walletbackend=sqlite      (opt-in: run the node on <datadir>/wallet.sqlite\n");
+    printf("                              instead of wallet.dat; export one first)\n");
     printf("  /rescan                    (walk the chain for coins this wallet owns\n");
     printf("                              but never recorded, then exit)\n");
     printf("\n");
@@ -530,10 +551,119 @@ int main(int argc, char* argv[])
 
     ParseStartupArguments(argc, argv);
 
+#ifdef _WIN32
+    // Set by the GUI "Restart Now" button: give the previous instance a moment
+    // to release the network port and the block-index files before this one
+    // opens them, so a relaunch does not race the process it is replacing.
+    if (arg(argc, argv, "/restartwait") || arg(argc, argv, "-restartwait"))
+        Sleep(2500);
+#endif
+
+    // Hidden self-test helper. It must run before LoadWallet(): several
+    // storage-sanity scenarios deliberately make wallet.dat unsafe to load.
+    string strSelfTestMutateWallet =
+        argval2(argc, argv, "/selftestmutatewallet", "-selftestmutatewallet");
+    if (!strSelfTestMutateWallet.empty())
+        return RunSelfTestMutateWallet(strSelfTestMutateWallet);
+
     string strSelfTest = argval2(argc, argv, "/selftest", "-selftest");
     if (!strSelfTest.empty())
         return RunSelfTest(strSelfTest);
 
+    // Storage diagnostics must run before LoadWallet(). Their job is to
+    // explain wallet.dat states that the normal wallet loader may refuse.
+    string strWalletStorageAuditJson =
+        argval2(argc, argv, "/walletstorageauditjson", "-walletstorageauditjson");
+    bool fDiagWantsAudit = arg(argc,argv,"/walletstorageaudit") ||
+                           arg(argc,argv,"-walletstorageaudit") ||
+                           !strWalletStorageAuditJson.empty();
+    bool fDiagWantsCheck = arg(argc,argv,"/walletstoragecheck") ||
+                           arg(argc,argv,"-walletstoragecheck");
+
+    // These diagnostics normally read wallet.dat. Under -walletbackend=sqlite
+    // the live wallet is wallet.sqlite, so open it read-only as the active
+    // store and let the same scan walk the right file. This runs before the
+    // normal backend resolution further down, so validate the value here too.
+    string strDiagBackend = argval2(argc, argv, "/walletbackend", "-walletbackend");
+    if ((fDiagWantsAudit || fDiagWantsCheck) && !strDiagBackend.empty() &&
+        strDiagBackend != "sqlite" && strDiagBackend != "bdb")
+    {
+        AttachTerminal();
+        fprintf(stderr, "Unknown wallet backend '%s'. Use -walletbackend=sqlite "
+                        "or -walletbackend=bdb.\n", strDiagBackend.c_str());
+        return 1;
+    }
+    bool fDiagSQLite = (strDiagBackend == "sqlite") &&
+                       (fDiagWantsAudit || fDiagWantsCheck);
+    if (fDiagSQLite)
+    {
+        AttachTerminal();
+        string strSQLitePath = GetAppDir() + "/wallet.sqlite";
+        string strOpenErr;
+        if (!WalletSQLiteRuntimeOpenReadOnly(strSQLitePath, strOpenErr))
+        {
+            fprintf(stderr, "Cannot open the SQLite wallet for diagnostics: %s\n",
+                    strOpenErr.c_str());
+            return 1;
+        }
+    }
+
+    if (fDiagWantsAudit)
+    {
+        int nRet = CmdWalletStorageAudit(strWalletStorageAuditJson);
+        if (fDiagSQLite)
+            WalletSQLiteRuntimeClose();
+        else
+            DBFlush(true);
+        return nRet;
+    }
+
+    if (fDiagWantsCheck)
+    {
+        int nRet = CmdWalletStorageCheck();
+        if (fDiagSQLite)
+            WalletSQLiteRuntimeClose();
+        else
+            DBFlush(true);
+        return nRet;
+    }
+
+    string strWalletSQLiteExport =
+        argval2(argc, argv, "/walletsqliteexport", "-walletsqliteexport");
+    if (!strWalletSQLiteExport.empty())
+    {
+        int nRet = CmdWalletSQLiteExport(strWalletSQLiteExport);
+        DBFlush(true);
+        return nRet;
+    }
+
+    string strWalletSQLiteVerify =
+        argval2(argc, argv, "/walletsqliteverify", "-walletsqliteverify");
+    if (!strWalletSQLiteVerify.empty())
+    {
+        int nRet = CmdWalletSQLiteVerify(strWalletSQLiteVerify);
+        DBFlush(true);
+        return nRet;
+    }
+
+    string strWalletSQLiteRestore =
+        argval2(argc, argv, "/walletsqliterestore", "-walletsqliterestore");
+    if (!strWalletSQLiteRestore.empty())
+    {
+        int nRet = CmdWalletSQLiteRestore(strWalletSQLiteRestore);
+        DBFlush(true);
+        return nRet;
+    }
+
+    string strWalletSQLiteLoadCheck =
+        argval2(argc, argv, "/walletsqliteloadcheck", "-walletsqliteloadcheck");
+    if (!strWalletSQLiteLoadCheck.empty())
+    {
+        int nRet = CmdWalletSQLiteLoadCheck(strWalletSQLiteLoadCheck);
+        DBFlush(true);
+        return nRet;
+    }
+
     // Berkeley DB reports failure by throwing, and CDB's constructor lets it
     // through. Nothing on this path caught anything, so an unreadable
     // blkindex.dat or wallet.dat -- one written by another platform's Berkeley
@@ -578,18 +708,171 @@ int main(int argc, char* argv[])
     printf("Loading wallet...\n");
     try
     {
-        if (!LoadWallet())
+        string strWalletBackend =
+            argval2(argc, argv, "/walletbackend", "-walletbackend");
+        string strWalletSQLite =
+            argval2(argc, argv, "/walletsqlite", "-walletsqlite");
+
+        // A misspelled backend must not fall through to Berkeley DB silently --
+        // someone who typed -walletbackend=sqlit meant to run on SQLite, and
+        // opening wallet.dat instead of telling them is the wrong surprise.
+        if (!strWalletBackend.empty() &&
+            strWalletBackend != "sqlite" && strWalletBackend != "bdb")
         {
-            // LoadWallet() explains itself into strWalletLoadError when it
-            // knows why -- an unsupported wallet format, for one, which is the
-            // case this whole path exists to make visible.
-            FatalStartupError(fHeadlessStartup, "Cannot open wallet.dat.",
-                              strWalletLoadError);
+            AttachTerminal();
+            FatalStartupError(fHeadlessStartup,
+                strprintf("Unknown wallet backend '%s'.", strWalletBackend.c_str()),
+                "Use -walletbackend=sqlite for the SQLite backend, or "
+                "-walletbackend=bdb for the Berkeley DB wallet.dat.");
             return 1;
         }
 
+        if (strWalletBackend == "sqlite")
+        {
+            // Explicit opt-in via the flag. The node runs on
+            // <datadir>/wallet.sqlite and never opens wallet.dat, but the user
+            // has to have exported one first -- there is no silent migration,
+            // and wallet.dat is left
+            // exactly where it is so a plain restart goes back to it.
+            AttachTerminal();
+            string strSQLitePath = GetAppDir() + "/wallet.sqlite";
+            if (!FileExists(strSQLitePath.c_str()))
+            {
+                string strGuide = strprintf(
+                    "No SQLite wallet was found at\n"
+                    "  %s\n\n"
+                    "The SQLite wallet backend is opt-in and needs an exported\n"
+                    "wallet first. Your Berkeley DB wallet.dat is not touched by\n"
+                    "any of this.\n\n"
+                    "  Step 1  Export your existing wallet to SQLite:\n"
+                    "            bitflash -walletsqliteexport=\"%s\"\n"
+                    "  Step 2  Start again with the SQLite backend:\n"
+                    "            bitflash -walletbackend=sqlite\n\n"
+                    "To go back to Berkeley DB at any time, just start without\n"
+                    "-walletbackend=sqlite. wallet.dat is still your wallet.",
+                    strSQLitePath.c_str(), strSQLitePath.c_str());
+                FatalStartupError(fHeadlessStartup,
+                                  "The SQLite wallet backend needs an exported wallet first.",
+                                  strGuide);
+                return 1;
+            }
+
+            if (!LoadWalletFromSQLiteRuntime(strSQLitePath))
+            {
+                FatalStartupError(fHeadlessStartup,
+                                  "Cannot open the SQLite wallet backend.",
+                                  strWalletLoadError);
+                return 1;
+            }
+
+            fprintf(stderr,
+                    "Wallet backend: SQLite\n"
+                    "  %s\n"
+                    "Your Berkeley DB wallet.dat is left untouched. Restart without\n"
+                    "-walletbackend=sqlite to go back to it.\n",
+                    strSQLitePath.c_str());
+            fflush(stderr);
+            // Fall through and run the node normally: every CWalletDB read,
+            // write, and erase now routes to this SQLite file.
+        }
+        else if (!strWalletSQLite.empty())
+        {
+            AttachTerminal();
+            if (!LoadWalletFromSQLite(strWalletSQLite))
+            {
+                FatalStartupError(fHeadlessStartup,
+                                  "Cannot open SQLite wallet export.",
+                                  strWalletLoadError);
+                return 1;
+            }
+
+            fprintf(stderr, "SQLite wallet loaded: read-only staging backend\n");
+            fprintf(stderr, "SQLite runtime writes are not enabled yet; exiting before "
+                            "network, mining, GUI, or wallet mutation starts.\n");
+            fflush(stderr);
+            DBFlush(true);
+            return 0;
+        }
+        else if (strWalletBackend == "bdb")
+        {
+            // Explicit -walletbackend=bdb overrides the marker and the default:
+            // force Berkeley DB (LoadWallet creates wallet.dat if none exists).
+            if (!LoadWallet())
+            {
+                FatalStartupError(fHeadlessStartup, "Cannot open wallet.dat.",
+                                  strWalletLoadError);
+                return 1;
+            }
+        }
+        else
+        {
+            // No explicit -walletbackend on the command line. Resolve it from
+            // the persistent marker, then defaults: an existing wallet.sqlite
+            // stays on SQLite, an existing wallet.dat stays on Berkeley DB (the
+            // GUI offers to convert), and a fresh install starts on SQLite --
+            // the new default. wallet.dat is never touched by any of this.
+            string strSQLitePath = GetAppDir() + "/wallet.sqlite";
+            bool fSQLiteExists = FileExists(strSQLitePath.c_str());
+            bool fWalletDatExists = FileExists((GetAppDir() + "/wallet.dat").c_str());
+            string strMarker = ReadWalletBackendMarker();
+
+            bool fUseSQLite = false;
+            bool fCreateSQLite = false;
+            if (strMarker == "sqlite")
+            {
+                if (fSQLiteExists)          fUseSQLite = true;
+                else if (fWalletDatExists)  fUseSQLite = false;  // marker stale; real wallet is BDB
+                else { fUseSQLite = true; fCreateSQLite = true; }
+            }
+            else if (strMarker == "bdb")
+            {
+                fUseSQLite = false;
+            }
+            else  // no marker recorded yet
+            {
+                if (fSQLiteExists && !fWalletDatExists) fUseSQLite = true;
+                else if (fWalletDatExists)              fUseSQLite = false;
+                else { fUseSQLite = true; fCreateSQLite = true; }
+            }
+
+            if (fUseSQLite)
+            {
+                AttachTerminal();
+                bool fOk = fCreateSQLite
+                    ? CreateNewSQLiteWallet(strSQLitePath)
+                    : LoadWalletFromSQLiteRuntime(strSQLitePath);
+                if (!fOk)
+                {
+                    FatalStartupError(fHeadlessStartup,
+                        fCreateSQLite ? "Cannot create the SQLite wallet."
+                                      : "Cannot open the SQLite wallet.",
+                        strWalletLoadError);
+                    return 1;
+                }
+                // Record the choice so it is stable from here on.
+                if (ReadWalletBackendMarker() != "sqlite")
+                    WriteWalletBackendMarker("sqlite");
+                fprintf(stderr, "Wallet backend: SQLite\n  %s\n%s",
+                        strSQLitePath.c_str(),
+                        fCreateSQLite ? "A new wallet was created here.\n" : "");
+                fflush(stderr);
+                // Fall through and run the node; CWalletDB routes to this file.
+            }
+            else if (!LoadWallet())
+            {
+                // LoadWallet() explains itself into strWalletLoadError when it
+                // knows why -- an unsupported wallet format, for one, which is
+                // the case this whole path exists to make visible.
+                FatalStartupError(fHeadlessStartup, "Cannot open wallet.dat.",
+                                  strWalletLoadError);
+                return 1;
+            }
+        }
+
         // After the block index, so there is a chain to compare the wallet
-        // against, and before anything reports a balance.
+        // against, and before anything reports a balance. Works on either
+        // backend: RescanSpentFlags writes corrected flags through CWalletDB,
+        // which routes to SQLite when that backend is active.
         RescanSpentFlags();
     }
     catch (const std::exception& e)
@@ -858,6 +1141,10 @@ int main(int argc, char* argv[])
         // called here, which is why a wallet.dat copied elsewhere would not
         // open -- issue #40. Not optional.
         DBFlush(true);
+        // Checkpoint and close the SQLite wallet backend if one is active. A
+        // no-op on the Berkeley DB path. Safe to kill before this runs: WAL +
+        // synchronous=FULL already made every committed write durable.
+        WalletSQLiteRuntimeClose();
 #ifdef _WIN32
         // wallet.dat is safely flushed and closed now; let a pending console
         // control handler (CLOSE/LOGOFF/SHUTDOWN) return so the OS can finish.
@@ -874,6 +1161,7 @@ int main(int argc, char* argv[])
     StopNode();
     BtfStopManagedTor();
     DBFlush(true);
+    WalletSQLiteRuntimeClose();
     return ret;
 #endif
 }