From 4660b7ba94b071e8e7cc69c06e04b5ff64f12dd4 Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Thu, 13 Aug 2026 17:21:01 -0300 Subject: docs: inventory Berkeley DB storage --- docs/storage.md | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 docs/storage.md (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md new file mode 100644 index 0000000..4dd35bd --- /dev/null +++ b/docs/storage.md @@ -0,0 +1,115 @@ +# Bitflash Storage Inventory + +This document records the Berkeley DB surface that still exists in Bitflash. +It is a migration map, not a promise that all stores should move at once. + +The immediate priority is `wallet.dat`: it holds spend authority and recovery +metadata. `blkindex.dat` can be rebuilt from block files, so it is important for +startup correctness but less dangerous than wallet storage. + +## Berkeley DB Environment + +All `CDB` stores use one Berkeley DB environment rooted at: + +```text +/database/ +``` + +The environment creates log files such as: + +```text +database/log.0000000001 +``` + +Those logs are part of the legacy BDB risk model. They are not user-facing +wallet files, but they can contain historical records while the environment is +live. Wallet encryption already rewrites `wallet.dat` and purges BDB environment +logs after the encrypted file is installed. + +## Common Records + +Every `CDB` file can carry: + +| Key | Value | Purpose | +|---|---|---| +| `"version"` | `int` | Database format version written when a DB is opened in create mode. | + +Keys and values are serialized with the existing `CDataStream` format. A future +SQLite migration must preserve byte-for-byte value semantics until a higher +level schema migration is intentionally introduced. + +## `wallet.dat` + +`wallet.dat` is accessed through `CWalletDB`. + +| Key shape | Value | Secret? | Meaning | +|---|---|---:|---| +| `("name", address)` | `string` | No | Address book label. | +| `("tx", txid)` | `CWalletTx` | No | Wallet transaction state, including spend flags and metadata. | +| `("key", pubkey)` | `CPrivKey` | Yes | Plain private key record for unencrypted wallets or old backups. | +| `"defaultkey"` | `vector` | No | Default public key shown by the wallet. | +| `"hdmaster"` | `vector` | Yes | Plain BIP32 master private key material. | +| `"hdchaincode"` | `vector` | Yes | Plain BIP32 chain code. | +| `"hdnext"` | `unsigned int` | No | Legacy deterministic derivation next index. | +| `"hdschema"` | `int` | No | Deterministic derivation schema (`legacy` or `bip44`). | +| `"hdcointype"` | `unsigned int` | No | BIP44 coin type. Bitflash is `4346950`. | +| `"hdreceivenext"` | `unsigned int` | No | BIP44 external receive-chain next index. | +| `"hdchangenext"` | `unsigned int` | No | BIP44 internal change-chain next index. | +| `("mkey", id)` | `CWalletMasterKey` | Sensitive | Encrypted wallet master-key metadata. | +| `("ckey", pubkey)` | `vector` | Yes | Private key encrypted under the wallet master key. | +| `"cryptedhdmaster"` | `vector` | Yes | Encrypted BIP32 master private key material. | +| `"cryptedhdchaincode"` | `vector` | Yes | Encrypted BIP32 chain code. | +| `"walletminversion"` | `int` | No | Minimum wallet format version required to open this wallet safely. | +| `("pool", index)` | `vector` | No | Public key reserved in the key pool. The private key lives in `key`/`ckey`. | +| `("setting", name)` | typed setting value | Usually no | Wallet/UI settings such as mining mode, fees, and backup timestamps. | + +### Wallet Migration Rules + +A future wallet backend must keep these properties: + +1. Unknown future wallet formats fail closed with a clear message. +2. Encrypted wallets must encrypt private keys and HD seed material. +3. A locked encrypted wallet may expose public keys and balances, but not + private keys or decrypted HD seed material. +4. `-backupwallet` must produce a wallet file that opens in a different datadir + without copying `database/`. +5. `-recoveryaudit` must distinguish: + - no recovery phrase installed; + - recovery phrase present; + - recovery phrase encrypted and locked, requiring unlock before coverage can + be audited. + +## `blkindex.dat` + +`blkindex.dat` is accessed through `CTxDB`. + +| Key shape | Value | Rebuildable? | Meaning | +|---|---|---:|---| +| `("tx", txid)` | `CTxIndex` | Yes | Transaction disk position and spent-output index state. | +| `("owner", hash160, CDiskTxPos)` | `int height` | Yes | Legacy owner transaction index cursor range. | +| `("blockindex", blockhash)` | `CDiskBlockIndex` | Yes | Block index entry. | +| `"hashBestChain"` | `uint256` | Yes | Current best-chain tip hash. | + +`blkindex.dat` can be rebuilt from `blk*.dat` plus consensus validation. A +future migration may therefore prefer a rebuild path over a byte-preserving +conversion path. + +## Other Legacy Stores + +| File | Class | Records | Current risk | +|---|---|---|---| +| `reviews.dat` | `CReviewDB` | `("user", hash)`, `("reviews", hash)` | Legacy market/review storage. Not part of wallet safety. | +| `market.dat` | `CMarketDB` | No explicit typed records in current wrappers. | Legacy market storage shell. | + +The old IP address database (`addr.dat`) has been removed. Peer discovery is +handled by Nostr, `.btf`, rendezvous relays, and direct onion peers. + +## Recommended Migration Order + +1. Keep adding storage self-tests around current BDB behavior. +2. Add an experimental wallet backend behind an explicit flag. +3. Implement BDB-to-new-wallet migration as a one-way copy with a backup. +4. Make the new wallet backend default only after cross-version restore, + encryption, backup, and recovery-audit tests pass. +5. Treat `blkindex.dat` separately; it is safer to rebuild than to migrate under + pressure. -- cgit v1.2.3 From 95bd9cbcb1d17f4bc6223e6bfe3a92bfb8c3067d Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Fri, 14 Aug 2026 09:08:29 -0300 Subject: wallet: add read-only record scan boundary --- docs/storage.md | 3 +++ src/db.cpp | 39 ++++++++++++++++++++++++++++++++++ src/db.h | 11 ++++++++++ src/walletcmd.cpp | 63 ++++++++++++++++++++++--------------------------------- 4 files changed, 78 insertions(+), 38 deletions(-) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index 4dd35bd..e533589 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -41,6 +41,9 @@ level schema migration is intentionally introduced. ## `wallet.dat` `wallet.dat` is accessed through `CWalletDB`. +Read-only tooling should go through `ScanWalletRecords()` instead of opening a +Berkeley DB cursor directly. That keeps migration checks, audits, and future +backends behind one record-stream boundary. | Key shape | Value | Secret? | Meaning | |---|---|---:|---| diff --git a/src/db.cpp b/src/db.cpp index 9f62964..4fa72af 100644 --- a/src/db.cpp +++ b/src/db.cpp @@ -932,6 +932,45 @@ bool CWalletDB::LoadWallet(vector& vchDefaultKeyRet) return true; } +bool ScanWalletRecords(CWalletRecordVisitor& visitor, string& strErrorRet) +{ + class CWalletScanDB : public CWalletDB + { + public: + CWalletScanDB() : CWalletDB("r") { } + using CDB::GetCursor; + using CDB::ReadAtCursor; + }; + + strErrorRet.clear(); + CWalletScanDB dbWallet; + Dbc* pcursor = dbWallet.GetCursor(); + if (!pcursor) + { + strErrorRet = "could not open a wallet cursor"; + return false; + } + CAutoCursor cursorGuard(pcursor); + + loop + { + CDataStream ssKey(SER_DISK); + CDataStream ssValue(SER_DISK); + int ret = dbWallet.ReadAtCursor(pcursor, ssKey, ssValue); + if (ret == DB_NOTFOUND) + break; + if (ret != 0) + { + strErrorRet = strprintf("could not read wallet record: Berkeley DB error %d", ret); + return false; + } + if (!visitor.VisitWalletRecord(ssKey, ssValue, strErrorRet)) + return false; + } + + return true; +} + bool LoadWallet() { vector vchDefaultKey; diff --git a/src/db.h b/src/db.h index 4fb5cfd..2816957 100644 --- a/src/db.h +++ b/src/db.h @@ -524,6 +524,17 @@ public: bool LoadWallet(vector& vchDefaultKeyRet); }; +class CWalletRecordVisitor +{ +public: + virtual ~CWalletRecordVisitor() { } + virtual bool VisitWalletRecord(const CDataStream& ssKey, + const CDataStream& ssValue, + string& strErrorRet) = 0; +}; + +bool ScanWalletRecords(CWalletRecordVisitor& visitor, string& strErrorRet); + bool LoadWallet(); // Writes a wallet.dat that opens on its own, anywhere -- see the definition in diff --git a/src/walletcmd.cpp b/src/walletcmd.cpp index ce0da02..71f1ec3 100644 --- a/src/walletcmd.cpp +++ b/src/walletcmd.cpp @@ -553,51 +553,38 @@ static void CountWalletStorageType(const std::string& strType, static bool ReadWalletStorageCounts(WalletStorageAuditCounts& counts, std::string& strError) { - class CWalletAuditDB : public CWalletDB + class CWalletStorageCountVisitor : public CWalletRecordVisitor { public: - CWalletAuditDB() : CWalletDB("r") { } - using CDB::GetCursor; - using CDB::ReadAtCursor; - }; - - CWalletAuditDB walletdb; - Dbc* pcursor = walletdb.GetCursor(); - if (!pcursor) - { - strError = "cannot open wallet.dat cursor"; - return false; - } + WalletStorageAuditCounts& counts; - for (;;) - { - CDataStream ssKey(SER_DISK); - CDataStream ssValue(SER_DISK); - int ret = walletdb.ReadAtCursor(pcursor, ssKey, ssValue); - if (ret == DB_NOTFOUND) - break; - if (ret != 0) - { - pcursor->close(); - strError = strprintf("Berkeley DB cursor read failed (%d)", ret); - return false; - } + explicit CWalletStorageCountVisitor(WalletStorageAuditCounts& countsIn) + : counts(countsIn) { } - counts.nTotal++; - try + bool VisitWalletRecord(const CDataStream& ssKeyIn, + const CDataStream& ssValue, + std::string& strErrorRet) { - std::string strType; - ssKey >> strType; - CountWalletStorageType(strType, counts); - } - catch (...) - { - counts.nMalformed++; + (void)ssValue; + (void)strErrorRet; + counts.nTotal++; + try + { + CDataStream ssKey = ssKeyIn; + std::string strType; + ssKey >> strType; + CountWalletStorageType(strType, counts); + } + catch (...) + { + counts.nMalformed++; + } + return true; } - } + }; - pcursor->close(); - return true; + CWalletStorageCountVisitor visitor(counts); + return ScanWalletRecords(visitor, strError); } static std::string WalletStorageAuditJson(const WalletStorageAuditCounts& counts) -- cgit v1.2.3 From 5352185c7198a183c5906898e16bc4a6bb4e8654 Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Fri, 14 Aug 2026 09:55:53 -0300 Subject: wallet: route rewrite copies through record scan --- docs/storage.md | 4 ++- src/db.cpp | 105 +++++++++++++++++++++++++++----------------------------- 2 files changed, 54 insertions(+), 55 deletions(-) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index e533589..e4a66d2 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -43,7 +43,9 @@ level schema migration is intentionally introduced. `wallet.dat` is accessed through `CWalletDB`. Read-only tooling should go through `ScanWalletRecords()` instead of opening a Berkeley DB cursor directly. That keeps migration checks, audits, and future -backends behind one record-stream boundary. +backends behind one record-stream boundary. Wallet rewrite code should also use +that boundary when it needs to copy existing records into a newly written +wallet file. | Key shape | Value | Secret? | Meaning | |---|---|---:|---| diff --git a/src/db.cpp b/src/db.cpp index 4fa72af..b90219e 100644 --- a/src/db.cpp +++ b/src/db.cpp @@ -1354,70 +1354,68 @@ public: { return Write(make_pair(string("pool"), nIndex), vchPubKey); } +}; + +static bool IsWalletEncryptionPrivateRecord(const string& strType) +{ + return strType == "key" || + strType == "pool" || + strType == "hdmaster" || + strType == "hdchaincode" || + strType == "mkey" || + strType == "ckey" || + strType == "cryptedhdmaster" || + strType == "cryptedhdchaincode" || + strType == "walletminversion"; +} - bool CopyPublicRecordsTo(CWalletRewriteDB& dbTo, int& nCopiedRet, string& strErrorRet) +class CWalletPublicRecordCopyVisitor : public CWalletRecordVisitor +{ +public: + CWalletRewriteDB& dbTo; + int& nCopied; + + CWalletPublicRecordCopyVisitor(CWalletRewriteDB& dbToIn, int& nCopiedIn) + : dbTo(dbToIn), nCopied(nCopiedIn) { } + + bool VisitWalletRecord(const CDataStream& ssKey, + const CDataStream& ssValue, + string& strErrorRet) { - nCopiedRet = 0; - Dbc* pcursor = GetCursor(); - if (!pcursor) + string strType; + try { - strErrorRet = "could not open a wallet cursor"; - return false; + CDataStream ssType = ssKey; + ssType >> strType; } - - unsigned int fFlags = DB_NEXT; - loop + catch (...) { - CDataStream ssKey; - CDataStream ssValue; - int ret = ReadAtCursor(pcursor, ssKey, ssValue, fFlags); - if (ret == DB_NOTFOUND) - break; - if (ret != 0) - { - pcursor->close(); - strErrorRet = strprintf("could not read wallet record: Berkeley DB error %d", ret); - return false; - } - - string strType; - try - { - CDataStream ssType = ssKey; - ssType >> strType; - } - catch (...) - { - pcursor->close(); - strErrorRet = "wallet.dat has a record whose key type cannot be read"; - return false; - } + strErrorRet = "wallet.dat has a record whose key type cannot be read"; + return false; + } - if (strType == "key" || - strType == "pool" || - strType == "hdmaster" || - strType == "hdchaincode" || - strType == "mkey" || - strType == "ckey" || - strType == "cryptedhdmaster" || - strType == "cryptedhdchaincode" || - strType == "walletminversion") - continue; + if (IsWalletEncryptionPrivateRecord(strType)) + return true; - if (!dbTo.WriteRaw(ssKey, ssValue)) - { - pcursor->close(); - strErrorRet = strprintf("could not copy wallet record '%s'", strType.c_str()); - return false; - } - nCopiedRet++; + if (!dbTo.WriteRaw(ssKey, ssValue)) + { + strErrorRet = strprintf("could not copy wallet record '%s'", strType.c_str()); + return false; } - - pcursor->close(); + nCopied++; return true; } }; +static bool CopyPublicWalletRecordsTo(CWalletRewriteDB& dbTo, + int& nCopiedRet, + string& strErrorRet) +{ + nCopiedRet = 0; + CWalletPublicRecordCopyVisitor visitor(dbTo, nCopiedRet); + return ScanWalletRecords(visitor, strErrorRet); +} + static string WalletEncryptBackupPath() { string strBase = GetAppDir() + "/wallet.dat.before-encrypt."; @@ -1594,9 +1592,8 @@ bool EncryptWallet(const string& strPassphrase, string& strBackupRet, string& st int nCopied = 0; { - CWalletRewriteDB dbSource("wallet.dat", "r"); CWalletRewriteDB dbTarget(strTempFile.c_str(), "c+"); - if (!dbSource.CopyPublicRecordsTo(dbTarget, nCopied, strErrorRet)) + if (!CopyPublicWalletRecordsTo(dbTarget, nCopied, strErrorRet)) { remove(strTempPath.c_str()); return false; -- cgit v1.2.3 From dd9db5ba5407b9c30b4f41fc1153db39f0770154 Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Fri, 14 Aug 2026 11:08:34 -0300 Subject: wallet: install backups via temporary file --- docs/storage.md | 3 ++- src/db.cpp | 40 ++++++++++++++++++++++++++++++++++------ src/selftest.cpp | 7 +++++++ 3 files changed, 43 insertions(+), 7 deletions(-) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index e4a66d2..c1cda45 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -77,7 +77,8 @@ A future wallet backend must keep these properties: 3. A locked encrypted wallet may expose public keys and balances, but not private keys or decrypted HD seed material. 4. `-backupwallet` must produce a wallet file that opens in a different datadir - without copying `database/`. + without copying `database/`, and must not leave a half-written final backup + path if the copy or install step fails. 5. `-recoveryaudit` must distinguish: - no recovery phrase installed; - recovery phrase present; diff --git a/src/db.cpp b/src/db.cpp index b90219e..0ff58e7 100644 --- a/src/db.cpp +++ b/src/db.cpp @@ -1030,6 +1030,13 @@ bool BackupWallet(const string& strDest) if (!FileExists(strSrc.c_str())) return error("BackupWallet() : %s does not exist\n", strSrc.c_str()); + string strTemp = strprintf("%s.tmp.%lld", strDest.c_str(), GetTime()); + for (int i = 1; FileExists(strTemp.c_str()); i++) + strTemp = strprintf("%s.tmp.%lld.%d", strDest.c_str(), GetTime(), i); + string strOld = strprintf("%s.old.%lld", strDest.c_str(), GetTime()); + for (int i = 1; FileExists(strOld.c_str()); i++) + strOld = strprintf("%s.old.%lld.%d", strDest.c_str(), GetTime(), i); + CRITICAL_BLOCK(cs_db) { // Committing first, so the copy is not missing the newest records. @@ -1044,11 +1051,11 @@ bool BackupWallet(const string& strDest) FILE* pfIn = fopen(strSrc.c_str(), "rb"); if (!pfIn) return error("BackupWallet() : cannot read %s\n", strSrc.c_str()); - FILE* pfOut = fopen(strDest.c_str(), "wb"); + FILE* pfOut = fopen(strTemp.c_str(), "wb"); if (!pfOut) { fclose(pfIn); - return error("BackupWallet() : cannot write %s\n", strDest.c_str()); + return error("BackupWallet() : cannot write %s\n", strTemp.c_str()); } char buf[65536]; @@ -1065,18 +1072,39 @@ bool BackupWallet(const string& strDest) if (!fOk) { - remove(strDest.c_str()); // half a wallet is worse than none - return error("BackupWallet() : copy to %s failed\n", strDest.c_str()); + remove(strTemp.c_str()); // half a wallet is worse than none + return error("BackupWallet() : copy to %s failed\n", strTemp.c_str()); } // Without this the copy is welded to this node's database/ dir. - int ret = dbenv.lsn_reset(strDest.c_str(), 0); + int ret = dbenv.lsn_reset(strTemp.c_str(), 0); if (ret != 0) printf("BackupWallet() : warning -- lsn_reset returned %d; the copy " "may only open beside this node's database/ directory\n", ret); + + bool fHadOldDest = FileExists(strDest.c_str()); + if (fHadOldDest && rename(strDest.c_str(), strOld.c_str()) != 0) + { + remove(strTemp.c_str()); + return error("BackupWallet() : could not move old %s aside\n", strDest.c_str()); + } + if (rename(strTemp.c_str(), strDest.c_str()) != 0) + { + remove(strTemp.c_str()); + if (fHadOldDest) + rename(strOld.c_str(), strDest.c_str()); + return error("BackupWallet() : could not install %s\n", strDest.c_str()); + } + if (fHadOldDest) + remove(strOld.c_str()); } catch (const std::exception& e) - { return error("BackupWallet() : %s\n", e.what()); } + { + remove(strTemp.c_str()); + if (FileExists(strOld.c_str()) && !FileExists(strDest.c_str())) + rename(strOld.c_str(), strDest.c_str()); + return error("BackupWallet() : %s\n", e.what()); + } } // Recorded here rather than by whoever called, so that a backup taken with diff --git a/src/selftest.cpp b/src/selftest.cpp index 638aa66..81c4439 100644 --- a/src/selftest.cpp +++ b/src/selftest.cpp @@ -1391,6 +1391,13 @@ static int RunWalletPortabilitySelfTest() nFail += Check(nBackupRet == 0 && FileExists(strPortableWallet.c_str()), "/backupwallet writes a portable wallet.dat") ? 0 : 1; + string strBackupOut2 = tmp + "/backupwallet-overwrite.txt"; + int nBackupRet2 = RunBitflashChild(strExe, vBackupArgs, NULL, &strBackupOut2); + nFail += Check(nBackupRet2 == 0 && FileExists(strPortableWallet.c_str()) && + !DirectoryHasFileWithPrefix(tmp, "portable-wallet.dat.tmp.") && + !DirectoryHasFileWithPrefix(tmp, "portable-wallet.dat.old."), + "/backupwallet replaces an existing backup without leaving temp files") ? 0 : 1; + nFail += Check(CopyFileLocal(strOriginal + "/wallet.dat", strRawCopy + "/wallet.dat"), "a raw wallet.dat can be copied without database logs") ? 0 : 1; -- cgit v1.2.3 From c49b92a9a79a398b51200e69f7dde8a467b9c925 Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Fri, 14 Aug 2026 15:03:21 -0300 Subject: wallet: add experimental SQLite record store --- .github/workflows/ci.yml | 1 + Makefile | 8 +- docs/storage.md | 19 ++++ src/Makefile | 10 +- src/main_gui.cpp | 3 +- src/makefile.mingw | 8 +- src/selftest.cpp | 108 +++++++++++++++++++- src/wallet_sqlite.cpp | 254 +++++++++++++++++++++++++++++++++++++++++++++++ src/wallet_sqlite.h | 40 ++++++++ 9 files changed, 439 insertions(+), 12 deletions(-) create mode 100644 src/wallet_sqlite.cpp create mode 100644 src/wallet_sqlite.h (limited to 'docs/storage.md') diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f4045f4..e22b5a4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -56,6 +56,7 @@ jobs: mingw-w64-ucrt-x86_64-glfw mingw-w64-ucrt-x86_64-openssl mingw-w64-ucrt-x86_64-db + mingw-w64-ucrt-x86_64-sqlite3 mingw-w64-ucrt-x86_64-libsodium mingw-w64-ucrt-x86_64-nlohmann-json mingw-w64-ucrt-x86_64-boost diff --git a/Makefile b/Makefile index 60d1227..fab2507 100644 --- a/Makefile +++ b/Makefile @@ -43,10 +43,11 @@ deps-apt: @if command -v apt-get >/dev/null 2>&1; then \ $(SUDO) apt-get install -y build-essential cmake git pkg-config autoconf \ libtool libssl-dev libdb5.3++-dev libsodium-dev nlohmann-json3-dev \ - libboost-system-dev libglfw3-dev libgl-dev python3-pil clang; \ + libboost-system-dev libglfw3-dev libgl-dev libsqlite3-dev \ + python3-pil clang; \ else \ echo "Not an apt system — install: g++ cmake git autoconf libtool"; \ - echo " libssl libdb++ libsodium nlohmann-json boost glfw3 opengl python3-pil"; \ + echo " libssl libdb++ libsodium nlohmann-json boost glfw3 opengl sqlite3 python3-pil"; \ fi deps-secp256k1: @@ -113,7 +114,7 @@ img = ico.convert('RGBA'); \ img.save('/tmp/bitflash.AppDir/bitflash.png'); \ import shutil; shutil.copy('/tmp/bitflash.AppDir/bitflash.png', \ '/tmp/bitflash.AppDir/usr/share/icons/hicolor/256x256/apps/bitflash.png')" - @for lib in libsodium.so.23 libdb_cxx-5.3.so libglfw.so.3; do \ + @for lib in libsodium.so.23 libdb_cxx-5.3.so libsqlite3.so.0 libglfw.so.3; do \ path=$$(ldconfig -p | grep " $$lib " | awk '{print $$NF}' | head -1); \ [ -z "$$path" ] && path=$$(find /usr/lib /lib -name "$$lib" 2>/dev/null | head -1); \ [ -n "$$path" ] && cp -L "$$path" /tmp/bitflash.AppDir/usr/lib/ && echo " bundled $$lib" || echo " missing $$lib"; \ @@ -131,6 +132,7 @@ deps-windows: mingw-w64-ucrt-x86_64-glfw \ mingw-w64-ucrt-x86_64-openssl \ mingw-w64-ucrt-x86_64-db \ + mingw-w64-ucrt-x86_64-sqlite3 \ mingw-w64-ucrt-x86_64-libsodium \ mingw-w64-ucrt-x86_64-nlohmann-json \ mingw-w64-ucrt-x86_64-boost \ diff --git a/docs/storage.md b/docs/storage.md index c1cda45..f75f9b9 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -85,6 +85,25 @@ A future wallet backend must keep these properties: - recovery phrase encrypted and locked, requiring unlock before coverage can be audited. +## Experimental SQLite Wallet Store + +The SQLite wallet path starts as a byte-preserving record store, not as a new +wallet format: + +```sql +CREATE TABLE wallet_records ( + key BLOB PRIMARY KEY NOT NULL, + value BLOB NOT NULL +); +PRAGMA user_version=1; +``` + +Keys and values are the same serialized `CDataStream` bytes used by `wallet.dat`. +The first implementation is exercised only by `-selftest=wallet-sqlite`; the +runtime wallet still opens Berkeley DB. The purpose is to prove schema creation, +raw record write/read, close/reopen behavior, and value preservation before any +BDB-to-SQLite migration or runtime backend flag exists. + ## `blkindex.dat` `blkindex.dat` is accessed through `CTxDB`. diff --git a/src/Makefile b/src/Makefile index 19ad33a..cfa40f6 100644 --- a/src/Makefile +++ b/src/Makefile @@ -40,7 +40,7 @@ FUZZ_LDFLAGS = -fsanitize=fuzzer,address,undefined LIBS = -L$(SECP_LIB) -lsecp256k1 \ -L$(RANDOMX_DIR)/build -lrandomx \ - -lsodium -lssl -lcrypto -ldb_cxx -lpthread \ + -lsodium -lssl -lcrypto -ldb_cxx -lsqlite3 -lpthread \ -lglfw -lGL # Every header a .cpp here includes has to be listed, not just the interesting @@ -53,7 +53,7 @@ LIBS = -L$(SECP_LIB) -lsecp256k1 \ HEADERS = headers_core.h headers.h util.h main.h serialize.h uint256.h \ key.h bignum.h script.h db.h base58.h net.h nostr.h \ randomx_pow.h btfaddr.h btfchan.h btfrv.h btftunnel.h market.h \ - selftest.h walletcmd.h compat.h sockcount.h proxy.h tor.h font_roboto.h + selftest.h walletcmd.h wallet_sqlite.h compat.h sockcount.h proxy.h tor.h font_roboto.h IMGUI_OBJS = obj/imgui.o obj/imgui_draw.o obj/imgui_tables.o \ obj/imgui_widgets.o obj/imgui_impl_glfw.o obj/imgui_impl_opengl3.o @@ -61,7 +61,8 @@ IMGUI_OBJS = obj/imgui.o obj/imgui_draw.o obj/imgui_tables.o \ NODE_OBJS = obj/util.o obj/script.o obj/db.o obj/net.o obj/main.o \ obj/market.o obj/sha.o obj/nostr.o obj/randomx_pow.o \ obj/btfaddr.o obj/btfchan.o obj/btfrv.o obj/btftunnel.o obj/rpc.o \ - obj/selftest.o obj/bip32.o obj/walletcmd.o obj/proxy.o obj/tor.o + obj/selftest.o obj/bip32.o obj/walletcmd.o obj/wallet_sqlite.o \ + obj/proxy.o obj/tor.o GUI_OBJS = obj/gui.o obj/main_gui.o @@ -116,7 +117,7 @@ bitflash: $(IMGUI_OBJS) $(NODE_OBJS) $(GUI_OBJS) NODE_LIBS = -L$(SECP_LIB) -lsecp256k1 \ -L$(RANDOMX_DIR)/build -lrandomx \ - -lsodium -lssl -lcrypto -ldb_cxx -lpthread + -lsodium -lssl -lcrypto -ldb_cxx -lsqlite3 -lpthread NODE_ONLY_OBJS = $(patsubst obj/%,obj-node/%,$(NODE_OBJS)) \ obj-node/main_gui.o obj-node/headless.o @@ -187,6 +188,7 @@ tests: test_bip32 bitflash-node ./bitflash-node -selftest=wallet-keypool ./bitflash-node -selftest=wallet-hd ./bitflash-node -selftest=wallet-format + ./bitflash-node -selftest=wallet-sqlite ./bitflash-node -selftest=wallet-crypto ./bitflash-node -selftest=wallet-encrypt ./bitflash-node -selftest=net-message diff --git a/src/main_gui.cpp b/src/main_gui.cpp index 705b4ef..965c818 100644 --- a/src/main_gui.cpp +++ b/src/main_gui.cpp @@ -114,7 +114,8 @@ static void PrintUsage() printf(" /gen\n"); printf(" /nogui or /daemon\n"); printf(" /selftest=wallet-keypool, wallet-hd, wallet-format, wallet-storage-sanity,\n"); - printf(" db-env-reopen, wallet-crypto, wallet-encrypt, wallet-portability,\n"); + printf(" db-env-reopen, wallet-sqlite, wallet-crypto, wallet-encrypt,\n"); + printf(" wallet-portability,\n"); printf(" net-message,\n"); printf(" consensus-limits, pool-stratum,\n"); printf(" parse-money, socks5-proxy, or managed-tor\n"); diff --git a/src/makefile.mingw b/src/makefile.mingw index 333be96..2ff4e11 100644 --- a/src/makefile.mingw +++ b/src/makefile.mingw @@ -24,7 +24,7 @@ CXXFLAGS = -std=gnu++14 -O2 $(WARNFLAGS) -fpermissive -pthread \ LIBS = -L$(DEPS)/lib -lsecp256k1 \ -L$(RANDOMX)/build -lrandomx \ - -lsodium -lssl -lcrypto -ldb_cxx -lpthread \ + -lsodium -lssl -lcrypto -ldb_cxx -lsqlite3 -lpthread \ -LC:/msys64/ucrt64/lib \ -lglfw3 -lopengl32 -lgdi32 \ -lkernel32 -luser32 -lcomdlg32 -lwinspool -lwinmm \ @@ -37,7 +37,8 @@ IMGUI_OBJS = obj/imgui.o obj/imgui_draw.o obj/imgui_tables.o \ NODE_OBJS = obj/util.o obj/script.o obj/db.o obj/net.o obj/main.o \ obj/market.o obj/sha.o obj/nostr.o obj/randomx_pow.o \ obj/btfaddr.o obj/btfchan.o obj/btfrv.o obj/btftunnel.o obj/rpc.o \ - obj/selftest.o obj/bip32.o obj/walletcmd.o obj/proxy.o obj/tor.o + obj/selftest.o obj/bip32.o obj/walletcmd.o obj/wallet_sqlite.o \ + obj/proxy.o obj/tor.o GUI_OBJS = obj/gui.o obj/main_gui.o obj/bitflash_res.o @@ -51,7 +52,7 @@ GUI_OBJS = obj/gui.o obj/main_gui.o obj/bitflash_res.o HEADERS = headers_core.h headers.h util.h main.h serialize.h uint256.h \ key.h bignum.h script.h db.h base58.h net.h nostr.h \ randomx_pow.h btfaddr.h btfchan.h btfrv.h btftunnel.h market.h \ - selftest.h walletcmd.h sockcount.h proxy.h tor.h compat.h font_roboto.h + selftest.h walletcmd.h wallet_sqlite.h sockcount.h proxy.h tor.h compat.h font_roboto.h all: bitflash.exe @@ -107,6 +108,7 @@ tests: test_bip32.exe bitflash.exe ./bitflash.exe -selftest=wallet-keypool ./bitflash.exe -selftest=wallet-hd ./bitflash.exe -selftest=wallet-format + ./bitflash.exe -selftest=wallet-sqlite ./bitflash.exe -selftest=wallet-crypto ./bitflash.exe -selftest=wallet-encrypt ./bitflash.exe -selftest=net-message diff --git a/src/selftest.cpp b/src/selftest.cpp index 31c168f..f9c4a8e 100644 --- a/src/selftest.cpp +++ b/src/selftest.cpp @@ -13,6 +13,7 @@ #include "sockcount.h" #include "tor.h" #include "walletcmd.h" +#include "wallet_sqlite.h" extern int RunPoolStratumSelfTest(); @@ -62,6 +63,11 @@ static std::string HexStrLocal(const std::vector& v) return out; } +static vector DataStreamBytes(const CDataStream& ss) +{ + return vector(ss.begin(), ss.end()); +} + static bool MakeTempDir(std::string& out) { #ifdef _WIN32 @@ -1030,6 +1036,104 @@ static int RunDbEnvReopenSelfTest() return nFail == 0 ? 0 : 1; } +static int RunWalletSQLiteSelfTest() +{ + fflush(stdout); + printf("wallet-sqlite self-test\n"); + + std::string tmp; + if (!MakeTempDir(tmp)) + { + printf(" FAIL could not create a temporary data directory\n"); + return 1; + } + + int nFail = 0; + string strPath = tmp + "/wallet.sqlite"; + string strError; + + try + { + CDataStream ssKeySetting(SER_DISK); + ssKeySetting << make_pair(string("setting"), string("sqlite-roundtrip")); + CDataStream ssValueSetting(SER_DISK); + ssValueSetting << (int64)424242; + + vector vchPubKey; + vchPubKey.push_back(0x04); + for (int i = 0; i < 64; i++) + vchPubKey.push_back((unsigned char)i); + CDataStream ssKeyDefault(SER_DISK); + ssKeyDefault << string("defaultkey"); + CDataStream ssValueDefault(SER_DISK); + ssValueDefault << vchPubKey; + + { + CWalletDBSQLite db; + nFail += Check(db.Open(strPath, strError), + "SQLite wallet opens and creates schema") ? 0 : 1; + nFail += Check(db.BeginTransaction(strError), + "SQLite wallet begins a write transaction") ? 0 : 1; + nFail += Check(db.WriteRecord(DataStreamBytes(ssKeySetting), + DataStreamBytes(ssValueSetting), + strError), + "SQLite wallet writes a serialized setting record") ? 0 : 1; + nFail += Check(db.WriteRecord(DataStreamBytes(ssKeyDefault), + DataStreamBytes(ssValueDefault), + strError), + "SQLite wallet writes a serialized defaultkey record") ? 0 : 1; + nFail += Check(db.CommitTransaction(strError), + "SQLite wallet commits a write transaction") ? 0 : 1; + int nCount = 0; + nFail += Check(db.CountRecords(nCount, strError) && nCount == 2, + "SQLite wallet counts raw records") ? 0 : 1; + } + + { + CWalletDBSQLite db; + nFail += Check(db.Open(strPath, strError), + "SQLite wallet reopens") ? 0 : 1; + vector vchRead; + nFail += Check(db.ReadRecord(DataStreamBytes(ssKeySetting), + vchRead, + strError), + "SQLite wallet reads a serialized setting record") ? 0 : 1; + int64 nReadValue = 0; + CDataStream ssRead(vchRead, SER_DISK); + ssRead >> nReadValue; + nFail += Check(!ssRead.fail() && nReadValue == 424242, + "SQLite wallet preserves serialized values byte-for-byte") ? 0 : 1; + + vchRead.clear(); + nFail += Check(db.ReadRecord(DataStreamBytes(ssKeyDefault), + vchRead, + strError), + "SQLite wallet reads a serialized defaultkey record") ? 0 : 1; + vector vchReadPubKey; + CDataStream ssReadDefault(vchRead, SER_DISK); + ssReadDefault >> vchReadPubKey; + nFail += Check(!ssReadDefault.fail() && vchReadPubKey == vchPubKey, + "SQLite wallet preserves public key blobs") ? 0 : 1; + } + } + catch (const std::exception& e) + { + printf(" FAIL exception: %s\n", e.what()); + nFail++; + } + catch (...) + { + printf(" FAIL unknown exception\n"); + nFail++; + } + + RemoveTree(tmp); + printf("%s (%d failure%s)\n", nFail == 0 ? "ALL TESTS PASSED" : "TESTS FAILED", + nFail, nFail == 1 ? "" : "s"); + fflush(stdout); + return nFail == 0 ? 0 : 1; +} + static void ClearWalletRuntimeForTest() { CRITICAL_BLOCK(cs_mapKeys) @@ -2289,6 +2393,8 @@ int RunSelfTest(const std::string& name) return RunWalletStorageSanitySelfTest(); if (name == "db-env-reopen") return RunDbEnvReopenSelfTest(); + if (name == "wallet-sqlite") + return RunWalletSQLiteSelfTest(); if (name == "wallet-crypto") return RunWalletCryptoSelfTest(); if (name == "wallet-encrypt") @@ -2309,6 +2415,6 @@ int RunSelfTest(const std::string& name) return RunManagedTorSelfTest(); printf("Unknown self-test '%s'\n", name.c_str()); - printf("Known self-tests: wallet-keypool, wallet-hd, wallet-format, wallet-storage-sanity, db-env-reopen, wallet-crypto, wallet-encrypt, wallet-portability, net-message, consensus-limits, pool-stratum, parse-money, socks5-proxy, managed-tor\n"); + printf("Known self-tests: wallet-keypool, wallet-hd, wallet-format, wallet-storage-sanity, db-env-reopen, wallet-sqlite, wallet-crypto, wallet-encrypt, wallet-portability, net-message, consensus-limits, pool-stratum, parse-money, socks5-proxy, managed-tor\n"); return 1; } diff --git a/src/wallet_sqlite.cpp b/src/wallet_sqlite.cpp new file mode 100644 index 0000000..4104ac6 --- /dev/null +++ b/src/wallet_sqlite.cpp @@ -0,0 +1,254 @@ +// Copyright (c) 2026 Bitflash developers +// Distributed under the MIT/X11 software license. + +#include "headers.h" +#include "wallet_sqlite.h" + +static void SetSQLiteError(sqlite3* pdb, const string& strContext, + string& strError) +{ + if (pdb) + strError = strContext + ": " + sqlite3_errmsg(pdb); + else + strError = strContext; +} + +CWalletDBSQLite::CWalletDBSQLite() : pdb(NULL) +{ +} + +CWalletDBSQLite::~CWalletDBSQLite() +{ + Close(); +} + +bool CWalletDBSQLite::Open(const string& strPath, string& strError) +{ + Close(); + + int ret = sqlite3_open_v2(strPath.c_str(), &pdb, + SQLITE_OPEN_READWRITE | + SQLITE_OPEN_CREATE | + SQLITE_OPEN_FULLMUTEX, + NULL); + if (ret != SQLITE_OK) + { + SetSQLiteError(pdb, "cannot open SQLite wallet", strError); + Close(); + return false; + } + + sqlite3_busy_timeout(pdb, 5000); + + if (!Exec("PRAGMA journal_mode=WAL;", strError)) + { + Close(); + return false; + } + if (!Exec("PRAGMA synchronous=FULL;", strError)) + { + Close(); + return false; + } + if (!InitSchema(strError)) + { + Close(); + return false; + } + return true; +} + +void CWalletDBSQLite::Close() +{ + if (pdb) + { + sqlite3_close(pdb); + pdb = NULL; + } +} + +bool CWalletDBSQLite::BeginTransaction(string& strError) +{ + if (!pdb) + { + strError = "SQLite wallet is not open"; + return false; + } + return Exec("BEGIN IMMEDIATE;", strError); +} + +bool CWalletDBSQLite::CommitTransaction(string& strError) +{ + if (!pdb) + { + strError = "SQLite wallet is not open"; + return false; + } + return Exec("COMMIT;", strError); +} + +bool CWalletDBSQLite::RollbackTransaction(string& strError) +{ + if (!pdb) + { + strError = "SQLite wallet is not open"; + return false; + } + return Exec("ROLLBACK;", strError); +} + +bool CWalletDBSQLite::Exec(const char* pszSql, string& strError) +{ + char* pszErr = NULL; + int ret = sqlite3_exec(pdb, pszSql, NULL, NULL, &pszErr); + if (ret != SQLITE_OK) + { + if (pszErr) + { + strError = pszErr; + sqlite3_free(pszErr); + } + else + SetSQLiteError(pdb, "SQLite exec failed", strError); + return false; + } + return true; +} + +bool CWalletDBSQLite::InitSchema(string& strError) +{ + return Exec("CREATE TABLE IF NOT EXISTS wallet_records (" + "key BLOB PRIMARY KEY NOT NULL," + "value BLOB NOT NULL" + ");", + strError) && + Exec("PRAGMA user_version=1;", strError); +} + +bool CWalletDBSQLite::WriteRecord(const vector& vchKey, + const vector& vchValue, + string& strError) +{ + if (!pdb) + { + strError = "SQLite wallet is not open"; + return false; + } + if (vchKey.empty()) + { + strError = "SQLite wallet record key is empty"; + return false; + } + + sqlite3_stmt* stmt = NULL; + const char* pszSql = + "INSERT OR REPLACE INTO wallet_records(key, value) VALUES(?, ?);"; + int ret = sqlite3_prepare_v2(pdb, pszSql, -1, &stmt, NULL); + if (ret != SQLITE_OK) + { + SetSQLiteError(pdb, "cannot prepare SQLite wallet write", strError); + return false; + } + + ret = sqlite3_bind_blob(stmt, 1, &vchKey[0], (int)vchKey.size(), + SQLITE_TRANSIENT); + if (ret == SQLITE_OK) + ret = sqlite3_bind_blob(stmt, 2, + vchValue.empty() ? NULL : &vchValue[0], + (int)vchValue.size(), + SQLITE_TRANSIENT); + if (ret == SQLITE_OK) + ret = sqlite3_step(stmt); + + bool fOk = (ret == SQLITE_DONE); + if (!fOk) + SetSQLiteError(pdb, "cannot write SQLite wallet record", strError); + sqlite3_finalize(stmt); + return fOk; +} + +bool CWalletDBSQLite::ReadRecord(const vector& vchKey, + vector& vchValueRet, + string& strError) +{ + vchValueRet.clear(); + if (!pdb) + { + strError = "SQLite wallet is not open"; + return false; + } + if (vchKey.empty()) + { + strError = "SQLite wallet record key is empty"; + return false; + } + + sqlite3_stmt* stmt = NULL; + const char* pszSql = "SELECT value FROM wallet_records WHERE key = ?;"; + int ret = sqlite3_prepare_v2(pdb, pszSql, -1, &stmt, NULL); + if (ret != SQLITE_OK) + { + SetSQLiteError(pdb, "cannot prepare SQLite wallet read", strError); + return false; + } + + ret = sqlite3_bind_blob(stmt, 1, &vchKey[0], (int)vchKey.size(), + SQLITE_TRANSIENT); + if (ret != SQLITE_OK) + { + SetSQLiteError(pdb, "cannot bind SQLite wallet key", strError); + sqlite3_finalize(stmt); + return false; + } + + ret = sqlite3_step(stmt); + if (ret == SQLITE_ROW) + { + const unsigned char* pch = + (const unsigned char*)sqlite3_column_blob(stmt, 0); + int nBytes = sqlite3_column_bytes(stmt, 0); + if (nBytes > 0 && pch) + vchValueRet.assign(pch, pch + nBytes); + sqlite3_finalize(stmt); + return true; + } + + if (ret == SQLITE_DONE) + strError = "SQLite wallet record not found"; + else + SetSQLiteError(pdb, "cannot read SQLite wallet record", strError); + sqlite3_finalize(stmt); + return false; +} + +bool CWalletDBSQLite::CountRecords(int& nRecordsRet, string& strError) +{ + nRecordsRet = 0; + if (!pdb) + { + strError = "SQLite wallet is not open"; + return false; + } + + sqlite3_stmt* stmt = NULL; + int ret = sqlite3_prepare_v2(pdb, + "SELECT COUNT(*) FROM wallet_records;", + -1, &stmt, NULL); + if (ret != SQLITE_OK) + { + SetSQLiteError(pdb, "cannot prepare SQLite wallet count", strError); + return false; + } + + ret = sqlite3_step(stmt); + if (ret != SQLITE_ROW) + { + SetSQLiteError(pdb, "cannot count SQLite wallet records", strError); + sqlite3_finalize(stmt); + return false; + } + + nRecordsRet = sqlite3_column_int(stmt, 0); + sqlite3_finalize(stmt); + return true; +} diff --git a/src/wallet_sqlite.h b/src/wallet_sqlite.h new file mode 100644 index 0000000..9812d3a --- /dev/null +++ b/src/wallet_sqlite.h @@ -0,0 +1,40 @@ +// Copyright (c) 2026 Bitflash developers +// Distributed under the MIT/X11 software license. + +#ifndef BITFLASH_WALLET_SQLITE_H +#define BITFLASH_WALLET_SQLITE_H + +#include + +class CWalletDBSQLite +{ +public: + CWalletDBSQLite(); + ~CWalletDBSQLite(); + + bool Open(const string& strPath, string& strError); + void Close(); + + bool BeginTransaction(string& strError); + bool CommitTransaction(string& strError); + bool RollbackTransaction(string& strError); + + bool WriteRecord(const vector& vchKey, + const vector& vchValue, + string& strError); + bool ReadRecord(const vector& vchKey, + vector& vchValueRet, + string& strError); + bool CountRecords(int& nRecordsRet, string& strError); + +private: + sqlite3* pdb; + + bool Exec(const char* pszSql, string& strError); + bool InitSchema(string& strError); + + CWalletDBSQLite(const CWalletDBSQLite&); + void operator=(const CWalletDBSQLite&); +}; + +#endif -- cgit v1.2.3 From 59b9d11015f0e3a2475d0750fc534583c2ca9f77 Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Fri, 14 Aug 2026 15:56:12 -0300 Subject: wallet: export BDB records to SQLite --- docs/storage.md | 6 ++ src/Makefile | 1 + src/main_gui.cpp | 15 ++++- src/makefile.mingw | 1 + src/selftest.cpp | 148 +++++++++++++++++++++++++++++++++++++++++++++++++- src/wallet_sqlite.cpp | 10 ++++ src/wallet_sqlite.h | 1 + src/walletcmd.cpp | 97 +++++++++++++++++++++++++++++++++ src/walletcmd.h | 5 ++ 9 files changed, 281 insertions(+), 3 deletions(-) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index f75f9b9..bb414b5 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -104,6 +104,12 @@ runtime wallet still opens Berkeley DB. The purpose is to prove schema creation, raw record write/read, close/reopen behavior, and value preservation before any BDB-to-SQLite migration or runtime backend flag exists. +`-walletsqliteexport=FILE` is the next staging tool. It streams every raw +`wallet.dat` record through `ScanWalletRecords()` and writes those same key and +value bytes into the SQLite table inside one transaction. It refuses to +overwrite an existing export and still does not change the runtime wallet +backend. + ## `blkindex.dat` `blkindex.dat` is accessed through `CTxDB`. diff --git a/src/Makefile b/src/Makefile index cfa40f6..b83ff58 100644 --- a/src/Makefile +++ b/src/Makefile @@ -189,6 +189,7 @@ tests: test_bip32 bitflash-node ./bitflash-node -selftest=wallet-hd ./bitflash-node -selftest=wallet-format ./bitflash-node -selftest=wallet-sqlite + ./bitflash-node -selftest=wallet-sqlite-migration ./bitflash-node -selftest=wallet-crypto ./bitflash-node -selftest=wallet-encrypt ./bitflash-node -selftest=net-message diff --git a/src/main_gui.cpp b/src/main_gui.cpp index 965c818..6e89270 100644 --- a/src/main_gui.cpp +++ b/src/main_gui.cpp @@ -114,8 +114,8 @@ static void PrintUsage() printf(" /gen\n"); printf(" /nogui or /daemon\n"); printf(" /selftest=wallet-keypool, wallet-hd, wallet-format, wallet-storage-sanity,\n"); - printf(" db-env-reopen, wallet-sqlite, wallet-crypto, wallet-encrypt,\n"); - printf(" wallet-portability,\n"); + printf(" db-env-reopen, wallet-sqlite, wallet-sqlite-migration,\n"); + printf(" wallet-crypto, wallet-encrypt, wallet-portability,\n"); printf(" net-message,\n"); printf(" consensus-limits, pool-stratum,\n"); printf(" parse-money, socks5-proxy, or managed-tor\n"); @@ -181,6 +181,8 @@ static void PrintUsage() printf(" (write the same audit as deterministic JSON)\n"); printf(" /walletstoragecheck (fail closed on inconsistent or unsafe\n"); printf(" wallet.dat storage state)\n"); + printf(" /walletsqliteexport=FILE (copy raw wallet.dat records into an\n"); + printf(" experimental SQLite store, then exit)\n"); printf(" /rescan (walk the chain for coins this wallet owns\n"); printf(" but never recorded, then exit)\n"); printf("\n"); @@ -529,6 +531,15 @@ int main(int argc, char* argv[]) return nRet; } + string strWalletSQLiteExport = + argval2(argc, argv, "/walletsqliteexport", "-walletsqliteexport"); + if (!strWalletSQLiteExport.empty()) + { + int nRet = CmdWalletSQLiteExport(strWalletSQLiteExport); + DBFlush(true); + return nRet; + } + // Berkeley DB reports failure by throwing, and CDB's constructor lets it // through. Nothing on this path caught anything, so an unreadable // blkindex.dat or wallet.dat -- one written by another platform's Berkeley diff --git a/src/makefile.mingw b/src/makefile.mingw index 2ff4e11..14e6b32 100644 --- a/src/makefile.mingw +++ b/src/makefile.mingw @@ -109,6 +109,7 @@ tests: test_bip32.exe bitflash.exe ./bitflash.exe -selftest=wallet-hd ./bitflash.exe -selftest=wallet-format ./bitflash.exe -selftest=wallet-sqlite + ./bitflash.exe -selftest=wallet-sqlite-migration ./bitflash.exe -selftest=wallet-crypto ./bitflash.exe -selftest=wallet-encrypt ./bitflash.exe -selftest=net-message diff --git a/src/selftest.cpp b/src/selftest.cpp index f9c4a8e..7a54271 100644 --- a/src/selftest.cpp +++ b/src/selftest.cpp @@ -1134,6 +1134,150 @@ static int RunWalletSQLiteSelfTest() return nFail == 0 ? 0 : 1; } +class CWalletRecordMapVisitor : public CWalletRecordVisitor +{ +public: + std::map, vector >& mapRecords; + + explicit CWalletRecordMapVisitor( + std::map, vector >& mapRecordsIn) + : mapRecords(mapRecordsIn) { } + + bool VisitWalletRecord(const CDataStream& ssKey, + const CDataStream& ssValue, + string& strErrorRet) + { + vector vchKey = DataStreamBytes(ssKey); + if (mapRecords.count(vchKey)) + { + strErrorRet = "duplicate serialized wallet key"; + return false; + } + mapRecords[vchKey] = DataStreamBytes(ssValue); + return true; + } +}; + +static int RunWalletSQLiteMigrationSelfTest() +{ + fflush(stdout); + printf("wallet-sqlite-migration self-test\n"); + + std::string tmp; + std::string cwd; + if (!MakeTempDir(tmp)) + { + printf(" FAIL could not create a temporary data directory\n"); + return 1; + } + if (!GetCurrentDir(cwd)) + { + printf(" FAIL could not read current directory\n"); + RemoveTree(tmp); + return 1; + } + + int nFail = 0; + printf(" temp root: %s\n", tmp.c_str()); + + try + { +#ifdef _WIN32 + string strExe = cwd + "\\bitflash.exe"; +#else + string strExe = cwd + "/bitflash-node"; +#endif + string strWalletDir = tmp + "/wallet"; + nFail += Check(MakeDirLocal(strWalletDir), + "temporary wallet directory can be created") ? 0 : 1; + + string strCreateOut = tmp + "/create.txt"; + vector vCreateArgs; + vCreateArgs.push_back("-datadir=" + strWalletDir); + vCreateArgs.push_back("-nomanagedtor"); + vCreateArgs.push_back("-nogui"); + vCreateArgs.push_back("-newphrase"); + int nCreateRet = RunBitflashChild(strExe, vCreateArgs, NULL, &strCreateOut); + nFail += Check(nCreateRet == 0 && + FileContainsText(strCreateOut, "Write these twelve words down"), + "a phrase wallet can be created for SQLite export") ? 0 : 1; + + string strSQLite = tmp + "/wallet.sqlite"; + string strExportOut = tmp + "/sqlite-export.txt"; + vector vExportArgs; + vExportArgs.push_back("-datadir=" + strWalletDir); + vExportArgs.push_back("-nomanagedtor"); + vExportArgs.push_back("-nogui"); + vExportArgs.push_back("-walletsqliteexport=" + strSQLite); + int nExportRet = RunBitflashChild(strExe, vExportArgs, NULL, &strExportOut); + nFail += Check(nExportRet == 0 && + FileExists(strSQLite.c_str()) && + FileContainsText(strExportOut, "SQLite wallet export written"), + "wallet.dat can be exported to SQLite") ? 0 : 1; + + string strOverwriteOut = tmp + "/sqlite-export-overwrite.txt"; + int nOverwriteRet = + RunBitflashChild(strExe, vExportArgs, NULL, &strOverwriteOut); + nFail += Check(nOverwriteRet != 0 && + FileContainsText(strOverwriteOut, + "Refusing to overwrite existing SQLite wallet export"), + "SQLite wallet export refuses to overwrite") ? 0 : 1; + remove((strSQLite + "-wal").c_str()); + remove((strSQLite + "-shm").c_str()); + + std::map, vector > mapBDB; + string strOldDataDir = strSetDataDir; + string strScanError; + strSetDataDir = strWalletDir; + CWalletRecordMapVisitor visitor(mapBDB); + bool fScanOk = ScanWalletRecords(visitor, strScanError); + DBFlush(true); + strSetDataDir = strOldDataDir; + nFail += Check(fScanOk && !mapBDB.empty(), + "wallet.dat records can be scanned for comparison") ? 0 : 1; + + CWalletDBSQLite db; + string strError; + nFail += Check(db.Open(strSQLite, strError), + "exported SQLite wallet opens") ? 0 : 1; + int nSQLiteRecords = 0; + nFail += Check(db.CountRecords(nSQLiteRecords, strError) && + nSQLiteRecords == (int)mapBDB.size(), + "SQLite export has the same record count as wallet.dat") ? 0 : 1; + + bool fAllMatch = true; + for (std::map, vector >::const_iterator it = + mapBDB.begin(); it != mapBDB.end(); ++it) + { + vector vchSQLiteValue; + if (!db.ReadRecord(it->first, vchSQLiteValue, strError) || + vchSQLiteValue != it->second) + { + fAllMatch = false; + break; + } + } + nFail += Check(fAllMatch, + "SQLite export preserves every wallet.dat key/value byte-for-byte") ? 0 : 1; + } + catch (const std::exception& e) + { + printf(" FAIL exception: %s\n", e.what()); + nFail++; + } + catch (...) + { + printf(" FAIL unknown exception\n"); + nFail++; + } + + RemoveTree(tmp); + printf("%s (%d failure%s)\n", nFail == 0 ? "ALL TESTS PASSED" : "TESTS FAILED", + nFail, nFail == 1 ? "" : "s"); + fflush(stdout); + return nFail == 0 ? 0 : 1; +} + static void ClearWalletRuntimeForTest() { CRITICAL_BLOCK(cs_mapKeys) @@ -2395,6 +2539,8 @@ int RunSelfTest(const std::string& name) return RunDbEnvReopenSelfTest(); if (name == "wallet-sqlite") return RunWalletSQLiteSelfTest(); + if (name == "wallet-sqlite-migration") + return RunWalletSQLiteMigrationSelfTest(); if (name == "wallet-crypto") return RunWalletCryptoSelfTest(); if (name == "wallet-encrypt") @@ -2415,6 +2561,6 @@ int RunSelfTest(const std::string& name) return RunManagedTorSelfTest(); printf("Unknown self-test '%s'\n", name.c_str()); - printf("Known self-tests: wallet-keypool, wallet-hd, wallet-format, wallet-storage-sanity, db-env-reopen, wallet-sqlite, wallet-crypto, wallet-encrypt, wallet-portability, net-message, consensus-limits, pool-stratum, parse-money, socks5-proxy, managed-tor\n"); + printf("Known self-tests: wallet-keypool, wallet-hd, wallet-format, wallet-storage-sanity, db-env-reopen, wallet-sqlite, wallet-sqlite-migration, wallet-crypto, wallet-encrypt, wallet-portability, net-message, consensus-limits, pool-stratum, parse-money, socks5-proxy, managed-tor\n"); return 1; } diff --git a/src/wallet_sqlite.cpp b/src/wallet_sqlite.cpp index 4104ac6..4369af6 100644 --- a/src/wallet_sqlite.cpp +++ b/src/wallet_sqlite.cpp @@ -97,6 +97,16 @@ bool CWalletDBSQLite::RollbackTransaction(string& strError) return Exec("ROLLBACK;", strError); } +bool CWalletDBSQLite::Checkpoint(string& strError) +{ + if (!pdb) + { + strError = "SQLite wallet is not open"; + return false; + } + return Exec("PRAGMA wal_checkpoint(TRUNCATE);", strError); +} + bool CWalletDBSQLite::Exec(const char* pszSql, string& strError) { char* pszErr = NULL; diff --git a/src/wallet_sqlite.h b/src/wallet_sqlite.h index 9812d3a..5ef862d 100644 --- a/src/wallet_sqlite.h +++ b/src/wallet_sqlite.h @@ -18,6 +18,7 @@ public: bool BeginTransaction(string& strError); bool CommitTransaction(string& strError); bool RollbackTransaction(string& strError); + bool Checkpoint(string& strError); bool WriteRecord(const vector& vchKey, const vector& vchValue, diff --git a/src/walletcmd.cpp b/src/walletcmd.cpp index 6fb09a3..20e7e33 100644 --- a/src/walletcmd.cpp +++ b/src/walletcmd.cpp @@ -12,6 +12,7 @@ #include "headers_core.h" #include "bip32.h" #include "walletcmd.h" +#include "wallet_sqlite.h" #undef printf @@ -790,6 +791,102 @@ int CmdWalletStorageAudit(const std::string& strJsonOut) return 0; } +static vector DataStreamBytesLocal(const CDataStream& ss) +{ + return vector(ss.begin(), ss.end()); +} + +class CWalletSQLiteExportVisitor : public CWalletRecordVisitor +{ +public: + CWalletDBSQLite& db; + int& nCopied; + + CWalletSQLiteExportVisitor(CWalletDBSQLite& dbIn, int& nCopiedIn) + : db(dbIn), nCopied(nCopiedIn) { } + + bool VisitWalletRecord(const CDataStream& ssKey, + const CDataStream& ssValue, + string& strErrorRet) + { + if (!db.WriteRecord(DataStreamBytesLocal(ssKey), + DataStreamBytesLocal(ssValue), + strErrorRet)) + return false; + nCopied++; + return true; + } +}; + +int CmdWalletSQLiteExport(const std::string& strDest) +{ + AttachTerminal(); + + if (strDest.empty()) + { + fprintf(stderr, "Missing SQLite wallet export path.\n"); + return 1; + } + if (FileExists(strDest.c_str())) + { + fprintf(stderr, "Refusing to overwrite existing SQLite wallet export: %s\n", + strDest.c_str()); + return 1; + } + + string strError; + CWalletDBSQLite db; + if (!db.Open(strDest, strError)) + { + fprintf(stderr, "Cannot open SQLite wallet export: %s\n", strError.c_str()); + return 1; + } + if (!db.BeginTransaction(strError)) + { + fprintf(stderr, "Cannot begin SQLite wallet export: %s\n", strError.c_str()); + return 1; + } + + int nCopied = 0; + CWalletSQLiteExportVisitor visitor(db, nCopied); + if (!ScanWalletRecords(visitor, strError)) + { + string strRollbackError; + db.RollbackTransaction(strRollbackError); + fprintf(stderr, "Cannot export wallet.dat to SQLite: %s\n", strError.c_str()); + return 1; + } + if (!db.CommitTransaction(strError)) + { + fprintf(stderr, "Cannot commit SQLite wallet export: %s\n", strError.c_str()); + return 1; + } + if (!db.Checkpoint(strError)) + { + fprintf(stderr, "Cannot checkpoint SQLite wallet export: %s\n", strError.c_str()); + return 1; + } + + int nCount = 0; + if (!db.CountRecords(nCount, strError)) + { + fprintf(stderr, "Cannot verify SQLite wallet export count: %s\n", + strError.c_str()); + return 1; + } + if (nCount != nCopied) + { + fprintf(stderr, "SQLite wallet export count mismatch: copied %d, stored %d\n", + nCopied, nCount); + return 1; + } + + printf("SQLite wallet export written to %s\n", strDest.c_str()); + printf(" records copied: %d\n", nCopied); + fflush(stdout); + return 0; +} + int CmdEncryptWallet(const std::string& strPassphrase) { AttachTerminal(); diff --git a/src/walletcmd.h b/src/walletcmd.h index 6077669..b1a29d2 100644 --- a/src/walletcmd.h +++ b/src/walletcmd.h @@ -70,6 +70,11 @@ int CmdWalletStorageAudit(const std::string& strJsonOut = ""); // or migration-unsafe mix of storage records. int CmdWalletStorageCheck(); +// Diagnostic/migration staging: copy every raw wallet.dat record into an +// experimental SQLite key/value store, preserving serialized bytes. This does +// not change which backend the runtime wallet opens. +int CmdWalletSQLiteExport(const std::string& strDest); + // Rewrite wallet.dat so private keys and the HD seed are encrypted with the // given passphrase. Returns 0 on success and exits without starting the node. int CmdEncryptWallet(const std::string& strPassphrase); -- cgit v1.2.3 From df84b82162b38dcd57114510b9651af5d85f04cb Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Fri, 14 Aug 2026 16:20:41 -0300 Subject: wallet: scan SQLite wallet records --- docs/storage.md | 4 +++- src/selftest.cpp | 6 ++++++ src/wallet_sqlite.cpp | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++ src/wallet_sqlite.h | 3 +++ 4 files changed, 66 insertions(+), 1 deletion(-) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index bb414b5..2d748fa 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -108,7 +108,9 @@ BDB-to-SQLite migration or runtime backend flag exists. `wallet.dat` record through `ScanWalletRecords()` and writes those same key and value bytes into the SQLite table inside one transaction. It refuses to overwrite an existing export and still does not change the runtime wallet -backend. +backend. `CWalletDBSQLite::ScanRecords()` can stream those exported records back +through the same visitor shape, so tests can compare BDB and SQLite record +streams without introducing a SQLite runtime wallet loader yet. ## `blkindex.dat` diff --git a/src/selftest.cpp b/src/selftest.cpp index 7a54271..408dd41 100644 --- a/src/selftest.cpp +++ b/src/selftest.cpp @@ -1259,6 +1259,12 @@ static int RunWalletSQLiteMigrationSelfTest() } nFail += Check(fAllMatch, "SQLite export preserves every wallet.dat key/value byte-for-byte") ? 0 : 1; + + std::map, vector > mapSQLite; + CWalletRecordMapVisitor sqliteVisitor(mapSQLite); + nFail += Check(db.ScanRecords(sqliteVisitor, strError) && + mapSQLite == mapBDB, + "SQLite wallet scanner streams every exported record byte-for-byte") ? 0 : 1; } catch (const std::exception& e) { diff --git a/src/wallet_sqlite.cpp b/src/wallet_sqlite.cpp index 4369af6..8489cac 100644 --- a/src/wallet_sqlite.cpp +++ b/src/wallet_sqlite.cpp @@ -231,6 +231,60 @@ bool CWalletDBSQLite::ReadRecord(const vector& vchKey, return false; } +bool CWalletDBSQLite::ScanRecords(CWalletRecordVisitor& visitor, + string& strError) +{ + if (!pdb) + { + strError = "SQLite wallet is not open"; + return false; + } + + sqlite3_stmt* stmt = NULL; + const char* pszSql = "SELECT key, value FROM wallet_records ORDER BY rowid;"; + int ret = sqlite3_prepare_v2(pdb, pszSql, -1, &stmt, NULL); + if (ret != SQLITE_OK) + { + SetSQLiteError(pdb, "cannot prepare SQLite wallet scan", strError); + return false; + } + + while ((ret = sqlite3_step(stmt)) == SQLITE_ROW) + { + const unsigned char* pchKey = + (const unsigned char*)sqlite3_column_blob(stmt, 0); + int nKeyBytes = sqlite3_column_bytes(stmt, 0); + const unsigned char* pchValue = + (const unsigned char*)sqlite3_column_blob(stmt, 1); + int nValueBytes = sqlite3_column_bytes(stmt, 1); + + vector vchKey; + vector vchValue; + if (nKeyBytes > 0 && pchKey) + vchKey.assign(pchKey, pchKey + nKeyBytes); + if (nValueBytes > 0 && pchValue) + vchValue.assign(pchValue, pchValue + nValueBytes); + + CDataStream ssKey(vchKey, SER_DISK); + CDataStream ssValue(vchValue, SER_DISK); + if (!visitor.VisitWalletRecord(ssKey, ssValue, strError)) + { + sqlite3_finalize(stmt); + return false; + } + } + + if (ret != SQLITE_DONE) + { + SetSQLiteError(pdb, "cannot scan SQLite wallet records", strError); + sqlite3_finalize(stmt); + return false; + } + + sqlite3_finalize(stmt); + return true; +} + bool CWalletDBSQLite::CountRecords(int& nRecordsRet, string& strError) { nRecordsRet = 0; diff --git a/src/wallet_sqlite.h b/src/wallet_sqlite.h index 5ef862d..385a340 100644 --- a/src/wallet_sqlite.h +++ b/src/wallet_sqlite.h @@ -6,6 +6,8 @@ #include +class CWalletRecordVisitor; + class CWalletDBSQLite { public: @@ -26,6 +28,7 @@ public: bool ReadRecord(const vector& vchKey, vector& vchValueRet, string& strError); + bool ScanRecords(CWalletRecordVisitor& visitor, string& strError); bool CountRecords(int& nRecordsRet, string& strError); private: -- cgit v1.2.3 From 99c6b5f2971da51cca1c739b5e982ae2022acc85 Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Fri, 14 Aug 2026 16:34:43 -0300 Subject: wallet: verify SQLite wallet exports --- docs/storage.md | 2 + src/main_gui.cpp | 11 ++++++ src/selftest.cpp | 41 +++++++++++++++++++++ src/walletcmd.cpp | 107 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/walletcmd.h | 4 ++ 5 files changed, 165 insertions(+) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index 2d748fa..0a441bc 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -111,6 +111,8 @@ overwrite an existing export and still does not change the runtime wallet backend. `CWalletDBSQLite::ScanRecords()` can stream those exported records back through the same visitor shape, so tests can compare BDB and SQLite record streams without introducing a SQLite runtime wallet loader yet. +`-walletsqliteverify=FILE` compares the current `wallet.dat` stream with a +SQLite export and prints only record counts and mismatch counts. ## `blkindex.dat` diff --git a/src/main_gui.cpp b/src/main_gui.cpp index 6e89270..caacf13 100644 --- a/src/main_gui.cpp +++ b/src/main_gui.cpp @@ -183,6 +183,8 @@ static void PrintUsage() printf(" wallet.dat storage state)\n"); printf(" /walletsqliteexport=FILE (copy raw wallet.dat records into an\n"); printf(" experimental SQLite store, then exit)\n"); + printf(" /walletsqliteverify=FILE (compare wallet.dat with a SQLite export,\n"); + printf(" printing counts only, then exit)\n"); printf(" /rescan (walk the chain for coins this wallet owns\n"); printf(" but never recorded, then exit)\n"); printf("\n"); @@ -540,6 +542,15 @@ int main(int argc, char* argv[]) return nRet; } + string strWalletSQLiteVerify = + argval2(argc, argv, "/walletsqliteverify", "-walletsqliteverify"); + if (!strWalletSQLiteVerify.empty()) + { + int nRet = CmdWalletSQLiteVerify(strWalletSQLiteVerify); + DBFlush(true); + return nRet; + } + // Berkeley DB reports failure by throwing, and CDB's constructor lets it // through. Nothing on this path caught anything, so an unreadable // blkindex.dat or wallet.dat -- one written by another platform's Berkeley diff --git a/src/selftest.cpp b/src/selftest.cpp index 408dd41..4ba90c3 100644 --- a/src/selftest.cpp +++ b/src/selftest.cpp @@ -1134,6 +1134,8 @@ static int RunWalletSQLiteSelfTest() return nFail == 0 ? 0 : 1; } +namespace { + class CWalletRecordMapVisitor : public CWalletRecordVisitor { public: @@ -1158,6 +1160,8 @@ public: } }; +} // namespace + static int RunWalletSQLiteMigrationSelfTest() { fflush(stdout); @@ -1215,6 +1219,17 @@ static int RunWalletSQLiteMigrationSelfTest() FileContainsText(strExportOut, "SQLite wallet export written"), "wallet.dat can be exported to SQLite") ? 0 : 1; + string strVerifyOut = tmp + "/sqlite-verify.txt"; + vector vVerifyArgs; + vVerifyArgs.push_back("-datadir=" + strWalletDir); + vVerifyArgs.push_back("-nomanagedtor"); + vVerifyArgs.push_back("-nogui"); + vVerifyArgs.push_back("-walletsqliteverify=" + strSQLite); + int nVerifyRet = RunBitflashChild(strExe, vVerifyArgs, NULL, &strVerifyOut); + nFail += Check(nVerifyRet == 0 && + FileContainsText(strVerifyOut, "verification: ok"), + "SQLite wallet export verifies against wallet.dat") ? 0 : 1; + string strOverwriteOut = tmp + "/sqlite-export-overwrite.txt"; int nOverwriteRet = RunBitflashChild(strExe, vExportArgs, NULL, &strOverwriteOut); @@ -1265,6 +1280,32 @@ static int RunWalletSQLiteMigrationSelfTest() nFail += Check(db.ScanRecords(sqliteVisitor, strError) && mapSQLite == mapBDB, "SQLite wallet scanner streams every exported record byte-for-byte") ? 0 : 1; + + db.Close(); + CWalletDBSQLite dbCorrupt; + vector vchBadValue; + vchBadValue.push_back(0xba); + vchBadValue.push_back(0xad); + nFail += Check(!mapBDB.empty() && + dbCorrupt.Open(strSQLite, strError) && + dbCorrupt.BeginTransaction(strError) && + dbCorrupt.WriteRecord(mapBDB.begin()->first, + vchBadValue, + strError) && + dbCorrupt.CommitTransaction(strError) && + dbCorrupt.Checkpoint(strError), + "SQLite export can be deliberately corrupted for verifier test") ? 0 : 1; + dbCorrupt.Close(); + + string strVerifyBadOut = tmp + "/sqlite-verify-bad.txt"; + int nVerifyBadRet = + RunBitflashChild(strExe, vVerifyArgs, NULL, &strVerifyBadOut); + nFail += Check(nVerifyBadRet == 2 && + FileContainsText(strVerifyBadOut, + "record value mismatches: 1") && + FileContainsText(strVerifyBadOut, + "verification: failed"), + "SQLite wallet verifier detects a mismatched record value") ? 0 : 1; } catch (const std::exception& e) { diff --git a/src/walletcmd.cpp b/src/walletcmd.cpp index 20e7e33..36f6783 100644 --- a/src/walletcmd.cpp +++ b/src/walletcmd.cpp @@ -796,6 +796,8 @@ static vector DataStreamBytesLocal(const CDataStream& ss) return vector(ss.begin(), ss.end()); } +namespace { + class CWalletSQLiteExportVisitor : public CWalletRecordVisitor { public: @@ -818,6 +820,32 @@ public: } }; +class CWalletRecordMapVisitor : public CWalletRecordVisitor +{ +public: + std::map, vector >& mapRecords; + + explicit CWalletRecordMapVisitor( + std::map, vector >& mapRecordsIn) + : mapRecords(mapRecordsIn) { } + + bool VisitWalletRecord(const CDataStream& ssKey, + const CDataStream& ssValue, + string& strErrorRet) + { + vector vchKey = DataStreamBytesLocal(ssKey); + if (mapRecords.count(vchKey)) + { + strErrorRet = "duplicate serialized wallet key"; + return false; + } + mapRecords[vchKey] = DataStreamBytesLocal(ssValue); + return true; + } +}; + +} // namespace + int CmdWalletSQLiteExport(const std::string& strDest) { AttachTerminal(); @@ -887,6 +915,85 @@ int CmdWalletSQLiteExport(const std::string& strDest) return 0; } +int CmdWalletSQLiteVerify(const std::string& strPath) +{ + AttachTerminal(); + + if (strPath.empty()) + { + fprintf(stderr, "Missing SQLite wallet export path.\n"); + return 1; + } + if (!FileExists(strPath.c_str())) + { + fprintf(stderr, "SQLite wallet export does not exist: %s\n", + strPath.c_str()); + return 1; + } + + std::map, vector > mapBDB; + std::map, vector > mapSQLite; + string strError; + + CWalletRecordMapVisitor bdbVisitor(mapBDB); + if (!ScanWalletRecords(bdbVisitor, strError)) + { + fprintf(stderr, "Cannot scan wallet.dat records: %s\n", strError.c_str()); + return 1; + } + + CWalletDBSQLite db; + if (!db.Open(strPath, strError)) + { + fprintf(stderr, "Cannot open SQLite wallet export: %s\n", strError.c_str()); + return 1; + } + CWalletRecordMapVisitor sqliteVisitor(mapSQLite); + if (!db.ScanRecords(sqliteVisitor, strError)) + { + fprintf(stderr, "Cannot scan SQLite wallet export: %s\n", strError.c_str()); + return 1; + } + + int nMissing = 0; + int nMismatched = 0; + for (std::map, vector >::const_iterator it = + mapBDB.begin(); it != mapBDB.end(); ++it) + { + std::map, vector >::const_iterator sit = + mapSQLite.find(it->first); + if (sit == mapSQLite.end()) + nMissing++; + else if (sit->second != it->second) + nMismatched++; + } + + int nExtra = 0; + for (std::map, vector >::const_iterator it = + mapSQLite.begin(); it != mapSQLite.end(); ++it) + { + if (!mapBDB.count(it->first)) + nExtra++; + } + + printf("SQLite wallet export verification\n"); + printf(" wallet.dat records: %u\n", (unsigned int)mapBDB.size()); + printf(" SQLite records: %u\n", (unsigned int)mapSQLite.size()); + printf(" missing records: %d\n", nMissing); + printf(" extra records: %d\n", nExtra); + printf(" record value mismatches: %d\n", nMismatched); + if (nMissing == 0 && nExtra == 0 && nMismatched == 0) + { + printf(" verification: ok\n"); + fflush(stdout); + return 0; + } + + printf(" verification: failed\n"); + fflush(stdout); + return 2; +} + int CmdEncryptWallet(const std::string& strPassphrase) { AttachTerminal(); diff --git a/src/walletcmd.h b/src/walletcmd.h index b1a29d2..784c74c 100644 --- a/src/walletcmd.h +++ b/src/walletcmd.h @@ -75,6 +75,10 @@ int CmdWalletStorageCheck(); // not change which backend the runtime wallet opens. int CmdWalletSQLiteExport(const std::string& strDest); +// Diagnostic/migration staging: compare a SQLite wallet export with the current +// wallet.dat raw records. Prints counts only, never record keys or values. +int CmdWalletSQLiteVerify(const std::string& strPath); + // Rewrite wallet.dat so private keys and the HD seed are encrypted with the // given passphrase. Returns 0 on success and exits without starting the node. int CmdEncryptWallet(const std::string& strPassphrase); -- cgit v1.2.3 From df129cc672b9b894ff2558d3e9361ddab082433d Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Fri, 14 Aug 2026 17:02:35 -0300 Subject: wallet: restore BDB wallets from SQLite exports --- docs/storage.md | 3 ++ src/main_gui.cpp | 11 +++++ src/selftest.cpp | 35 +++++++++++++++ src/walletcmd.cpp | 126 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/walletcmd.h | 4 ++ 5 files changed, 179 insertions(+) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index 0a441bc..4947575 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -113,6 +113,9 @@ through the same visitor shape, so tests can compare BDB and SQLite record streams without introducing a SQLite runtime wallet loader yet. `-walletsqliteverify=FILE` compares the current `wallet.dat` stream with a SQLite export and prints only record counts and mismatch counts. +`-walletsqliterestore=FILE` rebuilds `wallet.dat` from a SQLite export only in +an empty data directory, then the verifier can prove the restored BDB file still +matches the export byte-for-byte. ## `blkindex.dat` diff --git a/src/main_gui.cpp b/src/main_gui.cpp index caacf13..cf96822 100644 --- a/src/main_gui.cpp +++ b/src/main_gui.cpp @@ -185,6 +185,8 @@ static void PrintUsage() printf(" experimental SQLite store, then exit)\n"); printf(" /walletsqliteverify=FILE (compare wallet.dat with a SQLite export,\n"); printf(" printing counts only, then exit)\n"); + printf(" /walletsqliterestore=FILE (rebuild wallet.dat from a SQLite export in\n"); + printf(" an empty data directory, then exit)\n"); printf(" /rescan (walk the chain for coins this wallet owns\n"); printf(" but never recorded, then exit)\n"); printf("\n"); @@ -551,6 +553,15 @@ int main(int argc, char* argv[]) return nRet; } + string strWalletSQLiteRestore = + argval2(argc, argv, "/walletsqliterestore", "-walletsqliterestore"); + if (!strWalletSQLiteRestore.empty()) + { + int nRet = CmdWalletSQLiteRestore(strWalletSQLiteRestore); + DBFlush(true); + return nRet; + } + // Berkeley DB reports failure by throwing, and CDB's constructor lets it // through. Nothing on this path caught anything, so an unreadable // blkindex.dat or wallet.dat -- one written by another platform's Berkeley diff --git a/src/selftest.cpp b/src/selftest.cpp index 4ba90c3..7d3a96e 100644 --- a/src/selftest.cpp +++ b/src/selftest.cpp @@ -1230,6 +1230,41 @@ static int RunWalletSQLiteMigrationSelfTest() FileContainsText(strVerifyOut, "verification: ok"), "SQLite wallet export verifies against wallet.dat") ? 0 : 1; + string strRestoredDir = tmp + "/restored-wallet"; + nFail += Check(MakeDirLocal(strRestoredDir), + "restored wallet directory can be created") ? 0 : 1; + string strRestoreOut = tmp + "/sqlite-restore.txt"; + vector vRestoreArgs; + vRestoreArgs.push_back("-datadir=" + strRestoredDir); + vRestoreArgs.push_back("-nomanagedtor"); + vRestoreArgs.push_back("-nogui"); + vRestoreArgs.push_back("-walletsqliterestore=" + strSQLite); + int nRestoreRet = RunBitflashChild(strExe, vRestoreArgs, NULL, &strRestoreOut); + nFail += Check(nRestoreRet == 0 && + FileContainsText(strRestoreOut, "records restored:"), + "SQLite wallet export restores into a fresh wallet.dat") ? 0 : 1; + + string strVerifyRestoredOut = tmp + "/sqlite-verify-restored.txt"; + vector vVerifyRestoredArgs; + vVerifyRestoredArgs.push_back("-datadir=" + strRestoredDir); + vVerifyRestoredArgs.push_back("-nomanagedtor"); + vVerifyRestoredArgs.push_back("-nogui"); + vVerifyRestoredArgs.push_back("-walletsqliteverify=" + strSQLite); + int nVerifyRestoredRet = + RunBitflashChild(strExe, vVerifyRestoredArgs, NULL, &strVerifyRestoredOut); + nFail += Check(nVerifyRestoredRet == 0 && + FileContainsText(strVerifyRestoredOut, + "verification: ok"), + "restored wallet.dat verifies against the SQLite export") ? 0 : 1; + + string strRestoreOverwriteOut = tmp + "/sqlite-restore-overwrite.txt"; + int nRestoreOverwriteRet = + RunBitflashChild(strExe, vRestoreArgs, NULL, &strRestoreOverwriteOut); + nFail += Check(nRestoreOverwriteRet == 1 && + FileContainsText(strRestoreOverwriteOut, + "Refusing to overwrite existing wallet.dat"), + "SQLite wallet restore refuses to overwrite wallet.dat") ? 0 : 1; + string strOverwriteOut = tmp + "/sqlite-export-overwrite.txt"; int nOverwriteRet = RunBitflashChild(strExe, vExportArgs, NULL, &strOverwriteOut); diff --git a/src/walletcmd.cpp b/src/walletcmd.cpp index 36f6783..d1bad98 100644 --- a/src/walletcmd.cpp +++ b/src/walletcmd.cpp @@ -844,6 +844,52 @@ public: } }; +class CWalletRawRestoreDB : public CWalletDB +{ +public: + CWalletRawRestoreDB(const char* pszMode="cr+", bool fTxn=true) + : CWalletDB(pszMode, fTxn) { } + + bool WriteRaw(CDataStream ssKey, CDataStream ssValue) + { + if (!pdb || ssKey.empty()) + return false; + + Dbt datKey((void*)&ssKey[0], ssKey.size()); + Dbt datValue(ssValue.empty() ? NULL : (void*)&ssValue[0], + ssValue.size()); + int ret = pdb->put(GetTxn(), &datKey, &datValue, 0); + + memset(datKey.get_data(), 0, datKey.get_size()); + if (datValue.get_data()) + memset(datValue.get_data(), 0, datValue.get_size()); + return ret == 0; + } +}; + +class CWalletSQLiteRestoreVisitor : public CWalletRecordVisitor +{ +public: + CWalletRawRestoreDB& db; + int& nCopied; + + CWalletSQLiteRestoreVisitor(CWalletRawRestoreDB& dbIn, int& nCopiedIn) + : db(dbIn), nCopied(nCopiedIn) { } + + bool VisitWalletRecord(const CDataStream& ssKey, + const CDataStream& ssValue, + string& strErrorRet) + { + if (!db.WriteRaw(ssKey, ssValue)) + { + strErrorRet = "could not write raw wallet record"; + return false; + } + nCopied++; + return true; + } +}; + } // namespace int CmdWalletSQLiteExport(const std::string& strDest) @@ -994,6 +1040,86 @@ int CmdWalletSQLiteVerify(const std::string& strPath) return 2; } +int CmdWalletSQLiteRestore(const std::string& strPath) +{ + AttachTerminal(); + + if (strPath.empty()) + { + fprintf(stderr, "Missing SQLite wallet export path.\n"); + return 1; + } + if (!FileExists(strPath.c_str())) + { + fprintf(stderr, "SQLite wallet export does not exist: %s\n", + strPath.c_str()); + return 1; + } + + string strWalletPath = GetAppDir() + "/wallet.dat"; + if (FileExists(strWalletPath.c_str())) + { + fprintf(stderr, "Refusing to overwrite existing wallet.dat: %s\n", + strWalletPath.c_str()); + return 1; + } + + string strError; + CWalletDBSQLite sqlite; + if (!sqlite.Open(strPath, strError)) + { + fprintf(stderr, "Cannot open SQLite wallet export: %s\n", strError.c_str()); + return 1; + } + + int nSQLiteRecords = 0; + if (!sqlite.CountRecords(nSQLiteRecords, strError)) + { + fprintf(stderr, "Cannot count SQLite wallet export records: %s\n", + strError.c_str()); + return 1; + } + if (nSQLiteRecords <= 0) + { + fprintf(stderr, "SQLite wallet export contains no records.\n"); + return 2; + } + + CWalletRawRestoreDB bdb("cr+", true); + if (!bdb.TxnBegin()) + { + fprintf(stderr, "Cannot begin Berkeley DB restore transaction.\n"); + return 1; + } + + int nCopied = 0; + CWalletSQLiteRestoreVisitor visitor(bdb, nCopied); + if (!sqlite.ScanRecords(visitor, strError)) + { + bdb.TxnAbort(); + fprintf(stderr, "Cannot restore SQLite wallet export: %s\n", + strError.c_str()); + return 1; + } + if (nCopied != nSQLiteRecords) + { + bdb.TxnAbort(); + fprintf(stderr, "SQLite wallet restore copied %d records, expected %d.\n", + nCopied, nSQLiteRecords); + return 2; + } + if (!bdb.TxnCommit()) + { + fprintf(stderr, "Cannot commit Berkeley DB restore transaction.\n"); + return 1; + } + + printf("SQLite wallet export restored to wallet.dat\n"); + printf(" records restored: %d\n", nCopied); + fflush(stdout); + return 0; +} + int CmdEncryptWallet(const std::string& strPassphrase) { AttachTerminal(); diff --git a/src/walletcmd.h b/src/walletcmd.h index 784c74c..50dc69e 100644 --- a/src/walletcmd.h +++ b/src/walletcmd.h @@ -79,6 +79,10 @@ int CmdWalletSQLiteExport(const std::string& strDest); // wallet.dat raw records. Prints counts only, never record keys or values. int CmdWalletSQLiteVerify(const std::string& strPath); +// Diagnostic/migration staging: rebuild wallet.dat from a SQLite export in an +// empty data directory. Refuses to overwrite an existing wallet.dat. +int CmdWalletSQLiteRestore(const std::string& strPath); + // Rewrite wallet.dat so private keys and the HD seed are encrypted with the // given passphrase. Returns 0 on success and exits without starting the node. int CmdEncryptWallet(const std::string& strPassphrase); -- cgit v1.2.3 From f7f9885e89e2c2527933b99ec713592ebf930483 Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Fri, 14 Aug 2026 17:23:21 -0300 Subject: wallet: clean up failed SQLite migrations --- docs/storage.md | 3 ++- src/selftest.cpp | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ src/walletcmd.cpp | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 87 insertions(+), 1 deletion(-) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index 4947575..aca3f6e 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -115,7 +115,8 @@ streams without introducing a SQLite runtime wallet loader yet. SQLite export and prints only record counts and mismatch counts. `-walletsqliterestore=FILE` rebuilds `wallet.dat` from a SQLite export only in an empty data directory, then the verifier can prove the restored BDB file still -matches the export byte-for-byte. +matches the export byte-for-byte. Failed exports remove the incomplete SQLite +file and failed restores remove the incomplete `wallet.dat`. ## `blkindex.dat` diff --git a/src/selftest.cpp b/src/selftest.cpp index 7d3a96e..343d9c5 100644 --- a/src/selftest.cpp +++ b/src/selftest.cpp @@ -1162,6 +1162,39 @@ public: } // namespace +static bool CreateSQLiteWalletWithEmptyKey(const string& strPath) +{ + sqlite3* pdb = NULL; + int ret = sqlite3_open_v2(strPath.c_str(), &pdb, + SQLITE_OPEN_READWRITE | + SQLITE_OPEN_CREATE | + SQLITE_OPEN_FULLMUTEX, + NULL); + if (ret != SQLITE_OK || !pdb) + { + if (pdb) + sqlite3_close(pdb); + return false; + } + + char* pszErr = NULL; + bool fOk = + sqlite3_exec(pdb, + "CREATE TABLE wallet_records (" + "key BLOB PRIMARY KEY NOT NULL," + "value BLOB NOT NULL" + ");", + NULL, NULL, &pszErr) == SQLITE_OK && + sqlite3_exec(pdb, + "INSERT INTO wallet_records(key, value) " + "VALUES(x'', x'01');", + NULL, NULL, &pszErr) == SQLITE_OK; + if (pszErr) + sqlite3_free(pszErr); + sqlite3_close(pdb); + return fOk; +} + static int RunWalletSQLiteMigrationSelfTest() { fflush(stdout); @@ -1265,6 +1298,26 @@ static int RunWalletSQLiteMigrationSelfTest() "Refusing to overwrite existing wallet.dat"), "SQLite wallet restore refuses to overwrite wallet.dat") ? 0 : 1; + string strBadSQLite = tmp + "/bad-empty-key.sqlite"; + nFail += Check(CreateSQLiteWalletWithEmptyKey(strBadSQLite), + "bad SQLite export with an empty key can be created") ? 0 : 1; + string strBadRestoreDir = tmp + "/bad-restore-wallet"; + nFail += Check(MakeDirLocal(strBadRestoreDir), + "bad restore wallet directory can be created") ? 0 : 1; + string strBadRestoreOut = tmp + "/sqlite-restore-bad.txt"; + vector vBadRestoreArgs; + vBadRestoreArgs.push_back("-datadir=" + strBadRestoreDir); + vBadRestoreArgs.push_back("-nomanagedtor"); + vBadRestoreArgs.push_back("-nogui"); + vBadRestoreArgs.push_back("-walletsqliterestore=" + strBadSQLite); + int nBadRestoreRet = + RunBitflashChild(strExe, vBadRestoreArgs, NULL, &strBadRestoreOut); + nFail += Check(nBadRestoreRet == 1 && + FileContainsText(strBadRestoreOut, + "Cannot restore SQLite wallet export") && + !FileExists((strBadRestoreDir + "/wallet.dat").c_str()), + "failed SQLite wallet restore removes incomplete wallet.dat") ? 0 : 1; + string strOverwriteOut = tmp + "/sqlite-export-overwrite.txt"; int nOverwriteRet = RunBitflashChild(strExe, vExportArgs, NULL, &strOverwriteOut); diff --git a/src/walletcmd.cpp b/src/walletcmd.cpp index d1bad98..9e4a981 100644 --- a/src/walletcmd.cpp +++ b/src/walletcmd.cpp @@ -798,6 +798,19 @@ static vector DataStreamBytesLocal(const CDataStream& ss) namespace { +static void RemoveSQLiteExportFiles(const string& strPath) +{ + remove(strPath.c_str()); + remove((strPath + "-wal").c_str()); + remove((strPath + "-shm").c_str()); +} + +static void RemoveFailedRestoreWallet(const string& strWalletPath) +{ + DBFlush(true); + remove(strWalletPath.c_str()); +} + class CWalletSQLiteExportVisitor : public CWalletRecordVisitor { public: @@ -912,11 +925,14 @@ int CmdWalletSQLiteExport(const std::string& strDest) CWalletDBSQLite db; if (!db.Open(strDest, strError)) { + RemoveSQLiteExportFiles(strDest); fprintf(stderr, "Cannot open SQLite wallet export: %s\n", strError.c_str()); return 1; } if (!db.BeginTransaction(strError)) { + db.Close(); + RemoveSQLiteExportFiles(strDest); fprintf(stderr, "Cannot begin SQLite wallet export: %s\n", strError.c_str()); return 1; } @@ -927,16 +943,22 @@ int CmdWalletSQLiteExport(const std::string& strDest) { string strRollbackError; db.RollbackTransaction(strRollbackError); + db.Close(); + RemoveSQLiteExportFiles(strDest); fprintf(stderr, "Cannot export wallet.dat to SQLite: %s\n", strError.c_str()); return 1; } if (!db.CommitTransaction(strError)) { + db.Close(); + RemoveSQLiteExportFiles(strDest); fprintf(stderr, "Cannot commit SQLite wallet export: %s\n", strError.c_str()); return 1; } if (!db.Checkpoint(strError)) { + db.Close(); + RemoveSQLiteExportFiles(strDest); fprintf(stderr, "Cannot checkpoint SQLite wallet export: %s\n", strError.c_str()); return 1; } @@ -944,12 +966,16 @@ int CmdWalletSQLiteExport(const std::string& strDest) int nCount = 0; if (!db.CountRecords(nCount, strError)) { + db.Close(); + RemoveSQLiteExportFiles(strDest); fprintf(stderr, "Cannot verify SQLite wallet export count: %s\n", strError.c_str()); return 1; } if (nCount != nCopied) { + db.Close(); + RemoveSQLiteExportFiles(strDest); fprintf(stderr, "SQLite wallet export count mismatch: copied %d, stored %d\n", nCopied, nCount); return 1; @@ -1097,6 +1123,8 @@ int CmdWalletSQLiteRestore(const std::string& strPath) if (!sqlite.ScanRecords(visitor, strError)) { bdb.TxnAbort(); + bdb.Close(); + RemoveFailedRestoreWallet(strWalletPath); fprintf(stderr, "Cannot restore SQLite wallet export: %s\n", strError.c_str()); return 1; @@ -1104,12 +1132,16 @@ int CmdWalletSQLiteRestore(const std::string& strPath) if (nCopied != nSQLiteRecords) { bdb.TxnAbort(); + bdb.Close(); + RemoveFailedRestoreWallet(strWalletPath); fprintf(stderr, "SQLite wallet restore copied %d records, expected %d.\n", nCopied, nSQLiteRecords); return 2; } if (!bdb.TxnCommit()) { + bdb.Close(); + RemoveFailedRestoreWallet(strWalletPath); fprintf(stderr, "Cannot commit Berkeley DB restore transaction.\n"); return 1; } -- cgit v1.2.3 From 3e778dd6cc9045862e99503b5b9af80ec682826f Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Fri, 14 Aug 2026 17:44:09 -0300 Subject: wallet: check SQLite exports against loader types --- docs/storage.md | 3 + src/main_gui.cpp | 11 +++ src/selftest.cpp | 54 ++++++++++++++ src/walletcmd.cpp | 209 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/walletcmd.h | 4 ++ 5 files changed, 281 insertions(+) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index aca3f6e..07c7170 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -117,6 +117,9 @@ SQLite export and prints only record counts and mismatch counts. an empty data directory, then the verifier can prove the restored BDB file still matches the export byte-for-byte. Failed exports remove the incomplete SQLite file and failed restores remove the incomplete `wallet.dat`. +`-walletsqliteloadcheck=FILE` parses a SQLite export using the same record +types the wallet loader understands, so a migration can prove the export is not +just byte-preserving but loader-compatible before runtime activation. ## `blkindex.dat` diff --git a/src/main_gui.cpp b/src/main_gui.cpp index cf96822..c9aac7d 100644 --- a/src/main_gui.cpp +++ b/src/main_gui.cpp @@ -187,6 +187,8 @@ static void PrintUsage() printf(" printing counts only, then exit)\n"); printf(" /walletsqliterestore=FILE (rebuild wallet.dat from a SQLite export in\n"); printf(" an empty data directory, then exit)\n"); + printf(" /walletsqliteloadcheck=FILE\n"); + printf(" (parse a SQLite export like the wallet loader)\n"); printf(" /rescan (walk the chain for coins this wallet owns\n"); printf(" but never recorded, then exit)\n"); printf("\n"); @@ -562,6 +564,15 @@ int main(int argc, char* argv[]) return nRet; } + string strWalletSQLiteLoadCheck = + argval2(argc, argv, "/walletsqliteloadcheck", "-walletsqliteloadcheck"); + if (!strWalletSQLiteLoadCheck.empty()) + { + int nRet = CmdWalletSQLiteLoadCheck(strWalletSQLiteLoadCheck); + DBFlush(true); + return nRet; + } + // Berkeley DB reports failure by throwing, and CDB's constructor lets it // through. Nothing on this path caught anything, so an unreadable // blkindex.dat or wallet.dat -- one written by another platform's Berkeley diff --git a/src/selftest.cpp b/src/selftest.cpp index 343d9c5..feb4076 100644 --- a/src/selftest.cpp +++ b/src/selftest.cpp @@ -1195,6 +1195,28 @@ static bool CreateSQLiteWalletWithEmptyKey(const string& strPath) return fOk; } +static bool CreateSQLiteWalletWithFutureMinVersion(const string& strPath) +{ + string strError; + CWalletDBSQLite db; + CDataStream ssKey(SER_DISK); + ssKey << string("walletminversion"); + CDataStream ssValue(SER_DISK); + ssValue << (WALLET_FORMAT_SUPPORTED + 1); + + if (!db.Open(strPath, strError)) + return false; + if (!db.BeginTransaction(strError)) + return false; + if (!db.WriteRecord(DataStreamBytes(ssKey), DataStreamBytes(ssValue), strError)) + return false; + if (!db.CommitTransaction(strError)) + return false; + if (!db.Checkpoint(strError)) + return false; + return true; +} + static int RunWalletSQLiteMigrationSelfTest() { fflush(stdout); @@ -1263,6 +1285,19 @@ static int RunWalletSQLiteMigrationSelfTest() FileContainsText(strVerifyOut, "verification: ok"), "SQLite wallet export verifies against wallet.dat") ? 0 : 1; + string strLoadCheckOut = tmp + "/sqlite-loadcheck.txt"; + vector vLoadCheckArgs; + vLoadCheckArgs.push_back("-datadir=" + strWalletDir); + vLoadCheckArgs.push_back("-nomanagedtor"); + vLoadCheckArgs.push_back("-nogui"); + vLoadCheckArgs.push_back("-walletsqliteloadcheck=" + strSQLite); + int nLoadCheckRet = + RunBitflashChild(strExe, vLoadCheckArgs, NULL, &strLoadCheckOut); + nFail += Check(nLoadCheckRet == 0 && + FileContainsText(strLoadCheckOut, + "load check: ok"), + "SQLite wallet export passes the loader compatibility check") ? 0 : 1; + string strRestoredDir = tmp + "/restored-wallet"; nFail += Check(MakeDirLocal(strRestoredDir), "restored wallet directory can be created") ? 0 : 1; @@ -1318,6 +1353,25 @@ static int RunWalletSQLiteMigrationSelfTest() !FileExists((strBadRestoreDir + "/wallet.dat").c_str()), "failed SQLite wallet restore removes incomplete wallet.dat") ? 0 : 1; + string strFutureSQLite = tmp + "/future-wallet.sqlite"; + nFail += Check(CreateSQLiteWalletWithFutureMinVersion(strFutureSQLite), + "SQLite export with a future wallet format can be created") ? 0 : 1; + string strFutureLoadCheckOut = tmp + "/sqlite-loadcheck-future.txt"; + vector vFutureLoadCheckArgs; + vFutureLoadCheckArgs.push_back("-datadir=" + strWalletDir); + vFutureLoadCheckArgs.push_back("-nomanagedtor"); + vFutureLoadCheckArgs.push_back("-nogui"); + vFutureLoadCheckArgs.push_back("-walletsqliteloadcheck=" + strFutureSQLite); + int nFutureLoadCheckRet = + RunBitflashChild(strExe, vFutureLoadCheckArgs, NULL, + &strFutureLoadCheckOut); + nFail += Check(nFutureLoadCheckRet == 2 && + FileContainsText(strFutureLoadCheckOut, + "unsupported future wallet format") && + FileContainsText(strFutureLoadCheckOut, + "load check: failed"), + "SQLite wallet load check rejects future wallet formats") ? 0 : 1; + string strOverwriteOut = tmp + "/sqlite-export-overwrite.txt"; int nOverwriteRet = RunBitflashChild(strExe, vExportArgs, NULL, &strOverwriteOut); diff --git a/src/walletcmd.cpp b/src/walletcmd.cpp index 9e4a981..4dda8dd 100644 --- a/src/walletcmd.cpp +++ b/src/walletcmd.cpp @@ -903,6 +903,171 @@ public: } }; +class CWalletSQLiteLoadCheckVisitor : public CWalletRecordVisitor +{ +public: + int nRecords; + int nUnknown; + string strFailure; + + CWalletSQLiteLoadCheckVisitor() : nRecords(0), nUnknown(0) { } + + bool Fail(const string& strMessage, string& strErrorRet) + { + strFailure = strMessage; + strErrorRet = strMessage; + return false; + } + + bool VisitWalletRecord(const CDataStream& ssKeyIn, + const CDataStream& ssValueIn, + string& strErrorRet) + { + nRecords++; + try + { + CDataStream ssKey = ssKeyIn; + CDataStream ssValue = ssValueIn; + string strType; + ssKey >> strType; + if (ssKey.fail()) + return Fail("malformed wallet record key", strErrorRet); + + if (strType == "name") + { + string strAddress; + string strName; + ssKey >> strAddress; + ssValue >> strName; + } + else if (strType == "tx") + { + uint256 hash; + CWalletTx wtx; + ssKey >> hash; + ssValue >> wtx; + } + else if (strType == "key") + { + vector vchPubKey; + CPrivKey vchPrivKey; + ssKey >> vchPubKey; + ssValue >> vchPrivKey; + } + else if (strType == "mkey") + { + unsigned int nID = 0; + CWalletMasterKey kMasterKey; + ssKey >> nID; + ssValue >> kMasterKey; + } + else if (strType == "ckey") + { + vector vchPubKey; + vector vchCryptedSecret; + ssKey >> vchPubKey; + ssValue >> vchCryptedSecret; + } + else if (strType == "defaultkey" || + strType == "hdmaster" || + strType == "hdchaincode" || + strType == "cryptedhdmaster" || + strType == "cryptedhdchaincode") + { + vector vch; + ssValue >> vch; + } + else if (strType == "hdnext" || + strType == "hdreceivenext" || + strType == "hdchangenext" || + strType == "hdcointype") + { + unsigned int nValue = 0; + ssValue >> nValue; + } + else if (strType == "hdschema") + { + int nSchema = 0; + ssValue >> nSchema; + if (!ssValue.fail() && + nSchema != HD_SCHEMA_NONE && + nSchema != HD_SCHEMA_LEGACY && + nSchema != HD_SCHEMA_BIP44) + return Fail("unsupported deterministic wallet schema", + strErrorRet); + } + else if (strType == "walletminversion") + { + int nMinVersion = 0; + ssValue >> nMinVersion; + if (!ssValue.fail() && nMinVersion > WALLET_FORMAT_SUPPORTED) + return Fail("unsupported future wallet format", strErrorRet); + } + else if (strType == "pool") + { + int64 nIndex = 0; + vector vchPubKey; + ssKey >> nIndex; + ssValue >> vchPubKey; + } + else if (strType == "setting") + { + string strKey; + ssKey >> strKey; + if (strKey == "nTransactionFee") + { + int64 nValue = 0; + ssValue >> nValue; + } + else if (strKey == "addrIncoming") + { + CAddress addr; + ssValue >> addr; + } + else if (strKey == "nMineMode") + { + int nValue = 0; + ssValue >> nValue; + } + else if (strKey == "fGenerateBitcoins") + { + int fValue = 0; + ssValue >> fValue; + } + else if (strKey == "strParticipantPool") + { + string strValue; + ssValue >> strValue; + } + } + else if (strType == "version") + { + int nVersion = 0; + ssValue >> nVersion; + } + else + { + nUnknown++; + } + + if (ssKey.fail()) + return Fail("malformed wallet record key", strErrorRet); + if (ssValue.fail()) + return Fail("malformed wallet record value", strErrorRet); + } + catch (const std::exception& e) + { + return Fail(strprintf("wallet record parse exception: %s", e.what()), + strErrorRet); + } + catch (...) + { + return Fail("wallet record parse exception", strErrorRet); + } + return true; + } +}; + } // namespace int CmdWalletSQLiteExport(const std::string& strDest) @@ -1152,6 +1317,50 @@ int CmdWalletSQLiteRestore(const std::string& strPath) return 0; } +int CmdWalletSQLiteLoadCheck(const std::string& strPath) +{ + AttachTerminal(); + + if (strPath.empty()) + { + fprintf(stderr, "Missing SQLite wallet export path.\n"); + return 1; + } + if (!FileExists(strPath.c_str())) + { + fprintf(stderr, "SQLite wallet export does not exist: %s\n", + strPath.c_str()); + return 1; + } + + string strError; + CWalletDBSQLite sqlite; + if (!sqlite.Open(strPath, strError)) + { + fprintf(stderr, "Cannot open SQLite wallet export: %s\n", strError.c_str()); + return 1; + } + + CWalletSQLiteLoadCheckVisitor visitor; + if (!sqlite.ScanRecords(visitor, strError)) + { + printf("SQLite wallet load check\n"); + printf(" records checked: %d\n", visitor.nRecords); + printf(" unknown records: %d\n", visitor.nUnknown); + printf(" load check: failed\n"); + printf(" failure: %s\n", strError.c_str()); + fflush(stdout); + return 2; + } + + printf("SQLite wallet load check\n"); + printf(" records checked: %d\n", visitor.nRecords); + printf(" unknown records: %d\n", visitor.nUnknown); + printf(" load check: ok\n"); + fflush(stdout); + return 0; +} + int CmdEncryptWallet(const std::string& strPassphrase) { AttachTerminal(); diff --git a/src/walletcmd.h b/src/walletcmd.h index 50dc69e..d993856 100644 --- a/src/walletcmd.h +++ b/src/walletcmd.h @@ -83,6 +83,10 @@ int CmdWalletSQLiteVerify(const std::string& strPath); // empty data directory. Refuses to overwrite an existing wallet.dat. int CmdWalletSQLiteRestore(const std::string& strPath); +// Diagnostic/migration staging: parse a SQLite wallet export as the runtime +// wallet loader would. Prints counts only, never record keys or values. +int CmdWalletSQLiteLoadCheck(const std::string& strPath); + // Rewrite wallet.dat so private keys and the HD seed are encrypted with the // given passphrase. Returns 0 on success and exits without starting the node. int CmdEncryptWallet(const std::string& strPassphrase); -- cgit v1.2.3 From f9ee957bd6d3f28dfee166a82f389aad666a0119 Mon Sep 17 00:00:00 2001 From: Nsable-sh Date: Fri, 14 Aug 2026 18:17:23 -0300 Subject: wallet: stage SQLite runtime loading --- docs/storage.md | 4 + src/db.cpp | 361 +++++++++++++++++++++++++++++++++++++++++++++++--- src/db.h | 1 + src/main_gui.cpp | 22 +++ src/selftest.cpp | 29 ++++ src/wallet_sqlite.cpp | 19 +++ src/wallet_sqlite.h | 1 + 7 files changed, 420 insertions(+), 17 deletions(-) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index 07c7170..08c4a01 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -120,6 +120,10 @@ file and failed restores remove the incomplete `wallet.dat`. `-walletsqliteloadcheck=FILE` parses a SQLite export using the same record types the wallet loader understands, so a migration can prove the export is not just byte-preserving but loader-compatible before runtime activation. +`-walletsqlite=FILE` is the first runtime staging flag: it loads a SQLite export +through the startup wallet path, initializes the in-memory wallet state, reports +success, and exits before network, mining, GUI, or wallet mutation begins. It is +deliberately read-only until the write path is routed to SQLite as well. ## `blkindex.dat` diff --git a/src/db.cpp b/src/db.cpp index 8b82fe5..581e5a7 100644 --- a/src/db.cpp +++ b/src/db.cpp @@ -3,6 +3,7 @@ // file license.txt or http://www.opensource.org/licenses/mit-license.php. #include "headers.h" +#include "wallet_sqlite.h" @@ -650,6 +651,289 @@ bool CReviewDB::WriteReviews(uint256 hash, const vector& vReviews) // CWalletDB // +class CSQLiteWalletRuntimeLoadVisitor : public CWalletRecordVisitor +{ +public: + bool fHaveStoredMineMode; + bool fHaveStoredHDCoinType; + string strLastType; + + CSQLiteWalletRuntimeLoadVisitor() : + fHaveStoredMineMode(false), + fHaveStoredHDCoinType(false) + { + } + + bool VisitWalletRecord(const CDataStream& ssKeyIn, + const CDataStream& ssValueIn, + string& strErrorRet) + { + CDataStream ssKey = ssKeyIn; + CDataStream ssValue = ssValueIn; + string strType; + ssKey >> strType; + strLastType = strType; + + if (strType == "name") + { + string strAddress; + ssKey >> strAddress; + ssValue >> mapAddressBook[strAddress]; + } + else if (strType == "tx") + { + uint256 hash; + ssKey >> hash; + CWalletTx& wtx = mapWallet[hash]; + ssValue >> wtx; + + if (wtx.GetHash() != hash) + printf("Error in SQLite wallet export, hash mismatch\n"); + } + else if (strType == "key") + { + vector vchPubKey; + ssKey >> vchPubKey; + CPrivKey vchPrivKey; + ssValue >> vchPrivKey; + + mapKeys[vchPubKey] = vchPrivKey; + mapPubKeys[Hash160(vchPubKey)] = vchPubKey; + } + else if (strType == "mkey") + { + unsigned int nID = 0; + ssKey >> nID; + CWalletMasterKey kMasterKey; + ssValue >> kMasterKey; + mapMasterKeys[nID] = kMasterKey; + nWalletMasterKeyMaxID = max(nWalletMasterKeyMaxID, nID); + fWalletEncrypted = true; + fWalletLocked = true; + } + else if (strType == "ckey") + { + vector vchPubKey; + ssKey >> vchPubKey; + vector vchCryptedSecret; + ssValue >> vchCryptedSecret; + if (!AddCryptedKey(vchPubKey, vchCryptedSecret)) + { + strErrorRet = "SQLite wallet export has an unreadable encrypted key record"; + return false; + } + } + else if (strType == "defaultkey") + { + ssValue >> vchDefaultKey; + } + else if (strType == "hdmaster") + { + ssValue >> vchHDMaster; + } + else if (strType == "hdchaincode") + { + ssValue >> vchHDChainCode; + } + else if (strType == "hdnext") + { + ssValue >> nHDNext; + } + else if (strType == "hdschema") + { + ssValue >> nHDKeySchema; + } + else if (strType == "hdcointype") + { + ssValue >> nHDCoinType; + fHaveStoredHDCoinType = true; + } + else if (strType == "hdreceivenext") + { + ssValue >> nHDReceiveNext; + } + else if (strType == "hdchangenext") + { + ssValue >> nHDChangeNext; + } + else if (strType == "cryptedhdmaster") + { + ssValue >> vchCryptedHDMaster; + fWalletEncrypted = true; + fWalletLocked = true; + } + else if (strType == "cryptedhdchaincode") + { + ssValue >> vchCryptedHDChainCode; + fWalletEncrypted = true; + fWalletLocked = true; + } + else if (strType == "walletminversion") + { + int nMinVersion = 0; + ssValue >> nMinVersion; + if (nMinVersion > WALLET_FORMAT_SUPPORTED) + { + strWalletLoadError = strprintf( + "This SQLite wallet export was written by a newer version of " + "Bitflash (wallet format %d; this build understands %d). " + "Upgrade Bitflash before opening it.", + nMinVersion, WALLET_FORMAT_SUPPORTED); + strErrorRet = strWalletLoadError; + return false; + } + } + else if (strType == "pool") + { + int64 nIndex; + ssKey >> nIndex; + vector vchPubKey; + ssValue >> vchPubKey; + mapKeyPool[nIndex] = vchPubKey; + } + else if (strType == "setting") + { + string strKey; + ssKey >> strKey; + if (strKey == "nTransactionFee") ssValue >> nTransactionFee; + if (strKey == "addrIncoming") ssValue >> addrIncoming; + + if (!fMineModeFromCommandLine) + { + if (strKey == "nMineMode") { ssValue >> nMineMode; fHaveStoredMineMode = true; } + if (strKey == "fGenerateBitcoins") ssValue >> fGenerateBitcoins; + if (strKey == "strParticipantPool") ssValue >> strParticipantPool; + } + } + + if (ssKey.fail()) + { + strErrorRet = strprintf("malformed SQLite wallet record key while handling '%s'", + strLastType.c_str()); + return false; + } + if (ssValue.fail()) + { + strErrorRet = strprintf("malformed SQLite wallet record value while handling '%s'", + strLastType.c_str()); + return false; + } + return true; + } + + vector vchDefaultKey; +}; + +static bool FinishLoadedWallet(const vector& vchDefaultKey, + bool fMayCreateDefaultKey) +{ + if (!vchDefaultKey.empty() && mapPubKeys.count(Hash160(vchDefaultKey))) + { + keyUser.SetPubKey(vchDefaultKey); + if (mapKeys.count(vchDefaultKey)) + keyUser.SetPrivKey(mapKeys[vchDefaultKey]); + } + else if (IsWalletEncrypted()) + { + printf("LoadWallet: encrypted wallet has no usable default public key. " + "The file was left untouched.\n"); + return false; + } + else + { + if (!fMayCreateDefaultKey) + { + strWalletLoadError = + "SQLite wallet export has no usable default public key"; + printf("LoadWallet: %s\n", strWalletLoadError.c_str()); + return false; + } + + keyUser.MakeNewKey(); + if (!AddKey(keyUser)) + return false; + if (!SetAddressBookName(PubKeyToAddress(keyUser.GetPubKey()), "Your Address")) + return false; + CWalletDB().WriteDefaultKey(keyUser.GetPubKey()); + } + + return true; +} + +static bool ReconcileLoadedWalletMetadata(bool fHaveStoredMineMode, + bool fHaveStoredHDCoinType) +{ + // An encrypted wallet has a seed; it just has not been decrypted yet. + // + // The reset in the else branch was written when "no seed in memory after + // loading" could only mean "this wallet has no seed". With encryption that + // is no longer true, and taking the else branch here wiped the schema and + // the derivation counters that had just been read correctly out of the + // file. Unlocking later restores the seed but nothing restores those, so a + // BIP44 wallet came back deriving legacy addresses -- different addresses + // from the ones its own recovery phrase produces, while the audit reported + // the phrase as covering nothing. + if (HaveHDSeed() || !vchCryptedHDMaster.empty()) + { + if (nHDKeySchema == HD_SCHEMA_NONE) + { + nHDKeySchema = HD_SCHEMA_LEGACY; + printf("LoadWallet: deterministic seed has no schema metadata; " + "treating it as legacy m/index'\n"); + } + else if (nHDKeySchema != HD_SCHEMA_LEGACY && + nHDKeySchema != HD_SCHEMA_BIP44) + { + printf("LoadWallet: deterministic wallet schema %d (%s) is not " + "supported by this build\n", + nHDKeySchema, HDKeySchemaName(nHDKeySchema).c_str()); + return false; + } + if (!fHaveStoredHDCoinType) + { + nHDCoinType = HD_BIP44_COIN_TYPE_BITFLASH; + printf("LoadWallet: deterministic seed has no BIP44 coin type metadata; " + "using Bitflash coin type %u\n", nHDCoinType); + } + } + else + { + nHDKeySchema = HD_SCHEMA_NONE; + nHDNext = 0; + nHDReceiveNext = 0; + nHDChangeNext = 0; + nHDCoinType = HD_BIP44_COIN_TYPE_BITFLASH; + } + + // fGenerateBitcoins and nMineMode only mean anything together, and a + // wallet.dat can easily hold one without the other: Bitcoin 0.1.0 already + // wrote fGenerateBitcoins when mining was toggled, years before this fork + // had a mode at all. Restoring that lone flag put a node into the one state + // the miner cannot act on -- generate on, mode relay -- so every miner + // thread started and returned at the relay guard, and the machine sat there + // looking like it was mining while hashing nothing. That happened to a + // 32-core node on the first 1.2.11 start. + // + // So the flag is only believed when the mode it belongs to was stored with + // it, and the pair is made coherent either way. + if (!fMineModeFromCommandLine) + { + if (!fHaveStoredMineMode) + { + if (fGenerateBitcoins) + printf("LoadWallet: ignoring a stored fGenerateBitcoins with no mining mode " + "beside it -- it predates this setting and cannot be read on its own\n"); + fGenerateBitcoins = 0; + } + else + { + fGenerateBitcoins = (nMineMode != MINE_RELAY) ? 1 : 0; + } + } + + return true; +} + bool CWalletDB::LoadWallet(vector& vchDefaultKeyRet) { // Whether wallet.dat actually carried a mining mode, as opposed to just the @@ -1005,31 +1289,74 @@ bool LoadWallet() if (!CWalletDB("cr").LoadWallet(vchDefaultKey)) return false; - if (!vchDefaultKey.empty() && mapPubKeys.count(Hash160(vchDefaultKey))) + return FinishLoadedWallet(vchDefaultKey, true); +} + +bool LoadWalletFromSQLite(const string& strPath) +{ + strWalletLoadError.clear(); + + if (!FileExists(strPath.c_str())) { - // Set keyUser - keyUser.SetPubKey(vchDefaultKey); - if (mapKeys.count(vchDefaultKey)) - keyUser.SetPrivKey(mapKeys[vchDefaultKey]); + strWalletLoadError = strprintf("SQLite wallet export does not exist: %s", + strPath.c_str()); + printf("LoadWallet: %s\n", strWalletLoadError.c_str()); + return false; } - else if (IsWalletEncrypted()) + + CWalletDBSQLite sqlite; + string strError; + if (!sqlite.OpenReadOnly(strPath, strError)) { - printf("LoadWallet: encrypted wallet has no usable default public key. " - "The file was left untouched.\n"); + strWalletLoadError = strError; + printf("LoadWallet: cannot open SQLite wallet export: %s\n", + strError.c_str()); return false; } - else + + CSQLiteWalletRuntimeLoadVisitor visitor; + try { - // Create new keyUser and set as default key - keyUser.MakeNewKey(); - if (!AddKey(keyUser)) - return false; - if (!SetAddressBookName(PubKeyToAddress(keyUser.GetPubKey()), "Your Address")) - return false; - CWalletDB().WriteDefaultKey(keyUser.GetPubKey()); + CRITICAL_BLOCK(cs_mapKeys) + CRITICAL_BLOCK(cs_mapWallet) + { + if (!sqlite.ScanRecords(visitor, strError)) + { + if (strWalletLoadError.empty()) + strWalletLoadError = strError; + printf("LoadWallet: SQLite wallet export could not be read. " + "Failed while handling a '%s' record: %s\n", + visitor.strLastType.c_str(), strError.c_str()); + return false; + } + } + } + catch (const std::exception& e) + { + strWalletLoadError = strprintf( + "SQLite wallet export could not be read while handling a '%s' record: %s", + visitor.strLastType.c_str(), e.what()); + printf("LoadWallet: %s\n", strWalletLoadError.c_str()); + return false; + } + catch (...) + { + strWalletLoadError = strprintf( + "SQLite wallet export could not be read while handling a '%s' record", + visitor.strLastType.c_str()); + printf("LoadWallet: %s\n", strWalletLoadError.c_str()); + return false; } - return true; + if (!ReconcileLoadedWalletMetadata(visitor.fHaveStoredMineMode, + visitor.fHaveStoredHDCoinType)) + { + if (strWalletLoadError.empty()) + strWalletLoadError = "SQLite wallet metadata is not supported by this build"; + return false; + } + + return FinishLoadedWallet(visitor.vchDefaultKey, false); } diff --git a/src/db.h b/src/db.h index 53bda59..59ff24a 100644 --- a/src/db.h +++ b/src/db.h @@ -536,6 +536,7 @@ public: bool ScanWalletRecords(CWalletRecordVisitor& visitor, string& strErrorRet); bool LoadWallet(); +bool LoadWalletFromSQLite(const string& strPath); // Writes a wallet.dat that opens on its own, anywhere -- see the definition in // db.cpp for why a plain file copy does not. Still a point-in-time snapshot: diff --git a/src/main_gui.cpp b/src/main_gui.cpp index c9aac7d..ee875c7 100644 --- a/src/main_gui.cpp +++ b/src/main_gui.cpp @@ -189,6 +189,7 @@ static void PrintUsage() printf(" an empty data directory, then exit)\n"); printf(" /walletsqliteloadcheck=FILE\n"); printf(" (parse a SQLite export like the wallet loader)\n"); + printf(" /walletsqlite=FILE (stage-load a SQLite wallet export, then exit)\n"); printf(" /rescan (walk the chain for coins this wallet owns\n"); printf(" but never recorded, then exit)\n"); printf("\n"); @@ -617,6 +618,27 @@ int main(int argc, char* argv[]) printf("Loading wallet...\n"); try { + string strWalletSQLite = + argval2(argc, argv, "/walletsqlite", "-walletsqlite"); + if (!strWalletSQLite.empty()) + { + AttachTerminal(); + if (!LoadWalletFromSQLite(strWalletSQLite)) + { + FatalStartupError(fHeadlessStartup, + "Cannot open SQLite wallet export.", + strWalletLoadError); + return 1; + } + + fprintf(stderr, "SQLite wallet loaded: read-only staging backend\n"); + fprintf(stderr, "SQLite runtime writes are not enabled yet; exiting before " + "network, mining, GUI, or wallet mutation starts.\n"); + fflush(stderr); + DBFlush(true); + return 0; + } + if (!LoadWallet()) { // LoadWallet() explains itself into strWalletLoadError when it diff --git a/src/selftest.cpp b/src/selftest.cpp index feb4076..14bf45a 100644 --- a/src/selftest.cpp +++ b/src/selftest.cpp @@ -1298,6 +1298,19 @@ static int RunWalletSQLiteMigrationSelfTest() "load check: ok"), "SQLite wallet export passes the loader compatibility check") ? 0 : 1; + string strRuntimeOut = tmp + "/sqlite-runtime.txt"; + vector vRuntimeArgs; + vRuntimeArgs.push_back("-datadir=" + strWalletDir); + vRuntimeArgs.push_back("-nomanagedtor"); + vRuntimeArgs.push_back("-nogui"); + vRuntimeArgs.push_back("-walletsqlite=" + strSQLite); + int nRuntimeRet = + RunBitflashChild(strExe, vRuntimeArgs, NULL, &strRuntimeOut); + nFail += Check(nRuntimeRet == 0 && + FileContainsText(strRuntimeOut, + "SQLite wallet loaded: read-only staging backend"), + "SQLite wallet export loads through the runtime startup path") ? 0 : 1; + string strRestoredDir = tmp + "/restored-wallet"; nFail += Check(MakeDirLocal(strRestoredDir), "restored wallet directory can be created") ? 0 : 1; @@ -1372,6 +1385,22 @@ static int RunWalletSQLiteMigrationSelfTest() "load check: failed"), "SQLite wallet load check rejects future wallet formats") ? 0 : 1; + string strFutureRuntimeOut = tmp + "/sqlite-runtime-future.txt"; + vector vFutureRuntimeArgs; + vFutureRuntimeArgs.push_back("-datadir=" + strWalletDir); + vFutureRuntimeArgs.push_back("-nomanagedtor"); + vFutureRuntimeArgs.push_back("-nogui"); + vFutureRuntimeArgs.push_back("-walletsqlite=" + strFutureSQLite); + int nFutureRuntimeRet = + RunBitflashChild(strExe, vFutureRuntimeArgs, NULL, + &strFutureRuntimeOut); + nFail += Check(nFutureRuntimeRet == 1 && + FileContainsText(strFutureRuntimeOut, + "Cannot open SQLite wallet export") && + FileContainsText(strFutureRuntimeOut, + "newer version of Bitflash"), + "SQLite runtime startup rejects future wallet formats") ? 0 : 1; + string strOverwriteOut = tmp + "/sqlite-export-overwrite.txt"; int nOverwriteRet = RunBitflashChild(strExe, vExportArgs, NULL, &strOverwriteOut); diff --git a/src/wallet_sqlite.cpp b/src/wallet_sqlite.cpp index 8489cac..ca81573 100644 --- a/src/wallet_sqlite.cpp +++ b/src/wallet_sqlite.cpp @@ -58,6 +58,25 @@ bool CWalletDBSQLite::Open(const string& strPath, string& strError) return true; } +bool CWalletDBSQLite::OpenReadOnly(const string& strPath, string& strError) +{ + Close(); + + int ret = sqlite3_open_v2(strPath.c_str(), &pdb, + SQLITE_OPEN_READONLY | + SQLITE_OPEN_FULLMUTEX, + NULL); + if (ret != SQLITE_OK) + { + SetSQLiteError(pdb, "cannot open SQLite wallet read-only", strError); + Close(); + return false; + } + + sqlite3_busy_timeout(pdb, 5000); + return true; +} + void CWalletDBSQLite::Close() { if (pdb) diff --git a/src/wallet_sqlite.h b/src/wallet_sqlite.h index 385a340..5e6467b 100644 --- a/src/wallet_sqlite.h +++ b/src/wallet_sqlite.h @@ -15,6 +15,7 @@ public: ~CWalletDBSQLite(); bool Open(const string& strPath, string& strError); + bool OpenReadOnly(const string& strPath, string& strError); void Close(); bool BeginTransaction(string& strError); -- cgit v1.2.3 From 4c24a86a8aa01b02874e5bdb7856beed6e4cf6b6 Mon Sep 17 00:00:00 2001 From: bitflash Date: Wed, 19 Aug 2026 11:20:55 -0300 Subject: wallet/docs: SQLite is the default backend, not experimental Drop the "experimental" label from the wallet code and docs now that SQLite is the default backend for new wallets, is coin-proven, and carries encrypt, backup, atomic seed writes, and the storage diagnostics. - main_gui.cpp: -help text, the unknown-backend error (SQLite is no longer "experimental" and Berkeley DB is no longer "the default"), and the startup banner ("Wallet backend: SQLite"). - walletcmd.h: the export command comment. - docs/storage.md: rewrite the SQLite section from staging-only ("runtime still opens Berkeley DB", "no runtime backend flag exists yet") to the shipped state -- default backend, conversion wizard, per-backend ops -- and reframe the migration order as shipped in v1.2.20. --- docs/storage.md | 67 +++++++++++++++++++++++++++----------------------------- src/main_gui.cpp | 16 +++++++------- src/walletcmd.h | 6 ++--- 3 files changed, 43 insertions(+), 46 deletions(-) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index 08c4a01..b52a955 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -85,10 +85,11 @@ A future wallet backend must keep these properties: - recovery phrase encrypted and locked, requiring unlock before coverage can be audited. -## Experimental SQLite Wallet Store +## SQLite Wallet Store -The SQLite wallet path starts as a byte-preserving record store, not as a new -wallet format: +Since v1.2.20 SQLite is the default wallet backend. It is a byte-preserving +record store, not a new wallet format -- the same serialized `CDataStream` bytes +`wallet.dat` uses: ```sql CREATE TABLE wallet_records ( @@ -98,32 +99,26 @@ CREATE TABLE wallet_records ( PRAGMA user_version=1; ``` -Keys and values are the same serialized `CDataStream` bytes used by `wallet.dat`. -The first implementation is exercised only by `-selftest=wallet-sqlite`; the -runtime wallet still opens Berkeley DB. The purpose is to prove schema creation, -raw record write/read, close/reopen behavior, and value preservation before any -BDB-to-SQLite migration or runtime backend flag exists. - -`-walletsqliteexport=FILE` is the next staging tool. It streams every raw -`wallet.dat` record through `ScanWalletRecords()` and writes those same key and -value bytes into the SQLite table inside one transaction. It refuses to -overwrite an existing export and still does not change the runtime wallet -backend. `CWalletDBSQLite::ScanRecords()` can stream those exported records back -through the same visitor shape, so tests can compare BDB and SQLite record -streams without introducing a SQLite runtime wallet loader yet. -`-walletsqliteverify=FILE` compares the current `wallet.dat` stream with a -SQLite export and prints only record counts and mismatch counts. -`-walletsqliterestore=FILE` rebuilds `wallet.dat` from a SQLite export only in -an empty data directory, then the verifier can prove the restored BDB file still -matches the export byte-for-byte. Failed exports remove the incomplete SQLite -file and failed restores remove the incomplete `wallet.dat`. -`-walletsqliteloadcheck=FILE` parses a SQLite export using the same record -types the wallet loader understands, so a migration can prove the export is not -just byte-preserving but loader-compatible before runtime activation. -`-walletsqlite=FILE` is the first runtime staging flag: it loads a SQLite export -through the startup wallet path, initializes the in-memory wallet state, reports -success, and exits before network, mining, GUI, or wallet mutation begins. It is -deliberately read-only until the write path is routed to SQLite as well. +A fresh data directory starts on `wallet.sqlite`. An existing `wallet.dat` keeps +running on Berkeley DB until it is converted: the GUI offers a one-click +conversion on first run, and `-walletbackend=sqlite` / `-walletbackend=bdb` +(recorded in a plain-text `wallet-backend` marker file in the datadir) select a +backend explicitly. Conversion never modifies `wallet.dat`; it stays as a +fallback, and the change takes effect on restart. + +Every wallet operation runs on either backend: read, write, erase, key-pool and +HD-seed writes (the two halves of the seed bracketed in one SQLite transaction), +`-backupwallet` (checkpoint then copy the self-contained file), `-encryptwallet` +(an atomic rebuild-and-swap in one transaction), and the `-walletstorageaudit` / +`-walletstoragecheck` diagnostics, which read the active backend's file. + +The migration tooling remains available for scripted use. `-walletsqliteexport=FILE` +streams every raw `wallet.dat` record through `ScanWalletRecords()` into a SQLite +file in one transaction, refusing to overwrite. `-walletsqliteverify=FILE` +compares the two streams and prints only counts. `-walletsqliteloadcheck=FILE` +parses an export with the same record types the loader uses. +`-walletsqliterestore=FILE` rebuilds a `wallet.dat` from an export in an empty +data directory. Failed exports and restores remove their incomplete output. ## `blkindex.dat` @@ -150,12 +145,14 @@ conversion path. The old IP address database (`addr.dat`) has been removed. Peer discovery is handled by Nostr, `.btf`, rendezvous relays, and direct onion peers. -## Recommended Migration Order +## Migration Order (shipped in v1.2.20) -1. Keep adding storage self-tests around current BDB behavior. -2. Add an experimental wallet backend behind an explicit flag. -3. Implement BDB-to-new-wallet migration as a one-way copy with a backup. -4. Make the new wallet backend default only after cross-version restore, - encryption, backup, and recovery-audit tests pass. +1. Storage self-tests were added around Berkeley DB behavior first. +2. The SQLite backend landed behind an explicit flag, then behind a + `wallet-backend` marker and a GUI conversion wizard. +3. BDB-to-SQLite migration is a one-way copy that leaves `wallet.dat` as a backup. +4. SQLite became the default for new wallets only after cross-version restore, + encryption, backup, and recovery-audit tests passed -- and a live coin-proof + of receive, spend, restart, and encrypt on real mainnet funds. 5. Treat `blkindex.dat` separately; it is safer to rebuild than to migrate under pressure. diff --git a/src/main_gui.cpp b/src/main_gui.cpp index 55ca8c6..943799f 100644 --- a/src/main_gui.cpp +++ b/src/main_gui.cpp @@ -182,7 +182,7 @@ static void PrintUsage() printf(" /walletstoragecheck (fail closed on inconsistent or unsafe\n"); printf(" wallet.dat storage state)\n"); printf(" /walletsqliteexport=FILE (copy raw wallet.dat records into an\n"); - printf(" experimental SQLite store, then exit)\n"); + printf(" SQLite store, then exit)\n"); printf(" /walletsqliteverify=FILE (compare wallet.dat with a SQLite export,\n"); printf(" printing counts only, then exit)\n"); printf(" /walletsqliterestore=FILE (rebuild wallet.dat from a SQLite export in\n"); @@ -680,17 +680,17 @@ int main(int argc, char* argv[]) AttachTerminal(); FatalStartupError(fHeadlessStartup, strprintf("Unknown wallet backend '%s'.", strWalletBackend.c_str()), - "Use -walletbackend=sqlite for the experimental SQLite backend, " - "or omit -walletbackend (or -walletbackend=bdb) to use the " - "default Berkeley DB wallet.dat."); + "Use -walletbackend=sqlite for the SQLite backend, or " + "-walletbackend=bdb for the Berkeley DB wallet.dat."); return 1; } if (strWalletBackend == "sqlite") { - // Opt-in, experimental. The node runs on /wallet.sqlite and - // never opens wallet.dat, but the user has to have exported one - // first -- there is no silent migration, and wallet.dat is left + // Explicit opt-in via the flag. The node runs on + // /wallet.sqlite and never opens wallet.dat, but the user + // has to have exported one first -- there is no silent migration, + // and wallet.dat is left // exactly where it is so a plain restart goes back to it. AttachTerminal(); string strSQLitePath = GetAppDir() + "/wallet.sqlite"; @@ -724,7 +724,7 @@ int main(int argc, char* argv[]) } fprintf(stderr, - "Wallet backend: SQLite (experimental)\n" + "Wallet backend: SQLite\n" " %s\n" "Your Berkeley DB wallet.dat is left untouched. Restart without\n" "-walletbackend=sqlite to go back to it.\n", diff --git a/src/walletcmd.h b/src/walletcmd.h index 38cbfdb..7013d44 100644 --- a/src/walletcmd.h +++ b/src/walletcmd.h @@ -70,9 +70,9 @@ int CmdWalletStorageAudit(const std::string& strJsonOut = ""); // or migration-unsafe mix of storage records. int CmdWalletStorageCheck(); -// Diagnostic/migration staging: copy every raw wallet.dat record into an -// experimental SQLite key/value store, preserving serialized bytes. This does -// not change which backend the runtime wallet opens. +// Copy every raw wallet.dat record into a SQLite key/value store, preserving +// serialized bytes. A migration/scripting tool; it does not change which backend +// the runtime wallet opens. int CmdWalletSQLiteExport(const std::string& strDest); // Export the currently-loaded wallet to a SQLite file (shared by the command -- cgit v1.2.3 From 18ffcb21393c9e54653844d62c5ef97c58bc61d4 Mon Sep 17 00:00:00 2001 From: bitflash-maintainer Date: Tue, 1 Sep 2026 17:16:04 +0000 Subject: docs: this tree has no marketplace stores to document --- docs/storage.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) (limited to 'docs/storage.md') diff --git a/docs/storage.md b/docs/storage.md index b52a955..c7de4e7 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -137,10 +137,11 @@ conversion path. ## Other Legacy Stores -| File | Class | Records | Current risk | -|---|---|---|---| -| `reviews.dat` | `CReviewDB` | `("user", hash)`, `("reviews", hash)` | Legacy market/review storage. Not part of wallet safety. | -| `market.dat` | `CMarketDB` | No explicit typed records in current wrappers. | Legacy market storage shell. | +None. Upstream still documents `reviews.dat` (`CReviewDB`) and `market.dat` +(`CMarketDB`) here; this fork deleted the 2009 marketplace outright, along +with `CReviewDB`, `market.cpp` and `market.h`, so neither file is opened or +created by this tree. A datadir carrying them from an older build is simply +ignored. The old IP address database (`addr.dat`) has been removed. Peer discovery is handled by Nostr, `.btf`, rendezvous relays, and direct onion peers. -- cgit v1.2.3