upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs/how-to/migrate-ngit-relay-to-ngit-grasp.md
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2026-01-23 11:06:12 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2026-01-27 20:37:55 +0000
commita5504395c946bdf28b5ad0e0148ff371ca33d4d3 (patch)
tree3465b303ef9f0c8fba1269012740710018cd1797 /docs/how-to/migrate-ngit-relay-to-ngit-grasp.md
parent7536160c0ab1b64090ba9b5ab8ea6aef4747bb48 (diff)
Add Phase 3 migration scripts for categorization and comparison
- 20-categorize.sh: Categorizes git sync status into 4 categories - 21-compare-relays.sh: Compares prod vs archive to find gaps - Updated how-to doc with detailed Phase 3 outputs and directory structure - Tested with Jan 22 data: 231 complete in both, 276 complete in prod but missing from archive
Diffstat (limited to 'docs/how-to/migrate-ngit-relay-to-ngit-grasp.md')
-rw-r--r--docs/how-to/migrate-ngit-relay-to-ngit-grasp.md61
1 files changed, 37 insertions, 24 deletions
diff --git a/docs/how-to/migrate-ngit-relay-to-ngit-grasp.md b/docs/how-to/migrate-ngit-relay-to-ngit-grasp.md
index d01bbf2..4c3a4ba 100644
--- a/docs/how-to/migrate-ngit-relay-to-ngit-grasp.md
+++ b/docs/how-to/migrate-ngit-relay-to-ngit-grasp.md
@@ -51,7 +51,7 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c
51``` 51```
52┌─────────────────────────────────────────────────────────────────┐ 52┌─────────────────────────────────────────────────────────────────┐
53│ PHASE 1: Fetch Events (~30s, local) │ 53│ PHASE 1: Fetch Events (~30s, local) │
54scripts/migration/01-fetch-events.sh <relay> <output-dir> │ 54│ migration-scripts/01-fetch-events.sh <relay> <output-dir> │
55├─────────────────────────────────────────────────────────────────┤ 55├─────────────────────────────────────────────────────────────────┤
56│ Fetches from relay: │ 56│ Fetches from relay: │
57│ - kind 30618 (state events) │ 57│ - kind 30618 (state events) │
@@ -64,7 +64,7 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c
64 64
65┌─────────────────────────────────────────────────────────────────┐ 65┌─────────────────────────────────────────────────────────────────┐
66│ PHASE 2: Git Sync Check (~20 mins, VPS required) │ 66│ PHASE 2: Git Sync Check (~20 mins, VPS required) │
67scripts/migration/10-check-git-sync.sh <events> <git-base> <out>│ 67│ migration-scripts/10-check-git-sync.sh <events> <git-base> <out>│
68├─────────────────────────────────────────────────────────────────┤ 68├─────────────────────────────────────────────────────────────────┤
69│ For each state event, compares refs to actual git data on disk. │ 69│ For each state event, compares refs to actual git data on disk. │
70│ │ 70│ │
@@ -78,8 +78,8 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c
78 78
79┌─────────────────────────────────────────────────────────────────┐ 79┌─────────────────────────────────────────────────────────────────┐
80│ PHASE 3: Categorize & Compare (fast, local) │ 80│ PHASE 3: Categorize & Compare (fast, local) │
81scripts/migration/20-categorize.sh <sync-status> <output-dir> │ 81│ migration-scripts/20-categorize.sh <sync-status> <output-dir> │
82scripts/migration/21-compare-relays.sh <prod> <archive> <out> │ 82│ migration-scripts/21-compare-relays.sh <prod> <archive> <out> │
83├─────────────────────────────────────────────────────────────────┤ 83├─────────────────────────────────────────────────────────────────┤
84│ 20-categorize.sh applies 4-category logic: │ 84│ 20-categorize.sh applies 4-category logic: │
85│ - cat1: complete match (all refs match) │ 85│ - cat1: complete match (all refs match) │
@@ -87,18 +87,20 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c
87│ - cat3: partial match (some refs match) │ 87│ - cat3: partial match (some refs match) │
88│ - cat4: no match (git exists but refs don't match) │ 88│ - cat4: no match (git exists but refs don't match) │
89│ │ 89│ │
90│ 21-compare-relays.sh finds gaps: │ 90│ 21-compare-relays.sh compares prod vs archive: │
91│ - in prod but not archive │ 91│ - complete-in-both.txt (no action needed) │
92│ - in archive but not prod │ 92│ - complete-prod-missing-archive.txt (needs investigation) │
93│ - different status between relays │ 93│ - complete-prod-incomplete-archive.txt (sync in progress?) │
94│ - incomplete-in-both.txt (git data incomplete) │
95│ - in-archive-not-prod.txt (deleted or new) │
94│ │ 96│ │
95│ Output: category-{1,2,3,4}.txt, relay-gaps.txt 97│ Output: category-{1,2,3,4}.txt, comparison/*.txt, summary.txt
96└─────────────────────────────────────────────────────────────────┘ 98└─────────────────────────────────────────────────────────────────┘
97 99
98┌─────────────────────────────────────────────────────────────────┐ 100┌─────────────────────────────────────────────────────────────────┐
99│ PHASE 4: Log-Based Categories (VPS required) │ 101│ PHASE 4: Log-Based Categories (VPS required) │
100scripts/migration/30-extract-parse-failures.sh <service> <out> │ 102│ migration-scripts/30-extract-parse-failures.sh <service> <out> │
101scripts/migration/31-extract-purgatory-expiry.sh <service> <out>│ 103│ migration-scripts/31-extract-purgatory-expiry.sh <service> <out>│
102├─────────────────────────────────────────────────────────────────┤ 104├─────────────────────────────────────────────────────────────────┤
103│ Extracts structured log entries from journalctl: │ 105│ Extracts structured log entries from journalctl: │
104│ - Parse failures: [PARSE_FAIL] kind=X event_id=Y reason=Z │ 106│ - Parse failures: [PARSE_FAIL] kind=X event_id=Y reason=Z │
@@ -112,7 +114,7 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c
112 114
113┌─────────────────────────────────────────────────────────────────┐ 115┌─────────────────────────────────────────────────────────────────┐
114│ PHASE 5: Final Classification (fast, local) │ 116│ PHASE 5: Final Classification (fast, local) │
115scripts/migration/40-classify-actions.sh <all-inputs> <out> │ 117│ migration-scripts/40-classify-actions.sh <all-inputs> <out> │
116├─────────────────────────────────────────────────────────────────┤ 118├─────────────────────────────────────────────────────────────────┤
117│ Combines all data sources to produce final classification: │ 119│ Combines all data sources to produce final classification: │
118│ │ 120│ │
@@ -136,27 +138,38 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c
136work/migration-analysis-YYYYMMDD-HHMM/ 138work/migration-analysis-YYYYMMDD-HHMM/
137├── prod/ 139├── prod/
138│ ├── raw/ 140│ ├── raw/
139│ │ ├── state-events.json 141│ │ ├── state-events.json # Phase 1 output
140│ │ ├── announcements.json 142│ │ ├── announcements.json # Phase 1 output
141│ │ └── deletions.json 143│ │ └── deletions.json # Phase 1 output
142│ ├── git-sync-status.tsv 144│ ├── git-sync-status.tsv # Phase 2 output (optional)
143│ └── category-{1,2,3,4}.txt 145│ ├── category1-complete-match.txt # Phase 2/3 output
146│ ├── category2-empty-blank.txt # Phase 2/3 output
147│ ├── category3-partial-match.txt # Phase 2/3 output
148│ └── category4-no-match.txt # Phase 2/3 output
144├── archive/ 149├── archive/
145│ ├── raw/ 150│ ├── raw/
146│ │ ├── state-events.json 151│ │ ├── state-events.json
147│ │ ├── announcements.json 152│ │ ├── announcements.json
148│ │ └── deletions.json 153│ │ └── deletions.json
149│ ├── git-sync-status.tsv 154│ ├── git-sync-status.tsv
150│ └── category-{1,2,3,4}.txt 155│ ├── category1-complete-match.txt
156│ ├── category2-empty-blank.txt
157│ ├── category3-partial-match.txt
158│ └── category4-no-match.txt
151├── logs/ 159├── logs/
152│ ├── parse-failures.txt 160│ ├── parse-failures.txt # Phase 4 output
153│ └── purgatory-expired.txt 161│ └── purgatory-expired.txt # Phase 4 output
154├── comparison/ 162├── comparison/
155│ └── relay-gaps.txt 163│ ├── complete-in-both.txt # Phase 3 output (no action)
164│ ├── complete-prod-missing-archive.txt # Phase 3 output (investigate)
165│ ├── complete-prod-incomplete-archive.txt # Phase 3 output (sync in progress?)
166│ ├── incomplete-in-both.txt # Phase 3 output (git incomplete)
167│ ├── in-archive-not-prod.txt # Phase 3 output (deleted/new)
168│ └── summary.txt # Phase 3 output (human-readable)
156└── results/ 169└── results/
157 ├── no-action-required.txt 170 ├── no-action-required.txt # Phase 5 output
158 ├── action-required.txt 171 ├── action-required.txt # Phase 5 output
159 └── manual-investigation.txt 172 └── manual-investigation.txt # Phase 5 output
160``` 173```
161 174
162## Prerequisites 175## Prerequisites