diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-23 11:06:12 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-27 20:37:55 +0000 |
| commit | a5504395c946bdf28b5ad0e0148ff371ca33d4d3 (patch) | |
| tree | 3465b303ef9f0c8fba1269012740710018cd1797 /docs/how-to/migrate-ngit-relay-to-ngit-grasp.md | |
| parent | 7536160c0ab1b64090ba9b5ab8ea6aef4747bb48 (diff) | |
Add Phase 3 migration scripts for categorization and comparison
- 20-categorize.sh: Categorizes git sync status into 4 categories
- 21-compare-relays.sh: Compares prod vs archive to find gaps
- Updated how-to doc with detailed Phase 3 outputs and directory structure
- Tested with Jan 22 data: 231 complete in both, 276 complete in prod but missing from archive
Diffstat (limited to 'docs/how-to/migrate-ngit-relay-to-ngit-grasp.md')
| -rw-r--r-- | docs/how-to/migrate-ngit-relay-to-ngit-grasp.md | 61 |
1 files changed, 37 insertions, 24 deletions
diff --git a/docs/how-to/migrate-ngit-relay-to-ngit-grasp.md b/docs/how-to/migrate-ngit-relay-to-ngit-grasp.md index d01bbf2..4c3a4ba 100644 --- a/docs/how-to/migrate-ngit-relay-to-ngit-grasp.md +++ b/docs/how-to/migrate-ngit-relay-to-ngit-grasp.md | |||
| @@ -51,7 +51,7 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c | |||
| 51 | ``` | 51 | ``` |
| 52 | ┌─────────────────────────────────────────────────────────────────┐ | 52 | ┌─────────────────────────────────────────────────────────────────┐ |
| 53 | │ PHASE 1: Fetch Events (~30s, local) │ | 53 | │ PHASE 1: Fetch Events (~30s, local) │ |
| 54 | │ scripts/migration/01-fetch-events.sh <relay> <output-dir> │ | 54 | │ migration-scripts/01-fetch-events.sh <relay> <output-dir> │ |
| 55 | ├─────────────────────────────────────────────────────────────────┤ | 55 | ├─────────────────────────────────────────────────────────────────┤ |
| 56 | │ Fetches from relay: │ | 56 | │ Fetches from relay: │ |
| 57 | │ - kind 30618 (state events) │ | 57 | │ - kind 30618 (state events) │ |
| @@ -64,7 +64,7 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c | |||
| 64 | ↓ | 64 | ↓ |
| 65 | ┌─────────────────────────────────────────────────────────────────┐ | 65 | ┌─────────────────────────────────────────────────────────────────┐ |
| 66 | │ PHASE 2: Git Sync Check (~20 mins, VPS required) │ | 66 | │ PHASE 2: Git Sync Check (~20 mins, VPS required) │ |
| 67 | │ scripts/migration/10-check-git-sync.sh <events> <git-base> <out>│ | 67 | │ migration-scripts/10-check-git-sync.sh <events> <git-base> <out>│ |
| 68 | ├─────────────────────────────────────────────────────────────────┤ | 68 | ├─────────────────────────────────────────────────────────────────┤ |
| 69 | │ For each state event, compares refs to actual git data on disk. │ | 69 | │ For each state event, compares refs to actual git data on disk. │ |
| 70 | │ │ | 70 | │ │ |
| @@ -78,8 +78,8 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c | |||
| 78 | ↓ | 78 | ↓ |
| 79 | ┌─────────────────────────────────────────────────────────────────┐ | 79 | ┌─────────────────────────────────────────────────────────────────┐ |
| 80 | │ PHASE 3: Categorize & Compare (fast, local) │ | 80 | │ PHASE 3: Categorize & Compare (fast, local) │ |
| 81 | │ scripts/migration/20-categorize.sh <sync-status> <output-dir> │ | 81 | │ migration-scripts/20-categorize.sh <sync-status> <output-dir> │ |
| 82 | │ scripts/migration/21-compare-relays.sh <prod> <archive> <out> │ | 82 | │ migration-scripts/21-compare-relays.sh <prod> <archive> <out> │ |
| 83 | ├─────────────────────────────────────────────────────────────────┤ | 83 | ├─────────────────────────────────────────────────────────────────┤ |
| 84 | │ 20-categorize.sh applies 4-category logic: │ | 84 | │ 20-categorize.sh applies 4-category logic: │ |
| 85 | │ - cat1: complete match (all refs match) │ | 85 | │ - cat1: complete match (all refs match) │ |
| @@ -87,18 +87,20 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c | |||
| 87 | │ - cat3: partial match (some refs match) │ | 87 | │ - cat3: partial match (some refs match) │ |
| 88 | │ - cat4: no match (git exists but refs don't match) │ | 88 | │ - cat4: no match (git exists but refs don't match) │ |
| 89 | │ │ | 89 | │ │ |
| 90 | │ 21-compare-relays.sh finds gaps: │ | 90 | │ 21-compare-relays.sh compares prod vs archive: │ |
| 91 | │ - in prod but not archive │ | 91 | │ - complete-in-both.txt (no action needed) │ |
| 92 | │ - in archive but not prod │ | 92 | │ - complete-prod-missing-archive.txt (needs investigation) │ |
| 93 | │ - different status between relays │ | 93 | │ - complete-prod-incomplete-archive.txt (sync in progress?) │ |
| 94 | │ - incomplete-in-both.txt (git data incomplete) │ | ||
| 95 | │ - in-archive-not-prod.txt (deleted or new) │ | ||
| 94 | │ │ | 96 | │ │ |
| 95 | │ Output: category-{1,2,3,4}.txt, relay-gaps.txt │ | 97 | │ Output: category-{1,2,3,4}.txt, comparison/*.txt, summary.txt │ |
| 96 | └─────────────────────────────────────────────────────────────────┘ | 98 | └─────────────────────────────────────────────────────────────────┘ |
| 97 | ↓ | 99 | ↓ |
| 98 | ┌─────────────────────────────────────────────────────────────────┐ | 100 | ┌─────────────────────────────────────────────────────────────────┐ |
| 99 | │ PHASE 4: Log-Based Categories (VPS required) │ | 101 | │ PHASE 4: Log-Based Categories (VPS required) │ |
| 100 | │ scripts/migration/30-extract-parse-failures.sh <service> <out> │ | 102 | │ migration-scripts/30-extract-parse-failures.sh <service> <out> │ |
| 101 | │ scripts/migration/31-extract-purgatory-expiry.sh <service> <out>│ | 103 | │ migration-scripts/31-extract-purgatory-expiry.sh <service> <out>│ |
| 102 | ├─────────────────────────────────────────────────────────────────┤ | 104 | ├─────────────────────────────────────────────────────────────────┤ |
| 103 | │ Extracts structured log entries from journalctl: │ | 105 | │ Extracts structured log entries from journalctl: │ |
| 104 | │ - Parse failures: [PARSE_FAIL] kind=X event_id=Y reason=Z │ | 106 | │ - Parse failures: [PARSE_FAIL] kind=X event_id=Y reason=Z │ |
| @@ -112,7 +114,7 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c | |||
| 112 | ↓ | 114 | ↓ |
| 113 | ┌─────────────────────────────────────────────────────────────────┐ | 115 | ┌─────────────────────────────────────────────────────────────────┐ |
| 114 | │ PHASE 5: Final Classification (fast, local) │ | 116 | │ PHASE 5: Final Classification (fast, local) │ |
| 115 | │ scripts/migration/40-classify-actions.sh <all-inputs> <out> │ | 117 | │ migration-scripts/40-classify-actions.sh <all-inputs> <out> │ |
| 116 | ├─────────────────────────────────────────────────────────────────┤ | 118 | ├─────────────────────────────────────────────────────────────────┤ |
| 117 | │ Combines all data sources to produce final classification: │ | 119 | │ Combines all data sources to produce final classification: │ |
| 118 | │ │ | 120 | │ │ |
| @@ -136,27 +138,38 @@ The analysis is split into modular phases for fast iteration. Phases 1-3 and 5 c | |||
| 136 | work/migration-analysis-YYYYMMDD-HHMM/ | 138 | work/migration-analysis-YYYYMMDD-HHMM/ |
| 137 | ├── prod/ | 139 | ├── prod/ |
| 138 | │ ├── raw/ | 140 | │ ├── raw/ |
| 139 | │ │ ├── state-events.json | 141 | │ │ ├── state-events.json # Phase 1 output |
| 140 | │ │ ├── announcements.json | 142 | │ │ ├── announcements.json # Phase 1 output |
| 141 | │ │ └── deletions.json | 143 | │ │ └── deletions.json # Phase 1 output |
| 142 | │ ├── git-sync-status.tsv | 144 | │ ├── git-sync-status.tsv # Phase 2 output (optional) |
| 143 | │ └── category-{1,2,3,4}.txt | 145 | │ ├── category1-complete-match.txt # Phase 2/3 output |
| 146 | │ ├── category2-empty-blank.txt # Phase 2/3 output | ||
| 147 | │ ├── category3-partial-match.txt # Phase 2/3 output | ||
| 148 | │ └── category4-no-match.txt # Phase 2/3 output | ||
| 144 | ├── archive/ | 149 | ├── archive/ |
| 145 | │ ├── raw/ | 150 | │ ├── raw/ |
| 146 | │ │ ├── state-events.json | 151 | │ │ ├── state-events.json |
| 147 | │ │ ├── announcements.json | 152 | │ │ ├── announcements.json |
| 148 | │ │ └── deletions.json | 153 | │ │ └── deletions.json |
| 149 | │ ├── git-sync-status.tsv | 154 | │ ├── git-sync-status.tsv |
| 150 | │ └── category-{1,2,3,4}.txt | 155 | │ ├── category1-complete-match.txt |
| 156 | │ ├── category2-empty-blank.txt | ||
| 157 | │ ├── category3-partial-match.txt | ||
| 158 | │ └── category4-no-match.txt | ||
| 151 | ├── logs/ | 159 | ├── logs/ |
| 152 | │ ├── parse-failures.txt | 160 | │ ├── parse-failures.txt # Phase 4 output |
| 153 | │ └── purgatory-expired.txt | 161 | │ └── purgatory-expired.txt # Phase 4 output |
| 154 | ├── comparison/ | 162 | ├── comparison/ |
| 155 | │ └── relay-gaps.txt | 163 | │ ├── complete-in-both.txt # Phase 3 output (no action) |
| 164 | │ ├── complete-prod-missing-archive.txt # Phase 3 output (investigate) | ||
| 165 | │ ├── complete-prod-incomplete-archive.txt # Phase 3 output (sync in progress?) | ||
| 166 | │ ├── incomplete-in-both.txt # Phase 3 output (git incomplete) | ||
| 167 | │ ├── in-archive-not-prod.txt # Phase 3 output (deleted/new) | ||
| 168 | │ └── summary.txt # Phase 3 output (human-readable) | ||
| 156 | └── results/ | 169 | └── results/ |
| 157 | ├── no-action-required.txt | 170 | ├── no-action-required.txt # Phase 5 output |
| 158 | ├── action-required.txt | 171 | ├── action-required.txt # Phase 5 output |
| 159 | └── manual-investigation.txt | 172 | └── manual-investigation.txt # Phase 5 output |
| 160 | ``` | 173 | ``` |
| 161 | 174 | ||
| 162 | ## Prerequisites | 175 | ## Prerequisites |