| Age | Commit message (Collapse) | Author |
|---|
|
Fix pre-existing clippy lints:
- &PathBuf -> &Path in audit_cleanup.rs
- too_many_arguments on process_newly_available_git_data,
process_purgatory_announcements, and HttpService::new
- clone_on_copy for PublicKey (Copy type) in purgatory cleanup loop
|
|
|
|
Per design doc decision #4: state event arrival resets the 30-minute
protocol timer for purgatory announcements. This prevents premature
expiry during slow sync operations where the repo is actively receiving
metadata but git data hasn't arrived yet.
Extends expiry for all owners whose announcement authorized the state
event, and triggers revival if the announcement was soft-expired.
|
|
When a state event arrives and the required commits already exist in
another maintainer's repo on the same relay, process_state_with_git_data
copies the OIDs across and aligns refs — but never called
process_purgatory_announcements for the target repos. Any announcement
waiting in purgatory for that repo stayed there indefinitely.
Fix: after process_state_with_git_data, call process_newly_available_git_data
for each target repo (those that received copied OIDs) so purgatory
announcements are promoted immediately.
|
|
Route new announcements to purgatory instead of accepting immediately.
Announcements are promoted to the database when git data arrives,
ensuring we only serve announcements for repos with actual content.
Implemented:
- AnnouncementPurgatoryEntry type and DashMap store
- Route new announcements to purgatory (replacement announcements skip)
- Promote announcements on git data arrival (process_purgatory_announcements)
- Authorization checks purgatory announcements (fetch_repository_data_with_purgatory)
- State policy uses purgatory announcements for maintainer validation
- Cleanup task handles announcement expiry
- Updated count()/cleanup() to 3-tuples
Known broken:
- test_archive_read_only_creates_bare_repo fails: sync module does not
treat purgatory announcements as confirmed repos, so per-repo sync
(state events, PRs) is never triggered for purgatory announcements
- Announcement persistence (save/restore) not implemented
- SyncLevel (StateOnly vs Full) not implemented
- Soft expiry two-phase not implemented
- Expiry extension on state event / git auth not wired up
|
|
This merge includes critical bug fixes and comprehensive migration tooling
developed during the relay.ngit.dev migration effort.
Bug Fixes:
- Fix git protocol error handling to return HTTP 200 with ERR pkt-line
- Fix naughty list false positives and DNS failure identification
- Fix database query filters in load_existing_events (remove .since())
- Fix OID fetch tracking to distinguish 0 OIDs from successful fetches
- Fix purgatory event source tracking for filtered expiry logging
- Implement OID retry logic for 'not our ref' errors
Migration Tools & Documentation:
- Complete 5-phase migration analysis pipeline with orchestration script
- Phase 1: Event fetching from source relay
- Phase 2: Git sync verification
- Phase 3: Categorization and relay comparison
- Phase 4: Log extraction (parse failures, purgatory expiry)
- Phase 5: Action classification for migration decisions
- Comprehensive migration guide with lessons learned
- Troubleshooting guide for permission and corruption issues
Configuration:
- Add NGIT_LOG_LEVEL configuration option
- Update git throttle limits to 60/minute
- Improve logging throughout for better observability
|
|
Add EventSource enum (Direct/Sync) to purgatory entries to distinguish
between user-submitted events and sync-fetched events. This enables:
- WARN-level logging for direct submissions that expire (user should know)
- DEBUG-level logging for sync-fetched expirations (expected behavior)
- Source upgrade from Sync→Direct if user submits after sync
- Expiry timer reset on source upgrade (fresh 30-min window for user)
The source is included in [PURGATORY_EXPIRED] logs as source=direct or
source=sync for easy filtering.
|
|
State events from remote relays for repos we don't host are expected
rejections during proactive sync. Changed to only WARN for user-submitted
events (potential misconfiguration/attack) while using DEBUG for synced
events (normal operation).
This reduces log noise from ~1967 warnings to <10 warnings in a 30-second
production sync test, making real issues visible again.
|
|
|
|
Add comprehensive authorization checks to ensure state events are only
accepted from maintainers of accepted repository announcements. This
implements the core GRASP-01 requirement that pushes must match the
latest state announcement "respecting the maintainer set."
Changes:
1. StatePolicy authorization (src/nostr/policy/state.rs):
- Check authorization BEFORE git data validation (fail-fast)
- Reject if no announcement exists for repository
- Reject if author not in maintainer set
- Use existing helpers: fetch_repository_data() and
pubkey_authorised_for_repo_owners()
- Structured logging for all rejections
2. Purgatory invalidation (src/nostr/builder.rs):
- New method: check_purgatory_state_events_for_identifier()
- Called when announcements accepted (Accept and AcceptMaintainer)
- Re-evaluates state events in purgatory for the identifier
- Processes newly-authorized events (releases from purgatory)
- Keeps unauthorized events for natural expiry (30 min)
- Enables retroactive authorization when announcements arrive late
3. Purgatory sync authorization (src/git/sync.rs):
- Check authorization BEFORE processing git data
- Remove unauthorized events from purgatory (permanent rejection)
- Prevents processing even if git data arrives first
- Structured logging for monitoring
4. Rejected events tracking (src/sync/rejected_index.rs):
- Add support for tracking rejected state events
- New methods: add_state(), contains_state()
- Separate metrics for state rejections
- Enables sync to avoid re-fetching rejected states
5. Sync metrics (src/sync/metrics.rs, src/sync/mod.rs):
- Add state-specific metrics (hot cache, cold index)
- Track rejected states separately from announcements
- Support monitoring of authorization rejections
6. Comprehensive tests (tests/state_authorization.rs):
- test_reject_state_without_announcement
- test_reject_state_from_unauthorized_author
- test_accept_state_from_announcement_author
- test_accept_state_from_maintainer
Security Impact:
- Before: State events could be published by anyone
- After: Only maintainers can publish state events
- Defense-in-depth: Authorization checked at 3 points:
1. On arrival (StatePolicy)
2. On announcement acceptance (purgatory re-evaluation)
3. On git data arrival (purgatory sync)
All tests pass:
- 248 unit tests
- 51 NIP-34 announcement tests
- 4 new state authorization tests
- 9 rejected index tests
Closes: State authorization requirement from GRASP-01 spec
|
|
|
|
Adds expired event tracking to prevent proactive sync from repeatedly
fetching and re-adding events that expired from purgatory without
finding git data.
Key features:
- Track expired events for 7 days to prevent re-sync loops
- Distinguish synced vs user-submitted events (via socket address)
- Allow users to retry expired events (git data might now be available)
- Reject synced expired events (prevents infinite loop)
- Daily cleanup of expired event records older than 7 days
Implementation:
- Added expired_events: DashMap<EventId, Instant> to Purgatory
- Updated event_ids() to include both purgatory + expired events
- Added is_expired(), mark_expired(), cleanup_expired_events()
- Updated cleanup() to mark expired events automatically
- Added is_synced detection in WritePolicy (localhost:0 = synced)
- Policy layer checks is_synced && is_expired() before rejecting
Behavior:
- Negentropy: Filters expired events before fetching (optimal)
- REQ+EOSE: Rejects synced expired events at policy layer
- User submissions: Always allowed to retry (skip expired check)
Testing:
- Added 5 new tests for expired event tracking
- All 222 tests passing
Fixes the infinite re-sync loop where events without git data would
expire, get synced again, expire again, repeat forever.
|
|
Eliminates code duplication by extracting core event processing into
reusable functions. All state and PR event processing now uses the same
unified logic from src/git/process.rs.
Changes:
- Add src/git/process.rs with unified processing functions
- process_state_with_git_data() for state events
- process_pr_with_git_data() for PR events
- Pure functions with comprehensive result types
- Refactor policy handlers to use unified processing
- src/nostr/policy/state.rs: Remove ~70 lines of duplicated logic
- src/nostr/policy/pr_event.rs: Remove ~40 lines of duplicated logic
- Refactor purgatory processing to use unified functions
- src/git/sync.rs: Remove ~125 lines of duplicated logic
- Make extract_owner_from_repo_path() public for reuse
Benefits:
- DRY: Single source of truth for event processing
- Testable: Pure functions with clear contracts
- Maintainable: Changes happen in one place
- Consistent: All code paths use same logic
All 217 unit tests + 40 integration tests pass (257/257).
|
|
Phase 13 of purgatory-sync-redesign:
- Add sync loop startup in main.rs (RealSyncContext + ThrottleManager + start_sync_loop)
- Update add_state() and add_pr() to automatically enqueue for background sync
- Remove start_state_sync() call from state.rs (now handled by sync loop)
- Remove orphaned legacy functions: sync_state_git_data, fetch_missing_oids_from_server,
get_most_complete_local_repo, identify_missing_oids, get_date_of_most_recent_commit_on_default_branch
- Clean up unused imports in purgatory/mod.rs
|
|
- Remove duplicate AlignmentResult struct from nostr/policy/state.rs
- Remove duplicate align_repository_with_state method from StatePolicy
- Import and use the canonical implementation from git::sync
- Re-export AlignmentResult from git::sync in policy/mod.rs
The git::sync version is preferred as it:
- Handles symbolic refs (ref:) properly by skipping them
- Uses git::oid_exists which is more general than git::commit_exists
- Has a cleaner iteration pattern (delete first, then update/create)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Main lib (src/):
- Add #[allow(dead_code)] for build_info field (stored to prevent Prometheus unregistration)
- Add #[allow(dead_code)] for first_seen field (reserved for future rate limiting)
- Replace .or_insert_with(RelaySyncNeeds::default) with .or_default()
- Replace manual div_ceil implementations with .div_ceil(100)
Test code (tests/):
- Replace .expect(&format!(...)) with .unwrap_or_else(|_| panic!(...))
- Remove needless borrows in fetch_metrics() calls
- Add #[allow(dead_code)] and #[allow(unused_imports)] to test helpers module
grasp-audit:
- Apply cargo fmt to fix formatting
|
|
Split the ~900 line Nip34WritePolicy into focused sub-policies for improved
testability and maintainability:
- AnnouncementPolicy - Repository announcement validation
- StatePolicy - State event validation + ref alignment
- PrEventPolicy - PR/PR Update validation
- RelatedEventPolicy - Forward/backward reference checking
The main Nip34WritePolicy now delegates to these sub-policies via a shared
PolicyContext that provides domain, database, and git_data_path.
Also updates:
- README.md: Accurate project structure reflecting actual implementation
- docs/learnings: Marks this technical debt item as complete
|