| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
collect_all_authorized_maintainers
Both were pub functions with no callers. Clippy doesn't flag dead pub
items because the compiler treats them as potentially used by external
crates - only private items trigger the dead_code lint.
|
|
fetch_repository_data_{excluding,with}_purgatory
The old name was ambiguous - it wasn't clear whether purgatory was
included or not. The two variants are now explicitly named:
- fetch_repository_data_excluding_purgatory: DB only
- fetch_repository_data_with_purgatory: DB + purgatory overlay
SyncContext trait method also renamed to fetch_repository_data_with_purgatory
to match the free function it delegates to.
|
|
|
|
state event copy
When git data is fetched into owner A's repo and a state event for owner B
is released from purgatory (copying OIDs from A's repo to B's repo via
process_state_with_git_data), owner B's purgatory announcement was never
promoted. process_purgatory_announcements only promotes the announcement
for the owner derived from source_repo_path (owner A), so owner B's
announcement stayed in purgatory with its 30-minute expiry timer running.
30 minutes later the cleanup task would soft-expire owner B's entry,
deleting the bare repository even though the announcement had been
effectively satisfied.
Fix: after a state event is successfully saved to the database, iterate
over all announcements in db_repo_data and promote any purgatory
announcement for owners whose repos received OIDs via the copy (i.e.
repos other than source_repo_path).
|
|
01-fetch-events.sh: nak buffers output when stdout is not a TTY, causing
it to hang silently in non-interactive SSH sessions. Wrap with 'script'
to provide a pseudo-TTY, then strip the injected carriage returns and
connection banner line from the output.
40-classify-actions.sh: bash 5.3 treats ${#assoc[@]} and array iteration
as unbound variable errors under set -u when arrays are empty. Replace
${#assoc[@]} with explicit counters and guard array iterations with
set +u/set -u.
|
|
Extends purgatory to hold repository announcements until git data arrives,
preventing empty repositories from being served to clients.
When an announcement is received, a bare repo is created immediately and the
announcement is held in purgatory. It is only promoted and served once a git
push confirms real content exists. If no push arrives before expiry, the bare
repo is deleted and the announcement is silently discarded.
Key behaviours:
- Soft expiry: announcements are hidden from clients but kept alive while git
pushes are in progress, reviving on successful push
- Expiry is extended when a matching state event or git push is observed
- NIP-09 deletion events remove announcements from purgatory
- Purgatory state (announcements, state events, PR events, expired set) is
persisted to disk on graceful shutdown and restored on startup, with elapsed
downtime subtracted from expiry deadlines
- Purgatory announcements drive StateOnly sync in the sync system so state
events are fetched from listed relays before promotion
- SyncLevel added to RepoSyncIndex to distinguish purgatory repos (StateOnly)
from promoted repos (Full L2+L3 sync)
|
|
|
|
Extends purgatory persistence to include announcement purgatory entries.
On graceful shutdown, non-soft-expired announcements are serialised to
purgatory-state.json alongside state/PR/expired events; on startup they
are restored, skipping any entry whose bare repo path no longer exists.
Updates purgatory-design.md to reflect that purgatory persists through
graceful shutdown and documents the new PurgatoryState disk format.
Adds create_announcement_event helper to purgatory_helpers and three new
integration tests in purgatory_persistence covering the full save/restore
cycle, missing-repo skip, and the combined roundtrip with all entry types.
|
|
Remove the pre-implementation planning docs (announcements-purgatory-design.md
and announcements-purgatory-implementation.md) now that the feature is built.
Update the three living docs to reflect what was actually implemented:
- purgatory-design.md: expanded to cover all three purgatory stores
(announcement, state, PR), including AnnouncementPurgatoryEntry structure,
two-phase soft expiry lifecycle, expiry extension triggers, promotion flow,
and updated integration points and file structure
- grasp-02-proactive-sync.md: added SyncLevel enum (Full/StateOnly) to
RepoSyncNeeds, documented the purgatory announcement sync timer as the
registration path for purgatory announcements, updated filter building
to describe build_sync_level_aware_filters() and StateOnly behaviour
- grasp-02-proactive-sync-purgatory-git-data.md: expanded to cover
announcement purgatory as a third entry type, added Timeline E showing
soft-expiry and revival, replaced the single expiry section with separate
hard-expiry (state/PR) and two-phase soft-expiry (announcements) sections
with full justification for the 24-hour extended retention window
|
|
Kind 5 deletion events referencing a PR or PR-update event by e-tag now
remove the matching purgatory entry, provided the deletion author matches
the PR event author. Placeholders (git data arrived before the event) are
not removed since they have no author to verify against.
PR purgatory is keyed by event ID hex so this is an O(1) lookup, checked
before the O(n) announcement and state event scans.
|
|
The previous tests deleted purgatory announcements (kind 30617) and checked
for bare-repo absence via git ls-remote, which would corrupt shared-mode
test state by destroying repos other tests depend on.
New approach tests deletion of purgatory state events (kind 30618) instead:
- e-tag test: promotes a repo, creates a unique commit locally, submits a
state event pointing to it (enters purgatory), deletes the state event by
event ID, then verifies git push of that commit is rejected.
- a-tag coordinate test: promotes a repo, generates a fresh maintainer
keypair, sends a replacement announcement adding that maintainer, submits
a state event signed by the new maintainer (enters purgatory), deletes by
coordinate 30618:<new_maintainer_pubkey>:<identifier>, then verifies git
push is rejected.
Also extends DeletionPolicy to handle kind 30618 state events in purgatory
for both e-tag (event ID) and a-tag (coordinate) deletion paths.
|
|
Kind 5 deletion events signed by the announcement author now evict the
corresponding purgatory entry and delete the bare repository from disk.
Both NIP-09 reference styles are supported:
- e tag (event ID): matches the purgatory entry whose event ID equals the tag value
- a tag (coordinate 30617:<pubkey>:<identifier>): matches by coordinate, only
removes entries with created_at <= deletion event created_at per NIP-09 spec
Author-only enforcement: coordinate pubkey and e-tag owner must match the
deletion event pubkey; third-party deletion attempts are silently ignored.
Includes 6 unit tests and 2 integration tests (event ID and coordinate paths).
|
|
If remove_dir_all fails, leave the entry untouched so the next cleanup
cycle retries the deletion automatically. Previously a failed deletion
would still set soft_expired=true and extend the expiry, meaning the
bare repo would never be retried.
|
|
Per design doc decision #4: when git auth finds a matching state event
in purgatory that authorizes a push, extend the announcement's expiry.
The repo is actively receiving git data so the announcement should not
expire prematurely. Also triggers revival of soft-expired announcements.
|
|
Per design doc decision #4: state event arrival resets the 30-minute
protocol timer for purgatory announcements. This prevents premature
expiry during slow sync operations where the repo is actively receiving
metadata but git data hasn't arrived yet.
Extends expiry for all owners whose announcement authorized the state
event, and triggers revival if the announcement was soft-expired.
|
|
Two-phase expiry for announcement purgatory entries:
- Phase 1 (initial 30min timeout): delete bare repo, set soft_expired=true,
extend expiry by 24h so the event is retained for potential revival
- Phase 2 (24h extended timeout): fully remove from purgatory
Revival: extend_announcement_expiry() now recreates the bare git repo
when called on a soft-expired entry (triggered by state event or git auth),
clearing soft_expired and resetting the expiry window.
|
|
Remove the redundant inline kind-30617 registration block from the sync
event loop and the three is_generic/recompute_new_sync_filters_for_relay
calls from confirm_batch error paths. The purgatory announcement sync
timer (run_purgatory_announcement_sync) is now the sole registration path.
Consolidate NGIT_SYNC_BATCH_WINDOW_MS and NGIT_PURGATORY_SYNC_INTERVAL_MS
into a single NGIT_TEST=1 flag that sets both timers to 200ms, replacing
two ad-hoc env vars with one reusable test-mode flag.
|
|
When a state event arrives and the required commits already exist in
another maintainer's repo on the same relay, process_state_with_git_data
copies the OIDs across and aligns refs — but never called
process_purgatory_announcements for the target repos. Any announcement
waiting in purgatory for that repo stayed there indefinitely.
Fix: after process_state_with_git_data, call process_newly_available_git_data
for each target repo (those that received copied OIDs) so purgatory
announcements are promoted immediately.
|
|
When an owner announcement is promoted from purgatory via a git push,
any maintainer announcements sitting in the rejected_events_index hot
cache were never re-processed. The invalidate_and_get call only existed
in SyncManager::process_event_static (the nostr sync path); the git push
promotion path (http -> handlers -> git::sync) had no access to the
rejected_events_index at all.
Thread rejected_events_index and write_policy through the git push path:
- process_purgatory_announcements: after saving the promoted announcement,
parse its maintainers tag and call invalidate_and_get() for each, then
re-process any returned hot-cache events via admit_event + save
- process_newly_available_git_data: accept optional write_policy and
rejected_events_index, pass them through to process_purgatory_announcements
- handle_receive_pack: accept Arc<Nip34WritePolicy> and
Arc<RejectedEventsIndex>, pass them to process_newly_available_git_data
- HttpService / run_server: carry the two new fields, clone into each
handle_receive_pack call
- main.rs: obtain rejected_events_index from sync_manager before moving
it into its task; wrap write_policy in Arc for the HTTP server
- RealSyncContext::process_newly_available_git_data: pass None for both
new params (purgatory sync path already handles this via
SyncManager::process_event_static)
Also rewrite the maintainer_reprocessing integration tests to correctly
exercise the hot-cache path now that announcements require git data
before being released from purgatory:
- Start relay_b with relay_a as bootstrap so its SyncManager syncs
maintainer announcements via negentropy before the owner git push
- Use push_unique_git_data_to_relay (new helper) to give each maintainer
a distinct commit hash, preventing git from skipping pack transfer
- Make wait_for_event_on_relay poll in a retry loop so transient timing
gaps between DB write and query do not cause false negatives
|
|
Recursive discovery relied on announcement events being gossiped across
relays regardless of whether they listed the service. Now that
announcements enter purgatory until state event and git data arrive,
cross-relay discovery cannot be triggered by a synced announcement alone,
making the three-relay recursive discovery scenario impossible.
|
|
All sync tests now create a local git repo, send announcement + state
event to the source relay, and push git data to release both from
purgatory before the syncing relay starts bootstrap sync.
|
|
Previously run_sync_test used a SmartGitServer external to the relay,
but never pushed to the source relay itself. With the announcement
purgatory feature, announcements stay in purgatory until git data
arrives. By using push_to_relay to the source relay, both the
announcement and state event are released from purgatory before
the syncing relay starts, allowing the announcement to be synced.
|
|
wrong-commit PR tests
PRWrongCommitPushedBeforeEvent and test_push_to_nostr_ref_with_wrong_commit_after_event_received_rejected
were calling create_deterministic_commit_with_variant(CommitVariant::Owner) on a clone that already
had test.txt with 'Initial commit\n' content from OwnerStateDataPushed. Writing identical content
staged nothing so git commit failed silently.
Now that ValidRepoServed always depends on OwnerStateDataPushed (git data pushed), the clone is
never empty - use create_commit (unique file) instead since the wrong commit only needs to differ
from PR_TEST_COMMIT_HASH, not be deterministic.
|
|
OwnerStateDataPushed was secretly building and sending the state event
internally, with no corresponding fixture in the chain. Add OwnerRepoState
as the explicit 'state event sent, sitting in purgatory' step so the
dependency chain reads: ValidRepoSent -> OwnerRepoState -> OwnerStateDataPushed -> ValidRepoServed.
OwnerStateDataPushed now reads the state event from the OwnerRepoState cache
rather than rebuilding it, and only owns the git push + purgatory release.
|
|
user-submitted purgatory announcement
The test was failing because submitting an announcement directly to syncing_relay
(user-submitted, no bootstrap) leaves the announcement in purgatory with no mechanism
to trigger relay discovery - there are no existing sync connections whose batch EOSE
would fire recompute_new_sync_filters_for_relay.
Fix: start syncing_relay with source_grasp as bootstrap. The promoted announcement
syncs via L1 generic filter → purgatory (no local git data) → StateOnly subscription
→ state event → purgatory sync fetches git data → announcement promoted → SelfSubscriber
upgrades to Full → connects to mock_relay → PR event synced and promoted.
The test's primary purpose (PR event partial OID aggregation from multiple clone
URL sources) is fully preserved.
|
|
Instead of threading repo_sync_index through PolicyContext/builder.rs/main.rs
to handle user-submitted purgatory announcements, add a simple background
timer (run_purgatory_announcement_sync, every 5s) that scans the purgatory
for announcement entries and registers them in repo_sync_index as StateOnly.
This is simpler and covers both flows:
- Sync-path announcements: inline registration still happens during event
processing (sync/mod.rs:1839+), timer provides a safety net
- User-submitted announcements: SelfSubscriber never sees them (rejected
from DB), timer is the primary registration path
The timer calls sync_purgatory_announcements_to_index() which:
1. Snapshots purgatory via new announcements_for_sync() public method
2. Or_inserts StateOnly entries (never downgrades Full entries)
3. Detects newly added relay URLs and calls handle_new_sync_filters to
connect and subscribe - fixing the failing test that expected relay
discovery from a user-submitted purgatory announcement
Removes: repo_sync_index field from PolicyContext, set/get_repo_sync_index
methods, set_repo_sync_index on Nip34WritePolicy, wiring in main.rs, and
the inline AcceptPurgatory registration block in builder.rs.
|
|
redundant test
|
|
negentropy fallback
Three targeted fixes for purgatory announcement sync:
1. SelfSubscriber sync_level upgrade: After or_insert_with in process_batch,
always set entry.sync_level = SyncLevel::Full so that when a promoted
announcement is broadcast via notify_event and SelfSubscriber receives it,
an existing StateOnly entry gets upgraded to Full and PR event subscriptions
are triggered immediately (not delayed up to 24h).
2. Negentropy fallback filter split: In handle_eose, when falling back from
negentropy to REQ+EOSE, split batch_repos by SyncLevel and call
build_sync_level_aware_filters instead of build_layer2_and_layer3_filters.
Prevents StateOnly (purgatory) repos from getting Layer 2 #a/#A/#q filters
prematurely, which caused nostr-sdk client deduplication to permanently
drop PR events after orphan rejection.
3. Recompute sync filters after announcement batch EOSE: Add
recompute_new_sync_filters_for_relay calls at all three batch-completion
paths in handle_eose for generic filter (announcement) batches. This
triggers state-only subscriptions for any purgatory repos registered during
that batch, fixing the 24h delay before state event sync starts.
4. User-submitted purgatory announcements: Add repo_sync_index field to
PolicyContext with setter/getter, wire in main.rs after SyncManager
creation, and register in AcceptPurgatory handler so user-submitted
announcements get StateOnly sync started immediately.
5. Update archive tests: test_archive_without_state_events_does_not_sync_git
updated to reflect that StateOnly subscription now proactively fetches
state events from source relays. test_archive_read_only_creates_bare_repo
un-ignored as it now works end-to-end.
|
|
premature PR event delivery"
This reverts commit 806936e7d1aab5dfd0c2ad6b98a115122dc1785c.
|
|
after announcement promotion"
This reverts commit d76003b629a4a03dba23a8a1c41da6e4ac4c30cf.
|
|
The tests now correctly reflect the actual purgatory behavior:
1. Announcement goes to purgatory (StateOnly) - not immediately accepted
2. State event goes to purgatory
3. Git push promotes announcement to Full and releases state event
4. PR event is sent AFTER announcement promotion (accepted since repo is Full)
5. PR commit push releases PR event from purgatory
This matches the design: announcements require git data validation before
being promoted to the database, which means PR events can only be accepted
for repos with promoted announcements.
Also routes relay stdout to /tmp/relay-{port}.log for easier debugging.
|
|
announcement promotion
When git data arrives for a purgatory announcement and promotes it to the
database, the relay now:
1. Upgrades the announcement's sync level in RepoSyncIndex from StateOnly
to Full (git/sync.rs: process_purgatory_announcements)
2. Sends AddFilters actions to SyncManager for all connected relays, using
Full sync filters (Layer 2 #a/#A/#q) to subscribe to PR events
(purgatory/sync/context.rs: RealSyncContext.process_newly_available_git_data)
3. For user-submitted purgatory announcements, registers the repo in
RepoSyncIndex with StateOnly level and sends AddFilters to SyncManager
so it discovers and connects to relays listed in the announcement tags
(nostr/builder.rs: handle_announcement AcceptPurgatory path)
The RealSyncContext now accepts optional repo_sync_index and sync_action_tx
parameters. main.rs wires these up from SyncManager. PolicyContext gains
repo_sync_index and sync_action_tx fields for the write policy path.
|
|
premature PR event delivery
StateOnly repos in a pending batch had their repo IDs included in the
negentropy REQ+EOSE fallback, which called build_layer2_and_layer3_filters.
This generated #a/#A/#q tag filters for repos whose announcements were
still in purgatory (not yet promoted to the database).
When the remote relay responded with PR events matching those filters,
the write policy correctly rejected them as 'orphan' (no accepted repo
in DB yet). However, nostr-sdk's client-level deduplication then silently
dropped the same event on all subsequent deliveries, making it permanently
unavailable even after the announcement was promoted.
Fix: split batch_repos into full vs state-only by consulting repo_sync_index
at fallback time, then call build_sync_level_aware_filters which only
generates #a/#A/#q filters for Full repos. StateOnly repos only get
the kind 30618 + #d filter they were originally subscribed with.
|
|
Kind 30617 announcements now go to purgatory (not DB) until git data
arrives. Kind 1621 issues referencing purgatory-only repos are rejected.
Use kind 10317 (GitUserGraspList) from two keypairs instead - these are
unconditionally accepted and stored in DB, making them visible to
negentropy sync.
|
|
purgatory
remove_purgatory_announcement() was unconditionally wiping all state
events for an identifier when one owner's announcement was evicted.
State events are keyed by identifier alone, so this incorrectly
discarded state events belonging to a different owner's repository
sharing the same identifier string. Now only removes state events if
no other owner's announcement remains in purgatory for that identifier.
|
|
An older rejected announcement (e.g. a relay replay of a superseded
event) was incorrectly evicting a newer purgatory entry for the same
pubkey+identifier. Now only evict when the incoming event's created_at
is strictly greater than the stored entry's created_at.
|
|
The sync loop calls fetch_repository_data() to get clone URLs so it knows
where to fetch git data from. Previously this only queried the database,
which means an announcement still in purgatory (no git data yet) would
return no clone URLs, so the sync loop could never fetch the git data
needed to promote the announcement - a circular deadlock.
Fix by switching to fetch_repository_data_with_purgatory() which combines
database announcements with purgatory announcements. Update the trait
method's doc comment to document this behaviour.
The mock implementation in tests is unaffected since it returns
pre-configured data rather than delegating to either function.
|
|
Previously, has_active_announcement() only queried the database, so when
a newer announcement arrived for the same (pubkey, identifier) while the
original was still in purgatory, it was incorrectly routed as a brand-new
announcement (AcceptPurgatory) rather than replacing the existing entry.
This change splits the logic into two cases:
- If the existing entry is in the database: return Accept (replacement) as before
- If the existing entry is only in purgatory: replace the purgatory entry via
add_announcement() (which overwrites by key) and extend expiries for both the
announcement and any waiting state events, then return Accept
- If the owner sends a Reject-classified announcement (service removed) but has
a purgatory entry: clear the purgatory entry, delete the bare repo, and remove
any waiting state events before rejecting
Also add an explicit comment to find_accepted_repository() in related.rs
clarifying that it intentionally only checks the database. Related events
should only be accepted after the repository announcement has been promoted
(validated via git data) - this is correct behaviour, not a missing check.
|
|
is_maintainer_in_any_announcement only queried the database, missing
announcements still in purgatory. A maintainer's announcement (which
lists the recursive maintainer) may arrive and enter purgatory before
the recursive maintainer's announcement does, causing the maintainer
exception check to return false and reject the recursive maintainer's
announcement.
|
|
Add comments explaining that PR event processing (both incoming and
purgatory) should only use database announcements, not purgatory ones.
This is intentional because:
- Incoming PR events should only be accepted for validated announcements
- Purgatory PR events should only be released when announcement is promoted
- This prevents accepting PR events for announcements that fail validation
Differs from state event processing which uses fetch_repository_data_with_purgatory
because state events check authorization without releasing from purgatory.
|
|
When processing state events from purgatory, we need to check
authorization against announcements that may still be in purgatory
(not yet promoted to the database).
Previously, process_purgatory_state_events() used fetch_repository_data()
which only queries the database. This caused authorization failures when:
1. Git data arrives
2. Announcement is promoted from purgatory to database
3. State events are processed from purgatory
4. But db_repo_data was fetched BEFORE the announcement promotion
Now uses fetch_repository_data_with_purgatory() to include both
database and purgatory announcements, ensuring authorization works
correctly regardless of promotion timing.
|
|
Purgatory announcements need state events (kind 30618) synced from
external relays, but not full L2/L3 events (patches, issues, PRs)
which would be rejected anyway. This implements the SyncLevel concept
from the design doc (decision #6):
- Add SyncLevel enum (Full vs StateOnly) to RepoSyncNeeds
- When announcement enters purgatory during sync, register in
RepoSyncIndex with SyncLevel::StateOnly
- Add build_sync_level_aware_filters() that partitions repos by level:
StateOnly repos only get state event filters (kind 30618)
- Update derive_relay_targets to track state_only_repos separately
- Update compute_actions to handle both repo sets
- SelfSubscriber always uses SyncLevel::Full (promoted repos)
|
|
The partial fix treating ProcessResult::Purgatory as confirmed in
pending_sync_index would trigger full L2/L3 sync for purgatory
announcements. Per design (decision #6), purgatory announcements
should only sync state events via SyncLevel::StateOnly (not yet
implemented).
Ignore test_archive_read_only_creates_bare_repo until SyncLevel
is implemented in Phase 3.
|
|
Route new announcements to purgatory instead of accepting immediately.
Announcements are promoted to the database when git data arrives,
ensuring we only serve announcements for repos with actual content.
Implemented:
- AnnouncementPurgatoryEntry type and DashMap store
- Route new announcements to purgatory (replacement announcements skip)
- Promote announcements on git data arrival (process_purgatory_announcements)
- Authorization checks purgatory announcements (fetch_repository_data_with_purgatory)
- State policy uses purgatory announcements for maintainer validation
- Cleanup task handles announcement expiry
- Updated count()/cleanup() to 3-tuples
Known broken:
- test_archive_read_only_creates_bare_repo fails: sync module does not
treat purgatory announcements as confirmed repos, so per-repo sync
(state events, PRs) is never triggered for purgatory announcements
- Announcement persistence (save/restore) not implemented
- SyncLevel (StateOnly vs Full) not implemented
- Soft expiry two-phase not implemented
- Expiry extension on state event / git auth not wired up
|
|
PR events, issues, and comments need a queryable repo announcement
to reference. Changed PREvent and PREventGenerated fixtures and
related tests to depend on ValidRepoServed instead of ValidRepoSent.
This ensures tests will fail correctly when announcement purgatory
is implemented - events tagging a repo should require that repo
to be served (not in purgatory).
|
|
- Remove redundant test_pr_event_remains_in_purgatory_until_git_data
- Rename test_pr_event_git_push_accepted -> test_pr_event_in_purgatory_git_push_accepted
- Add PASS/FAIL meaning to each test's documentation
- Note black-box testing limitation for purgatory detection
|
|
Add new fixtures for testing PR purgatory mechanism:
- PREvent2Generated: PR event with different commit hash
- PREvent2Sent: PR event sent to relay (enters purgatory)
- PREvent2GitDataPushed: Git data pushed after event sent
- PREvent2Served: Full fixture with event served
Add PRTestCommit2 variant for second PR test commit.
Update purgatory tests to use new fixtures for proper PR purgatory testing.
|
|
The CommitVariant::file_content() methods were returning strings without
trailing newlines, but the expected hash constants were calculated with
trailing newlines. This caused hash mismatches in tests.
Updated all hash constants to match the actual commit hashes produced
with trailing newlines in the file content.
|
|
- Derive Default for config structs instead of manual impl
- Fix doc comment formatting in ArchiveConfig::matches
- Collapse nested if statement in validate_announcement
- Allow too_many_arguments for SyncManager::new
|
|
Add PurgatoryTests module with tests for GRASP-01 purgatory behavior:
- Announcement purgatory tests (tolerant of unimplemented feature)
- State event purgatory tests (already implemented)
- PR purgatory tests (tolerant of unimplemented feature)
Tests pass regardless of purgatory implementation status, enabling
development without breaking the test suite. When features are
implemented, tests will verify correct purgatory behavior.
|
|
- Rename ValidRepo to ValidRepoSent (announcement sent, may be in purgatory)
- Add ValidRepoServed (announcement queryable after git data pushed)
- Add send_event_and_note_purgatory() for tolerant purgatory detection
- Update fixtures to use tolerant method instead of strict assertion
- Update event_acceptance_policy tests to use ValidRepoServed
This enables tests to pass regardless of purgatory implementation status
while still having explicit purgatory tests that verify the behavior.
|
|
Replace string-based spec references with typed SpecRef enum for
compile-time validation and better IDE support. TestResult::new() now
accepts SpecRef enum plus a requirement description string for
test-specific context.
|
|
|
|
The PR_TEST_COMMIT_HASH constant was incorrect because the discovery test
used a different git identity (pr-test@example.com) than the actual
create_pr_test_commit function (test@grasp-audit.local from fixtures.rs).
This caused the same commit content to produce different hashes due to
different author/committer info being embedded in the commit object.
Fixed by updating the discovery test to use the same git identity as
clone_repo() in fixtures.rs, ensuring consistent commit hashes.
|
|
now we have added announcement purgatory to the protocol spec
|
|
- Integrate sync-only-state-events decision (SyncLevel concept)
- Add authorization must check purgatory decision
- Add soft expiry design (delete repo, retain event for 24h)
- Add purgatory lifecycle diagram
- Create separate implementation details document
- Remove inline questions (now resolved)
|
|
|
|
Listen for both SIGINT (Ctrl+C) and SIGTERM (systemd) signals to ensure
graceful shutdown cleanup runs when stopping the service via systemd.
Previously, only SIGINT was handled, causing purgatory state and rejected
events cache to be lost on every systemd restart. Now both signals trigger
the cleanup code that saves state files and removes placeholder refs.
Fixes issue 0f73
|
|
|
|
Previously, some IO errors in git handlers were logged while others were
not, leading to inconsistent observability. Additionally, the HTTP layer
logged all git errors redundantly, adding no useful context beyond what
was already logged at the source.
Changes:
- Add error logging to all previously unlogged IO operations in
handle_upload_pack and handle_receive_pack (stdin writes, stdout/stderr
reads, process waits)
- Remove redundant error logging at HTTP layer since all errors are now
logged at their source with full context
- Ensures consistent error-level logging for all git subprocess failures
This provides complete observability of git operations while eliminating
duplicate log entries that don't add value.
|
|
Only the final summary 'Aligned repository with state' remains at INFO level,
showing the total count of refs_created/refs_updated/refs_deleted.
|
|
Improves observability when pushes are rejected due to state events that
only partially match the pushed refs. Previously, logs only showed 'No
state event found' even when state events existed but didn't match.
Changes:
- Add diagnose_state_mismatch() to explain why state events don't match
- Log specific reasons: missing refs, wrong SHAs, or extra refs
- Update rejection message to 'No matching state event found' (more accurate)
- Add 4 unit tests for diagnostic function
Example diagnostic output:
WARN State event abc123 from authorized author doesn't match push:
refs/heads/main missing (state declares 9cc3d93b)
This addresses the issue where a push with only refs/heads/test was
rejected because the state event also declared refs/heads/main, but
logs didn't explain why the match failed.
|
|
Fixes race condition where user's push becomes no-op after state event
is applied between fetch and push. Now accepts these as successful
no-ops, matching Git's 'Everything up-to-date' behavior.
- Add early detection in get_state_authorization_for_specific_owner_repo
- Return success for all-noop pushes without requiring purgatory event
- Document behavior in inline-authorization.md
|
|
This merge includes critical bug fixes and comprehensive migration tooling
developed during the relay.ngit.dev migration effort.
Bug Fixes:
- Fix git protocol error handling to return HTTP 200 with ERR pkt-line
- Fix naughty list false positives and DNS failure identification
- Fix database query filters in load_existing_events (remove .since())
- Fix OID fetch tracking to distinguish 0 OIDs from successful fetches
- Fix purgatory event source tracking for filtered expiry logging
- Implement OID retry logic for 'not our ref' errors
Migration Tools & Documentation:
- Complete 5-phase migration analysis pipeline with orchestration script
- Phase 1: Event fetching from source relay
- Phase 2: Git sync verification
- Phase 3: Categorization and relay comparison
- Phase 4: Log extraction (parse failures, purgatory expiry)
- Phase 5: Action classification for migration decisions
- Comprehensive migration guide with lessons learned
- Troubleshooting guide for permission and corruption issues
Configuration:
- Add NGIT_LOG_LEVEL configuration option
- Update git throttle limits to 60/minute
- Improve logging throughout for better observability
|
|
Move migration guide and scripts to docs/archive/2026-01-relay-ngit-dev-migration/
with clear warnings that these are reference-only materials from a specific
migration context, not general-purpose tools.
These materials document the relay.ngit.dev migration from ngit-relay to
ngit-grasp in January 2026. The scripts were developed iteratively during
the migration and are specific to that context. They are preserved for:
- Historical reference
- Context for production fixes in this branch
- Inspiration for future migrations (not direct reuse)
The migration uncovered critical bugs now fixed in this branch:
- Git protocol error handling
- Naughty list false positives
- Purgatory event tracking
- Sync startup issues
- Configuration management
|
|
Add EventSource enum (Direct/Sync) to purgatory entries to distinguish
between user-submitted events and sync-fetched events. This enables:
- WARN-level logging for direct submissions that expire (user should know)
- DEBUG-level logging for sync-fetched expirations (expected behavior)
- Source upgrade from Sync→Direct if user submits after sync
- Expiry timer reset on source upgrade (fresh 30-min window for user)
The source is included in [PURGATORY_EXPIRED] logs as source=direct or
source=sync for easy filtering.
|
|
|
|
Add git ancestry comparison (22-compare-git-data.sh) to determine
commit relationships between prod and archive repos. Repos where
archive is ahead are now correctly classified as ready-for-migration
since ngit-grasp only accepts git data authorized by state events.
Previously, repos with different git data were flagged as needs-resync
even when archive had newer/better data than prod.
|
|
Previously, sync_identifier_from_url passed all needed OIDs to
process_newly_available_git_data, not just the OIDs that were
successfully fetched. This caused incorrect logging (new_oids_count
would show all needed OIDs, not just fetched ones).
While this didn't break functionality (the actual processing uses
can_apply_state which checks the repository on disk), it made
debugging confusing.
Changes:
- Rename oids_fetched to fetched_oids and change type from usize to Vec<String>
- Return Vec<String> from match arms instead of counts
- Pass fetched_oids (not needed_oids) to process_newly_available_git_data
- Return fetched_oids.len() at the end
This ensures logging accurately reflects which OIDs were actually
fetched from the remote.
|
|
|
|
When fetch_oids returns Ok(vec![]) (all requested OIDs missing from
remote), the log message now says 'Fetch returned no OIDs (not available
on remote)' instead of the misleading 'Fetch succeeded' with oids_fetched=0.
|
|
Add retry loop in fetch_oids that handles git's behavior of stopping
at the first missing OID. When a 'not our ref' error occurs:
- Parse the missing OID from stderr
- Remove it from the fetch list and track it as missing
- Retry with remaining OIDs until success or all OIDs exhausted
This ensures we fetch all available OIDs even when some are missing
from the remote, rather than failing the entire batch.
Also improves error reporting:
- Include URL in all error messages for easier debugging
- Log stderr even when domain is already on naughty list
|
|
Previously, all git upload-pack/receive-pack failures returned HTTP 500,
but the git smart HTTP protocol requires protocol-level errors (like
"not our ref") to be returned as HTTP 200 OK with an ERR pkt-line in
the response body.
Changes:
- Add build_git_protocol_error_response() to create HTTP 200 responses
with properly formatted ERR pkt-line ("ERR <message>\n")
- Add is_git_protocol_error() to detect protocol errors (exit code 128
with stderr content) vs transport errors
- Update handle_upload_pack() and handle_receive_pack() to return
protocol errors as HTTP 200 with ERR pkt-line
- Keep HTTP 500 for actual transport errors (spawn failures, I/O errors,
signals)
This allows git clients to properly parse and display protocol error
messages instead of seeing generic HTTP 500 errors.
|
|
|
|
Change protocol error detection to only match WebSocket-specific errors
(websocket, invalid frame) instead of generic 'protocol' keyword which
was incorrectly catching transient git protocol errors.
Git protocol errors like 'fatal: protocol error: bad line length' are
transient network issues that should use backoff/retry, not permanent
naughty list blocking. Only WebSocket/Nostr protocol violations indicate
persistent infrastructure problems.
Fixes production false positive:
- relay.ngit.dev: git protocol error + remote warning misclassified
Add production test cases for git protocol errors and warning combinations.
|
|
|
|
|
|
Remove --analysis-root flag and external data file dependencies. The script
now extracts repo/npub information directly from 'Added rejected announcement'
log entries (which include pubkey and identifier fields) and uses
`nak encode npub <hex-pubkey>` to convert hex pubkeys to npub format.
This simplification was enabled by the recent logging improvement that added
pubkey to the 'Added rejected announcement' log entries.
|
|
Strip URLs (http://, https://, git://, ws://, wss://) from error messages
before classification to prevent false positives from repository names,
paths, or identifiers containing keywords like 'ssl', 'certificate', etc.
- Add strip_urls() function to remove URLs before pattern matching
- Add WebSocket protocol support (ws://, wss://) for relay errors
- Filter remote warnings that don't indicate infrastructure problems
- Use more specific SSL/TLS patterns to avoid npub substring matches
- Reduce test suite from 40 to 13 tests, keeping only edge cases
Fixes false positives seen in production:
- git.shakespeare.diy: 'repository not found' with npub containing 'ssl'
- relay.ngit.dev: HTTP 500 error with npub containing 'ssl'
- gitnostr.com: remote permission warning misclassified as protocol error
|
|
failures
|
|
load_existing_events()
Root cause: `last_connected` was set to Timestamp::now() BEFORE
load_existing_events() was called (line 425), causing the database
query to filter out all existing events with .since(current_time).
The query became: SELECT * FROM events WHERE created_at >= <now>
Result: 0 events returned (nothing has created_at in the future)
Solution: Remove .since() filter from database queries entirely.
The `last_connected` field is now only used for WebSocket subscription
filters to avoid re-fetching events from remote relays on reconnect.
Rationale for this approach over reordering operations:
- Database queries are fast (indexed by kind and created_at)
- Loading all events on startup ensures consistency
- Eliminates subtle ordering dependency that could break in refactoring
- Cleaner mental model: database = full load, WebSocket = incremental
This fixes the issue where ~190 state events weren't being fetched
after deploying the database query fix (commit 4162c90).
Evidence: Production logs showed "Loaded announcements from database
count=0" when there should have been hundreds of announcements.
|
|
|
|
Previously, SelfSubscriber only saw events returned by the WebSocket
subscription to the local relay, which has limits on the number of
events returned. This caused repos with announcements in the database
to never get Layer 2/3 filters created, resulting in missing state events.
Now, on startup, we query the database directly with two separate queries:
1. Query announcements (30617) to populate repo_sync_index
2. Query root events (1617/1618/1621) to create Layer 3 filters
Both queries use .since(last_connected) if available for incremental
loading on reconnect.
Filters are created inline and made mutable to support the .since()
clause, rather than using a shared create_event_filter() method.
Fixes the issue where state events were missing for repos like cashbird
and creative-space that had announcements in the database but weren't
returned by the WebSocket subscription.
|
|
Add proper log level configuration following standard approach:
- CLI flag: --log-level <level>
- Environment variable: NGIT_LOG_LEVEL
- Default: info
- Supports simple levels (error, warn, info, debug, trace)
- Supports filter expressions (e.g., ngit_grasp=debug,actix_web=info)
Configuration is now consistent across all four sources:
1. src/config.rs - Config struct with log_level field
2. docs/reference/configuration.md - Full documentation
3. nix/module.nix - NixOS module with logLevel option
4. .env.example - Example configuration file
This replaces the previous RUST_LOG approach with proper integration
into the ngit-grasp configuration system, enabling trace logging from
CLI, environment variables, or NixOS configuration.
|
|
|
|
|
|
The new script implements the redesigned classification system with:
- Tier 1: No Action Required (complete in both, deleted, empty, archive-only)
- Tier 2: Action Required (complete in prod but missing/incomplete in archive)
- Tier 3: Manual Investigation (partial/no-match in prod, archive-only anomalies)
Produces cleaner output format with actionable categories and reasons.
|
|
caught a production bug where npub in url string contained "dns"
triggering false positive
|
|
Implements the redesigned migration analysis classification system:
Tier 1 - Ready for Migration (no action required):
- Complete in both prod and archive
- Deleted by user (kind 5 event)
- Empty in prod (cat2) - always no action, regardless of archive
- Archive-only (not in prod)
- Purgatory-only (not in prod)
Tier 2 - Needs Re-sync (action required):
- Complete in prod, missing/incomplete in archive
- Includes purgatory context (expired vs never-tried)
Tier 3 - Manual Review (investigation needed):
- Partial in prod (cat3)
- No-match in prod (cat4)
- Parse failures with complete prod
Key fixes:
- Use safe arithmetic ($((x + 1))) instead of ((x++)) with set -e
- Batch nak hex-to-npub conversions for deletion processing
- Handle NDJSON format for deletion files
Output: 352 ready, 295 resync, 46 review (693 total)
|
|
Phase 4 (30-extract-parse-failures.sh) now enriches parse failures with
repo name and npub by looking up event_id in announcements.json. This is
critical because 'Invalid announcement' rejections only log event_id and
kind, not the repo name or npub.
Phase 5 (40-classify-actions.sh) was also fixed to extract columns 4 and 5
(repo|npub) instead of columns 1 and 2 (event_id|kind) from parse-failures.txt.
Without this fix, action-required.txt showed unusable output like:
000014b2... | 30617 | parse failure logged | fix event format...
Now it correctly shows:
scripts | npub1hs5244... | parse failure logged | fix event format...
The enrichment uses jq to build a lookup table from announcements.json and
optionally uses 'nak' to convert hex pubkeys to npub format.
|
|
Filter parse failures to only those for announcements that are in
production but missing from the archive. This eliminates noise from
rejections of events from other relays that don't affect migration.
Before: 223 parse failures (all rejections from all relays)
After: 18 parse failures (only for missing announcements)
The filter works by:
1. Reading missing announcements from comparison data
2. Extracting event IDs from production announcements JSON
3. Filtering parse failures to only matching event IDs
|
|
The script was counting the same invalid announcement twice because:
- Write policy logs use hex event IDs
- Builder logs use note1 (bech32) event IDs
- Deduplication only worked within each format
Fix: Only extract from write policy logs (hex IDs) to avoid the
format mismatch. Builder logs contain the same events, so we don't
lose any data.
Result: 446 entries → 223 unique invalid announcements (correct count)
|
|
Update parse failures script to also extract 'Invalid announcement'
rejections from logs. These are announcement events that failed
validation (e.g., multiple clone tags instead of single tag with
multiple values).
Changes:
- Search for 'Event rejected by write policy' pattern with 'Invalid announcement'
- Search for 'Rejected repository announcement' pattern from builder
- Extract event_id, kind, and reason from rejection logs
- Combine with [PARSE_FAIL] entries in output
- Deduplicate entries by event_id
- Update header to clarify both patterns are captured
- Update migration guide to document this
- Fix SIGPIPE handling in purgatory script (minor)
This captures the ~446 unique announcements rejected for NIP-34 format
violations (multiple clone tags), which were previously unexplained
in the migration analysis.
|
|
Make scripts fully automatic with no manual intervention needed.
Changes:
- Add --no-pager to journalctl commands in validate-service.sh
- Add service existence validation with helpful error messages
- Capture and report journalctl stderr for better error visibility
- Improve error handling without failing on empty logs
The main issue was missing --no-pager in validate-service.sh which
could cause scripts to hang when run non-interactively (e.g., via SSH).
Tested locally - scripts run without hanging and produce correct output.
|
|
Add validation to ensure Phase 4 scripts use ngit-grasp service
(with structured logging) instead of ngit-relay service.
Changes:
- Add validate-service.sh helper for reusable service validation
- Add validation to run-migration-analysis.sh before Phase 4
- Add validation to 30-extract-parse-failures.sh
- Add validation to 31-extract-purgatory-expiry.sh
- Update migration guide with clear warnings about service selection
- Expand troubleshooting for 'Phase 4 finds no logs' issue
- Emphasize lesson learned in relay.ngit.dev notes
This prevents the issue where Phase 4 was run against ngit-relay.service
and found no parse failures because structured logging only exists in
ngit-grasp services.
|
|
- Add Gotchas section with common issues: git installation, localhost-only
archive relays, non-standard git paths, service name variations, and
permission requirements
- Add relay.ngit.dev-specific migration notes with actual paths, service
names, and analysis results (315 repos need re-sync, 382 purgatory expired)
- Enhance Running the Analysis section with path discovery guidance
- Expand Troubleshooting section with solutions for git not found, archive
connection failures, and wrong git paths
- Add git --version to prerequisite checks
- Update examples to use realistic localhost archive URLs
|
|
- 10-check-git-sync.sh: Check for git before running
- run-migration-analysis.sh: Include git in prerequisite checks
- Fixes script failures when git is not installed
|
|
- Rename guide: migrate-ngit-relay-to-ngit-grasp.md → migrate-to-ngit-grasp.md
- Remove ngit-relay and relay.ngit.dev specific references
- Use generic terminology: source/target relay, current implementation
- Add Compatibility section explaining requirements
- Update examples to be implementation-agnostic
- Update script comments to reference GRASP relay (not ngit-relay)
- Update README.md to link to the new guide
Scripts already work with any GRASP implementation via parameters.
|