Testing¶

Running tests¶

# Install dev dependencies first (once)
uv sync --group dev

# Run all unit tests
uv run pytest

# Run integration tests (requires live services — see below)
uv run pytest -m integration

# Run unit tests only, explicitly skipping integration tests
uv run pytest -m "not integration"

# Run a single test file
uv run pytest tests/test_store.py

# Run a single test by name
uv run pytest tests/test_store.py::test_add_paper_is_idempotent

Continuous integration¶

.github/workflows/tests.yml runs the unit suite on GitHub Actions — uv run pytest -m "not integration" on ubuntu-latest with Python 3.12, exactly the command you'd run locally.

It fires on pushes to main and on pull requests targeting main, and nothing else. Feature branches stay unwatched until they open a PR at main, which matches the git workflow here: work happens on a local branch, and main is the thing worth guarding.

Two caches keep a run to a couple of minutes:

uv's dependency cache, keyed off uv.lock, so the torch download that comes with sentence-transformers happens once per lockfile change.
~/.cache/huggingface, holding the embedding model and the reranker (~220 MB together). The cache key names both models, so swapping either one in the config means changing the key to match, otherwise CI keeps restoring the old pair and re-downloading the new one every run.

Integration tests never run in CI — they need an Anthropic API key and a live Ollama server, and -m "not integration" deselects them. Nothing in the unit suite reaches the network or needs a secret, so the workflow declares none.

Test infrastructure¶

Dedicated ChromaDB store¶

KB tests run against a real ChromaDB instance at tests/.chroma/ (gitignored, never committed). They open it directly rather than through kb server, so the whole suite runs with nothing started — the store fixture is passed explicitly to every KB function, which is also what keeps tests isolated from each other. Each test creates its own test_<uuid8> collection there and deletes it at teardown, which gets us three things for free: tests are fully isolated from each other, the store directory itself persists between runs without needing to be rebuilt, and the embedding model doesn't reload for every single test.

Real HuggingFace models¶

The KB tests use the real embedding model from the default config (BAAI/bge-small-en-v1.5) rather than a mock or a deterministic stub, built through the same build_embeddings() helper production uses, so the query prefix and normalization actually match real retrieval. Any test that calls search() loads the real cross-encoder reranker too (cross-encoder/ms-marco-MiniLM-L6-v2). Both models download once, to ~/.cache/huggingface/ (embedding ~133 MB, reranker ~90 MB), then come from local cache after that.

Why not mock them? A fake embedding function that returns zeros or random vectors would hide the failure modes that actually matter — ChromaDB filter behavior, the LangChain wrapper's batching logic, and search ranking all depend on real vector values, and you can't observe what the reranker actually does without running it. The one-off download is a cheap price for that fidelity, and it's the general policy here: pay a modest one-time setup cost rather than test against a mock that hides how the thing really behaves.

Real vs mocked, by boundary¶

Real: embeddings, the reranker, ChromaDB, PyMuPDF, and pymupdf4llm all run for real. PDF fixtures — annotated pages, scanned-like pages — are generated in-test with PyMuPDF, so no binary fixtures are committed and the real extraction/conversion paths run end to end.
Mocked, and deliberately so: the arxiv client's network calls (live HTTP to export.arxiv.org — _client.results is stubbed, using real arxiv.Result objects), the bioRxiv API (requests.get stubbed with fake responses shaped like the real payload), and the LLM API clients, mocked at the Ollama/Anthropic client boundary since they're billed per token or need a running model server. These are genuine system boundaries: what's under test at each is our retry and tool-loop logic, not the library's.

Retrieval-quality benchmark¶

test_retrieval_quality.py measures how good retrieval is, not just whether it runs. It seeds a small module-scoped corpus once — paper summaries and Markdown notes — and runs a fixed golden set of about 24 queries through the real search() pipeline (hybrid BM25+RRF, the default), checking that hit-rate@5 and MRR@5 stay above a floor. It does two jobs at once: catching regressions (a broken query prefix, chunker, or reranker drops the metrics well below floor) and giving you a place to actually watch retrieval accuracy while tuning. The corpus includes acronym and proper-noun queries (LoRA, BERT, Dr. Tanaka) plus author-name queries ("papers by Vaswani") that only match because of the embed_header prepended to every paper chunk.

The thresholds are set as a regression floor with margin, not at the current ceiling. On a corpus this small and topically distinct, the pipeline scores near-perfectly — tighten the thresholds only as far as stays stable across query rewording.

Integration tests¶

Tests marked @pytest.mark.integration need live external services:

Test	Requirement
`test_anthropic_client_initialises`	API key in `ANTHROPIC_API_KEY` env var or `~/.jarvis/config.toml [auth]`
`test_anthropic_models_list_confirms_auth`	API key + internet access
`test_ollama_is_reachable`	Running Ollama server (checked via `ollama.list()` at `http://localhost:11434`)

None of these spend a token — they only check that credentials and connectivity work.

What is tested¶

File	Module	Behaviours covered
`test_transcript.py`	`jarvis/core/transcript.py`	Round-trip identity for each adapter (neutral → wire → neutral). The Anthropic tool-result bundling rule and the OpenAI one-message-per-result rule. Ollama's name-keyed results and synthesised call ids. JSON-string vs mapping arguments, including malformed JSON. Opaque blocks replaying to the same provider+model and being dropped for any other model, provider, or wire format
`test_frontmatter.py`	`jarvis/kb/frontmatter.py`	Well-known keys mapping to named fields and the frontmatter block being stripped from the body. Unknown keys passing through as `x_<key>` (string, int, bool). The shadowing defence — `visibility`/`doc_type`/`source`/`file_path` in a note's own frontmatter cannot overwrite jarvis's schema. Tags joined to `"\|a\|b\|c\|"` with exact membership testing (`remote` must not match `remote-first`). Malformed YAML warning but keeping the note; nested values skipped with a warning; a note with no frontmatter left untouched; a `---` rule partway down a file not mistaken for a header. `record_header` for full, partial, and empty records
`test_drafts.py`	`jarvis/drafts/workspace.py`	The containment policy: plain names accepted; paths, traversal, hidden names, disallowed extensions and invalid ids refused; a symlinked draft folder pointing outside the root refused, which is why containment is checked against the root and not the draft folder; a symlinked file escaping a draft refused. Creation writing file + metadata and inheriting session visibility; `add_draft_file` free only for a new name; the size limit. `propose_edit` writing nothing. Hunk arithmetic — all hunks equal the proposed text, none equal unchanged, either one of two separated hunks applies exactly, insertions and deletions both work. Hunk `kind` taken from the opcodes rather than line counts, since a group carries context on both sides and neither side is ever empty. `old_spans`/`new_spans` covering only the lines that change — the regression behind "it says 3 blocks but highlights 4," checked with a surviving block that must not be marked, and spans asserted to be offsets into the hunk's own lines rather than into the file. Stale-proposal refusal after a hand-edit, one-shot tokens, unknown tokens. `.versions/` snapshot before every write, restore snapshotting first so it's itself undoable, a version from another file refused, save refused on a moved hash. Retention: stale removed, activity in any file keeping the whole draft alive, `keep` exempt, `retention_days = 0` disabling, dry-run deleting nothing, a symlinked draft folder never followed by the prune. And the reason a draft is a folder in the first place — `compile_latex` seeds its working directory from the whole draft, so a `.tex` reaches its `.bib` and chapters and nothing from any other draft, with `draft.json` excluded. Deleting a draft leaves a copy the user made elsewhere untouched
`test_webapp_editor.py`	`jarvis/webapp/app.py` (editor routes)	The human side of the sandbox via `TestClient`: `/drafts` listing with toolchain availability; save writing through and refusing a stale hash; every route refusing paths outside the sandbox. `/preview` escaping embedded HTML, tagging each block with the source line the scroll sync aims at, and the sandbox on the frame it lands in: the markup shipping `sandbox=""`, the script never granting `allow-scripts` — asserted by enumerating every value it puts on the attribute, literals included, and that nothing sets it by the property instead — and the compile path restoring `sandbox=""` before loading a PDF, so the same-origin flag the Markdown path needs for scroll sync cannot be inherited by an active content format compiled from a `.tex`. Compilation invoked with `-no-shell-escape`, `openin_any=p`/`openout_any=p`, an emptied `TEXMFHOME`, the configured timeout, and a working directory that is never the draft folder — asserted on the invocation, so the security properties are verified even with no TeX installed — plus a failed compile returning its log, a missing toolchain degrading to a clear 400, and a non-`.tex` refused. `/apply-edit` applying only the accepted hunks, one-shot tokens, unknown tokens, and a stale proposal refused after a hand-edit; `/discard-edit` leaving the file alone. `/proposals` listing what's still waiting in the same shape the live SSE event carries (so a re-opened suggestion renders through one code path) while withholding the whole proposed file, and `/proposals/discard-all` clearing them without touching a file. `/reveal` invoking a fixed argv with no shell, aimed at the file or its folder and always inside the sandbox, refusing every path outside it before any subprocess runs, and 404ing on a missing file. `_file_manager_command` is tested as a pure mapping across macOS/Windows/Linux/unknown from any host — living inline in the route meant it could only be exercised on whichever OS ran the suite, which is how the macOS shape got asserted as universal and only failed once CI ran on Linux. `/drafts/new` with a `draft_id` adding to an existing document rather than starting a second one, still containment-checked and still refusing an existing filename. The compile directory excluding `draft.json`. Restore putting a version back while snapshotting the current one
`test_config.py`	`jarvis/core/config.py`	Defaults when no TOML (incl. `digest_enabled` false, `figure_captions` false, `digest_hour` 5, `pdf_watch_minutes` 30); `[digest] enabled` parsed both ways; TOML overrides defaults; env vars override TOML; `~` in paths expanded; `[auth] api_key` loaded; `[chat] ollama_model` + `OLLAMA_MODEL` env; `[rag]` retrieval and figure-caption keys, including `hybrid` (defaults true, parses from TOML); `[digest]` bioRxiv keys; `[sync]` section defaults, overrides (incl. `pdf_watch_minutes`), and `PDF_WATCH_DIR` env override. `describe()` never renders an API key, says whether each is set, redacts a key that has leaked into some other field, and reports resolved values rather than what the file says (an env var winning over the file is the usual confusion). Plus two tests that keep the README honest: its OpenRouter config block actually parses into a working config, and every `uv run <command>` it names is a real entry point
`test_errors.py`	`jarvis/core/errors.py`	`@with_retries`: success on first try; retry on matching exception; raise after max attempts; no retry on non-matching exception; backoff grows exponentially with jitter (sleep monkeypatched)
`test_arxiv_convert.py`	`jarvis/digest/arxiv/convert.py`	`parse_arxiv_url()`: `/abs/` URL; `/pdf/` URL; version suffix preserved; non-arXiv URL returns None
`test_arxiv_fetch.py`	`jarvis/digest/arxiv/fetch.py`	`_to_paper` field mapping including `doi` (empty by default, surfaced when the arXiv result has one); fetch success; empty-feed-with-200 treated as transient (retried, then raises `FetchError`); recovery when the empty feed is transient; library errors wrapped as `FetchError`; single-paper fetch by ID; deduplication. The arxiv client's network calls are stubbed (real `arxiv.Result` objects, no HTTP); retries run with `time.sleep` monkeypatched
`test_biorxiv_fetch.py`	`jarvis/digest/biorxiv/fetch.py`	Record→paper mapping (doi→link, `bioRxiv:{category}` source); cursor pagination across pages and `max_results` cap; empty first page retried then raises `FetchError`; keyword matching (title/abstract, case-insensitive, non-match excluded, first-keyword source tag); DOI dedup across two matched keywords. `requests.get` stubbed with fake responses shaped like the real API; `time.sleep` monkeypatched
`test_images.py`	`jarvis/kb/images.py`, `jarvis/kb/store.py`	`extract_figures` keeps large images with their 1-indexed page and drops sub-`min_pixels` decoys; `max_figures` cap. `add_figures` (via the `enabled=True` per-document opt-in, since captions now default off) indexes `[FIGURE p.N]` caption chunks (`annotation_kind="figure"`) with a fake provider; private+anthropic skip writes nothing and never calls the model even with `enabled=True`; per-figure failure is tolerated; the config kill-switch disables it; delete-by-source sweeps figure chunks
`test_pdf_convert.py`	`jarvis/kb/convert.py`	Real pymupdf4llm conversion of PDFs generated in-test: text extracted; string returned with no intermediate files; scanned/image-only PDF raises `ConversionError`
`test_annotations.py`	`jarvis/kb/annotations.py`	Extraction from PDFs annotated in-test with PyMuPDF (the same annotation objects Preview/Foxit write — no binary fixtures): highlight text recovery; typed note on a highlight; underline treated as highlight; sticky note → comment; unannotated PDF → `[]`; Ink drawing ignored; multi-line highlight reading order
`test_store.py`	`jarvis/kb/store.py`	`add_texts` count; `add_paper` idempotency and authors line in content; title-based dedup (`_title_exists` normalisation, add skipped on same title from a different source, `allow_duplicate` forces the add); `add_papers_batch` returns `(added, skipped)`; visibility filter; privacy check (cloud and local); `delete_by_metadata`; `list_documents` deduplication and chunk count; `update_file_path` metadata and URI; `update_file_path` unknown source; `get_document_chunks` orders body chunks (by `chunk_index`) before annotation/figure chunks, and returns `[]` for an unknown source; `update_visibility` metadata-only; `add_annotations` indexing (and no-op on unannotated PDFs); delete-by-source sweeps body and annotation chunks together; `annotation_kind` search filter; `refresh_vault` add / update / delete / visibility re-check when config reclassifies a dir; embedding-model guard (mismatch / match / empty); re-ranking preserves visibility filter and `rerank=False` skips the reranker; chunk_index / section breadcrumb metadata; `KBCorruptionError` raised on an "Error finding id" failure (names `kb reindex`) and on the stale-collection-handle failure after a reindex swap ("Collection [...] does not exist" — names restarting the process, on both the search and add_texts paths), other failures stay a plain `RAGError` — and specifically a "connection reset by peer" is not reported as a stopped server, the regression that a looser substring match would cause; who owns the index — a long-lived process refuses to open the files when the server is down (the fault the server exists to remove), a one-shot command falls back to them, `rag_dir` ends up `0700` even when it started `0755`, and telemetry is off (HuggingFace and LangSmith environment variables set by importing `jarvis`, Chroma's own switch off in the settings the client is built with); `embed_header` prepended to every chunk; hybrid on/off routing (`_hybrid_search` called only when `cfg.hybrid`, byte-identical `similarity_search` path when disabled); `_reciprocal_rank_fusion` unit test; `update_paper_metadata` sets only the fields the caller passes; `doc_type` list filter (`["paper", "digest"]` returns both types, a plain string narrows to one); `add_figures` enabled matrix (`enabled=True` captions despite config off, `enabled=None` follows config); `find_pdf_notes` ignores vault `.md` notes and groups a legacy PDF note's chunks by source with title/visibility/chunk_count; `reclassify_notes_as_papers` flips only `doc_type` (content_hash/storage_mode/file_path/visibility untouched) and is a no-op on an empty source list. Records: a note's frontmatter lands on every chunk as flat metadata while the block itself is stripped from the indexed text and `doc_type` stays `"note"`; `category`/`status`/`entity` filters narrow correctly and a custom `x_` field filters without any code change; a tag filter requires every tag; the embedded record header makes "rejected job application" match a record whose body never says it; `refresh_vault` re-indexes a note whose `meta_schema` is behind (backfill with no migration script) and leaves current-schema notes alone across repeat sweeps — including a note with no frontmatter, which would otherwise re-index forever; `metadata_key_counts`/`metadata_value_counts` expose a typo'd frontmatter key as its own `x_` key rather than silently ignoring it; staged writes — `build_text_documents` produces chunks without changing `count()`, `commit_documents(replace_source=...)` deletes the old entry and adds the new one so only the replacement survives, and an empty commit with `replace_source` set still supersedes the old entry rather than leaving stale chunks
`test_metadata.py`	`jarvis/kb/metadata.py`	`infer_pdf_metadata`: DOI found by regex skips the LLM DOI guess, DOI guessed by the LLM when the regex misses, LLM failure degrades gracefully (title/authors absent, doi `""`); `resolve_pdf_metadata`: explicit overrides skip inference entirely, inference always runs for a local PDF even under Anthropic (papers are always public, so there is no private-note guard to apply here)
`test_kb_cli.py`	`jarvis/kb/cli.py`	`kb doctor`: healthy store reports success at each stage; a corrupted index (`KBCorruptionError`) exits non-zero with the diagnosis; a store that fails to open exits non-zero; an empty store skips the search probe entirely; `_check_legacy_pdf_notes` migration: a public legacy PDF note is reclassified to `doc_type="paper"` on a `y` answer, left as `"note"` on `n`, a private one is only listed (resolution options, `kb remove` mentioned) with the reclassify prompt never even shown, and an empty store prints nothing. The knowledge-base server: `kb doctor` says which route it reached the index by (server up vs "NOT running", naming `uv run kb server`), and `kb reindex --from-storage` refuses while the server holds the index open rather than racing it on `chroma.sqlite3`
`test_retrieval_quality.py`	`jarvis/kb/store.py`	Golden-set retrieval benchmark — hit-rate@5 and MRR@5 over a seeded corpus of paper summaries and markdown notes, run with hybrid retrieval (the default); includes author-name queries answered only by the embed_header
`test_privacy_guard.py`	`jarvis/chat/chat.py`	The chat-layer privacy guards: `read_file` vault containment, private-dir hard stop for cloud only, symlink-into-private-dir regression, path-escape rejection; `_search_notes` excluded-results caveat, private-only hard stop, no caveat for the local provider; `_get_document` — a public document is fine under Anthropic, a private source raises `PrivacyError` under Anthropic with no content leaked in the message, and the local provider reads it and reports `saw_private=True`. A parametrised matrix then re-runs the three hard stops (`read_file`, `_search_notes`, `_get_document`) across every cloud shape — `anthropic`, an `openrouter:<model>` spec, and a provider name this build has never heard of, which must fail closed — plus a check that a local Ollama spec whose model tag contains a colon (`ollama:qwen3-vl:30b`) is still classified local and can read private notes. Record filters cannot widen what a cloud provider sees: a cloud query naming a private record's exact category and status still hard-stops, and a mixed public/private result set returns only the public record with the "excluded" caveat
`test_daemon.py`	`jarvis/sync/daemon.py`	Pure decision functions plus real ingestion with the store fixture: `_build_scheduler` builds cleanly with digest/digest_catchup/vault_refresh jobs when the digest is enabled, omits both digest jobs (keeping vault_refresh) when it is not, adds pdf_scan only when a watch dir is set, and has a real (non-string) timezone — the regression for the `timezone="local"` crash-loop; `digest_is_overdue` (first start, within the week, missed slot, boundary); `run_digest_catchup_job` matrix (fresh success → not fired, 8-day-stale → fired, missing baseline → not fired, `run_digest_job` monkeypatched) plus a same-path two-call regression proving the job re-reads the status file each call rather than caching (stale fires, then a freshly written success on the same path does not); the digest double-fire lock (`run_digest_job` returns early without touching config when the lock is held) and a companion test proving the lock is released on the success path too (pipeline `main` stubbed, called once per `run_digest_job` call across two calls); `wait_for_stable` (settled / growing / vanished files); `ingest_pdf` add → skip (unchanged hash) → update (changed bytes), with a stub metadata-inference provider; `ingest_pdf` populates an inferred title/authors; `ingest_pdf` logs the stored title/authors/doi and source filename after a successful add; `_log_next_run_times` logs one "next run at" line per scheduled job; `_log_job_outcome` logs a job's next run time after both a success and an error event; `scan_watch_dir` listing and artifact skipping; `run_pdf_scan_job` (ingest + skip with status recorded to a tmp status file, per-file failure recorded without raising, unstable file left for the next cycle, no-op without a watch dir); `_validate_sync_config` (incl. `pdf_watch_minutes` ≥ 1, and ignoring a stale `digest_day`/`digest_hour` while the digest is disabled); status file round-trip, job recording, and corrupt-file handling
`test_pipeline_run.py`	`jarvis/digest/pipeline/run.py`	The digest→KB indexing tiers, with the store fixture and `download_arxiv_pdf` mocked to write a real tiny PDF (conversion/chunking/embedding run for real): score ≥ 9 arXiv paper indexed full text with score/track/`storage_mode="full_text"` and the title/authors embed header; bioRxiv doi.org link falls back to a summary entry; a download failure (404) warns and falls back; an already-indexed paper is skipped before any download; `index_scored_papers` routes 9 → full text, 8.5 → summary via `add_papers_batch`, 7 → not indexed, plus an exact-boundary case (score 9.0 → full text, score 8.0 → summary); `index_digest_file` stores `doc_type="digest"` with a `file://` source, dated title, and `storage_mode="full_text"`; the opt-in gate in `main()` (disabled → prints how to enable and fetches nothing, `--force` → runs anyway, enabled → runs with no flag). A provider stub raises on any LLM call, proving the whole tier path is LLM-free
`test_reingest_replace.py`	`jarvis/kb/cli.py`, `jarvis/chat/chat.py`	Regression coverage for the replace-on-duplicate reingest flow: a same-source duplicate re-add whose download/conversion then fails leaves the old entry's chunks (including a preseeded marker standing in for irreplaceable annotations) completely untouched, for both `jarvis.chat.chat._add_document` (arXiv full-text, `download_arxiv_pdf` stubbed to raise) and `jarvis.kb.cli.cmd_add` (local-PDF full-text, `pdf_to_markdown` stubbed to raise); a successful reingest on both paths deletes the old chunks exactly once and leaves only the new content, with a single entry for the source; a same-title-but-different-source duplicate with `allow_duplicate=true` never deletes the other entry
`test_sessions.py`	`jarvis/chat/sessions.py`	Save/load round-trip; pydantic message normalisation; empty sessions never written; malicious session-id rejection; pruning keeps newest unpinned and all pinned; sidebar ordering; `mark_private` flag + re-index; `check_resume` refuses a private session on any cloud provider (anthropic and openrouter alike) while allowing it locally, and now allows cross-provider resumes since the neutral transcript removed the reason for the old block; `new_session` splits a `provider:model` spec and resolves the provider's default model when none is named; v1→v2 migration of both an Anthropic wire transcript (tool_use/tool_result blocks convert, `turn_starts` dropped) and a thinking block (preserved but tagged with an unknown model so it can never replay), with an already-v2 session left untouched; `record_usage` accumulates per model, ignores a provider that reports nothing, and survives a save/load round-trip; a router's spend is attributed to the models that actually answered rather than to the router's own name, `served_model` records the most recent pick, stays empty when the answer came from the model that was asked for, survives a save/load so the header is still right after a refresh, and is cleared both by switching model and by the next direct answer — the reported bug was the `→ claude-sonnet-4.6` arrow surviving a switch away from `openrouter/auto`; chat-history search respects session privacy; delete removes file and chunks; rename round-trip, empty/whitespace rejected, 120-char cap, unknown id, and `update_chat_title` propagates to indexed chat chunks; compaction no-op below threshold, replaces old turns with a summary (fake provider), display untouched; token estimation; `needs_compaction` agrees with `maybe_compact`'s own decision on both sides of the threshold (and goes false once compacted); `rollback_turn` restores `messages`/`display`/`turn_starts` together, leaving earlier completed turns intact, and truncates back to the turn boundary so anything a PrivacyError turn had already committed goes too
`test_prompts.py`	`jarvis/core/prompts.py`, `jarvis/webapp/app.py` (prompt routes)	First use seeds every copy from the shipped default and is idempotent; seeding never overwrites an edit, which is the property that makes it safe to run on every startup; editing writes the copy and leaves the repo default untouched; Revert restores it; a legacy `~/.jarvis/system_prompt.md` is carried across rather than overwritten; an unknown or traversal-shaped name is refused by both the API and the routes without creating anything outside the prompts dir. The shipped defaults keep every placeholder the code substitutes — `{abstracts_text}` above all, without which the scoring prompt is sent with no papers in it — and the shipped scoring default is asserted to name nobody's speciality, since it used to ship with one researcher's topics in it. Then the part that matters: an edited system prompt reaches the next `build_system_prompt()` and an edited summary prompt reaches the next `_get_summary_prompt()`, so an edit applies to the next message rather than after a restart. Plus a tool-surface assertion that no chat tool can edit a prompt
`test_settings.py`	`jarvis/core/config.py` + `jarvis/chat/chat.py`	Response style lands in the system prompt (and absent when empty); `set_config_value` tomlkit round-trip preserving comments and other keys; creates missing file/section; `reset_config` reloads the singleton
`test_security.py`	`jarvis/kb/store.py`, `jarvis/chat/chat.py`, `jarvis/webapp/app.py`, `jarvis/core/config.py`	File deletion has been removed wholesale: `jarvis.kb.store.delete_local_file` no longer exists, and no `.unlink(` call survives anywhere in `jarvis/kb/store.py`, `jarvis/kb/cli.py`, or `jarvis/chat/chat.py`; `_remove_document` is a one-shot flow — no channel refuses, human decline blocks, human approval executes and never touches disk, a deferred (webapp) channel leaves a pending action that executes correctly later, and the "files on disk are never touched by jarvis" invariant line (with the full local path) appears in every preview and confirmation description; `truncate_middle` preserves head+tail so a `file:///` filename stays visible; webapp rejects foreign Host headers (TrustedHost); session-id traversal rejected; `pending_actions` dict (now `{token: {session_id, action}}`): an unknown/cleared token 409s without disturbing other entries, the matching token pops just its own entry and executes, two stacked tokens across different sessions are each independently confirmable with no session check, cancelling one pops only its own token, `/sessions/new` leaves every other session's dialogs pending (they still confirm normally), a resume clears only the resumed session's own tokens (a different session's survive), and starting a new `/chat` turn on one session clears only that session's tokens. API-key redaction: a provider error quoting the key is scrubbed before it becomes an `LLMError` (the leak that motivated this — the text goes to the screen, chat.log, and the saved session, which is indexed as a chat chunk); an Anthropic key supplied only as an env var is still caught, asserted against the premise that `load_config` does not fold it into Config the way it does OpenRouter's; ordinary text is untouched; a suspiciously short key is ignored rather than blanking whole messages; a failure inside redaction returns the text unchanged rather than replacing one error with another; and the config view never serves a key
`test_chat_logging.py`	`jarvis/chat/chat.py`	A tool wrapper that raises logs the exception (with traceback) to the `jarvis.chat` logger before returning its short error string, across more than one tool wrapper; `KBCorruptionError` is relayed verbatim (prefixed `[KNOWLEDGE BASE ERROR`) by `_retrieve_papers`, `_search_notes`, and `_search_chat_history`, with logging still firing first. An `isolated_log` fixture detaches the module's real `FileHandler` for the test so the run never appends to the user's actual `~/.jarvis/logs/chat.log`
`test_chat_tools.py`	`jarvis/chat/chat.py`	Chunk-first retrieval: `_retrieve_papers`/`_search_notes` return text past the old 300-char cutoff (a >300-char chunk asserted present verbatim, no `...` elision) and a `Section:` breadcrumb line when the chunk carries one; `_get_document` pagination over 22 indexed chunks (page 1 of 2 shows chunks 1-15 with the "Call get_document(...) for more" hint, page 2 of 2 shows chunks 16-22 with no further-page hint), an unknown source returns a `[No document found...]` string, and a `storage_mode="summary"` document gets the full-text-not-in-KB honesty note; `_dispatch_tool("get_document", ...)` wraps output in the `RETRIEVED DATA` markers and flags a fresh session private on a local-provider private hit, mirroring `read_file`/`retrieve_papers`/`search_notes`; the staged ingest — `_add_document` stopped during `summarize()` raises `TurnCancelled` (rather than swallowing it into a tool-error string) and leaves the real store's `count()` unchanged with no chunks under the paper's source, while an uncancelled add still indexes the paper in its single commit
`test_webapp_chat.py`	`jarvis/webapp/app.py`	The `/chat` turn lifecycle via `TestClient` with a fake provider (`agentic_turn` fully controlled, no live LLM): the early `save_session(session)` call lands before the LLM call with only the user's turn in `display` and no `store=`, the final save is indexed (`store=`) and includes the assistant turn; the busy guard (a session id in the `running` registry) makes a second `/chat` addressed at that same session 409, `DELETE` on it 409, and `GET /sessions` reports its id in the `busy` list; an uncaught exception in `agentic_turn` still logs an `ERROR` with a traceback, still yields a `reply` SSE event ("Internal error"), still saves the error turn, and the stream still terminates (`running` empties); an `LLMError` yields a `⚠️` reply, logs, and still saves; resuming a still-mid-turn session installs the exact live registry object the background thread is mutating (not a stale disk copy) and reports `busy: true`, and `GET /history` serves the finished reply once the turn is released — no reinstall step needed anywhere; two different sessions can be mid-turn at once (one blocked, one completing fully in between) with each session's own `display` holding only its own exchange and no cross-contamination; an unknown `session_id` 404s; a message addressed to a non-active session lands on that session and never touches whatever `_session["session"]` happens to be at that instant. A request FastAPI rejects at validation time (stale-tab `/chat` without `session_id`) returns the standard 422 detail list AND logs a `request validation failed` line to the `jarvis.chat` logger — 422s are no longer invisible in chat.log. Reuses the `isolated_log` fixture pattern from `test_chat_logging.py` so no test touches the real `~/.jarvis/logs/chat.log`. Model switching: `GET /models` returns the catalogue and makes no network call (a patched `requests.get` fails the test if it does), `POST /model` applies a switch, 409s for a private session moving to a cloud model and for a session mid-turn, and 400s on an unknown provider; a spec that is not in the catalogue is accepted (the picker has a text box for exactly this) while still obeying the privacy rule, and an edited config shows up in the picker without a restart; `GET /config/summary` reports the loaded config grouped for display, never serving an API key, and is read fresh so an edit shows up without a restart; `GET /info` reports the routed model and the per-model cost breakdown, and leaves `served` empty for an ordinary model; a turn records OpenRouter's reported spend on the session and reports it in the reply event, while a provider reporting nothing leaves the session with no cost entry at all. The `wired_session` fixture pins every test session to an explicit local spec and installs fakes into the real per-spec provider cache, with a strict `_provider_for` that fails the test rather than building a real client — an unpinned session would otherwise reach the network and bill the developer The forceful stop: `POST /chat/stop` on a blocked turn ends the SSE stream with a `stopped` event and nothing else (no `reply`), reports `rolled_back: true`, and leaves `messages`, `display`, and `turn_starts` all back at their pre-turn values with every save on that path store-free (so nothing is indexed); the same session then accepts a fresh `/chat` immediately, with only the second exchange in its history; a stop on an idle session 404s; a stop clears that session's `pending_actions` while another session's survive. Both sides of the commit race are pinned: a stop arriving after a reply has been committed reports `rolled_back: false` and leaves the exchange standing, while a stop that beats the commit makes the worker stand down (no reply event, the reply never reaches `display`, and the indexed save never runs). Also: a compacting turn emits `status` events in order (`compacting` then `thinking`) ahead of the reply.
`test_llm.py`	`jarvis/core/llm.py`	Unit tests with the LLM clients mocked at the API boundary: `make_provider` spec dispatch including `provider:model` specs, a refusal when OpenRouter names no model, and `split_spec` keeping slashes/dots in the model half; `is_cloud_provider` classifying local vs everything else and failing closed on an unknown name; Ollama tool loop uses dict arguments directly (no JSON parsing) and normalises the pydantic message; all three providers honour the `PrivacyError` contract (return the error text, restore message history, no further LLM call); Anthropic tool results bundled into one user message on the wire actually sent, not just in the neutral history; the OpenRouter tool loop (JSON-string arguments parsed before dispatch, one `role:"tool"` message per call), its routing hardening and usage-accounting flag present on every request, the deliberate absence of the leaderboard headers, cost accumulated across a multi-request turn and drained by `pop_usage()`, and no cost invented when none came back or for the local/Anthropic providers; a router response naming a different model reports it back as `openrouter:<served>` while the request still asks for the router, and a response with no model field never invents one.. All three providers stream every request, so their fakes are closeable streams that record whether they were closed — closing is what tells the server to stop generating — and each loop reassembles content split across chunks into one reply. The OpenRouter fake goes furthest, splitting content in half and every tool call into an id/name chunk plus two argument fragments, so the delta accumulator is driven the way the real API drives it: two concurrent tool calls come back with their own ids and their arguments JSON rejoined, and cost still arrives on the stream's final chunk (with `stream_options` deliberately absent). Cancellation, for all three: a token stopped before the loop's first iteration makes zero requests and leaves `messages` untouched; a token stopped mid-stream raises `TurnCancelled`, closes the stream, and leaves `messages` untouched; a stop landing during tool dispatch pops the orphaned assistant message so no tool call is left without its result. Integration tests (marked): Anthropic client init, models-list auth check, Ollama reachability via `ollama.list()`
`test_webapp_documents.py`	`jarvis/webapp/app.py`	The library routes via `TestClient` against a real ChromaDB store (`jarvis.kb.store.get_store` monkeypatched to return the `store` fixture, same pattern as `test_webapp_chat.py`'s `wired_session`): `GET /documents` lists every indexed paper (de-duplicated, all the fields the frontend table needs) and `q=` filters case-insensitively across title, authors, doi, and source; `?kind=notes` lists vault records with their category/status/entity/event_date/file_path and filters by category and status server-side, while an omitted kind still means papers; `POST /documents/meta` updates only the given fields (others untouched) and 404s on an unknown source; `POST /documents/remove` calls `execute_remove` with the expected `{ids, title}` action, actually removes only the matching paper's chunks end-to-end, and 404s on an unknown source; a dedicated regression test spies on every plausible deletion API (`Path.unlink`/`rmdir`, `os.remove`, `os.unlink`, `shutil.rmtree`) during a `/documents/remove` request and asserts none is ever called; a note's source 404s on both `/documents/meta` and `/documents/remove` (routes scoped to `doc_type="paper"`)
`test_webapp_frontend.py`	`jarvis/webapp/static/app.js`, `index.html`, `style.css`	Structural checks over the frontend, since the repo has no JS harness. Each one guards a bug that actually happened: every `getElementById` the script reaches for exists in the markup; a pane given `display` by an id selector also has an `#id.hidden` rule, or `.hidden` cannot out-specify it; nothing schedules a save on a timer and `saveDraft` reads `tab.doc` rather than the editor, so the buffer holding both sides of a review is structurally unreachable by a save; a tab's saved and unsaved close glyphs are different; a PDF export shows it started, disables the button while it runs, and clears its counter in a `finally` so a timeout cannot leave the bar claiming an export is still going; and every `⋮ → …` instruction in README.md names a menu item that exists in the markup — the README told people to click a ⌘ button for weeks after model switching moved into the menu. The module docstring records why a check for undefined function calls is deliberately absent: JavaScript cannot be lexed with regular expressions, and two attempts produced false results in both directions

What is not tested¶

Module	Reason
`jarvis/digest/pipeline/` (fetch + scoring halves)	Depend on live LLM responses; correctness is validated by running the pipeline. The post-scoring indexing tiers ARE tested — see `test_pipeline_run.py`
`jarvis/chat/chat.py` (full agentic loop)	The end-to-end loop needs a live LLM; the guards it relies on are covered by `test_privacy_guard.py` / `test_security.py`, the tool loop mechanics by `test_llm.py`'s boundary-mocked tests, and all KB behaviour by `test_store.py`
`jarvis/webapp/` (browser rendering)	Requires a live browser; `test_webapp_chat.py` and `test_security.py` cover the `/chat` turn lifecycle and the other routes against `TestClient` with a fake provider, so only the actual pixels/DOM are unverified here
`jarvis/webapp/static/app.js` copy button (`buildAssistantBubble`)	Pure frontend DOM/clipboard interaction; no JS test harness exists in this repo. Verified by manual click + paste.
`jarvis/webapp/static/app.js` selection-copy markdown conversion (`htmlFragmentToMarkdown`)	Pure frontend `copy`-event handling; no JS test harness exists. Verified by manual selection across bold/list/code + paste into a plain editor.
`jarvis/webapp/static/app.js` per-session input drafts (`drafts`, `switchDraft`)	Pure frontend state with no JS test harness. Verified manually: type a partial message, switch sessions (via sidebar, "New chat", and delete), confirm the draft doesn't leak into the new session and reappears when switching back; also confirm that a failed send restores its text into the textarea if still viewing that session, or into its draft if the user has since switched away.
`jarvis/webapp/static/app.js` busy-resume polling (`pollUntilTurnLands`)	Pure frontend polling/DOM update with no JS test harness; the `busy` list it polls for is covered server-side by `test_webapp_chat.py`. Verified manually: send a question, switch away before the reply lands, switch back and confirm a "Working..." placeholder appears and resolves into the finished reply.
`jarvis/webapp/static/app.js` per-session composer state (`inFlight`, `serverBusy`, `updateComposerState`)	Pure frontend state with no JS test harness; the server-side `busy` list it reads is covered by `test_webapp_chat.py`. Verified manually: start a slow question in session A, switch to session B, confirm B's send button stays enabled and a message sent there completes independently while A is still working; confirm A's sidebar row shows a pulsing busy dot throughout.
`jarvis/webapp/static/app.js` Stop button + rollback (`stopGeneration`, `rollbackTurn`, the `stopped`/`status` event branches)	Pure frontend DOM/fetch wiring with no JS test harness; `POST /chat/stop` and the SSE events it reacts to are covered server-side by `test_webapp_chat.py`. The server half was driven end to end against a live `uv run webapp --provider ollama` (stop returns in 0.02 s, the stream carries only `stopped`, `busy` clears, `/history` is empty, and the next message succeeds). What remains manual is the DOM: ask a question that triggers several tool calls, click Stop mid-reply, and confirm the composer returns at once, the turn vanishes from the transcript entirely, and the typed text is back in the input box; then send a different message on the same session and confirm it works with no 409. Also verified by hand: stopping while viewing a different session puts the text back into the stopped session's draft rather than the visible textarea; and a session long enough to trip `compact_after_tokens` shows the pulsing "Compacting conversation history..." indicator before reverting to "Working...".
`jarvis/webapp/static/app.js` library modal (open/close, search debounce, inline edit, two-step remove)	Pure frontend DOM/fetch wiring with no JS test harness; the routes it calls (`GET /documents`, `POST /documents/meta`, `POST /documents/remove`) are covered by `test_webapp_documents.py`. Verified manually: menu → "Library" lists everything with no truncation; typing in the search box narrows the table after a short pause and matches title/authors/doi/source; Edit → change title/authors/doi → Save persists (confirmed via `kb list` or reopening the modal) and Cancel discards; Remove shows the two-step "Database entry only…" confirmation with the paper's real path, and only Confirm removes it from the list and from `kb list`, while the PDF/file on disk is left untouched.
`jarvis/webapp/static/app.js` editor tabs (`tabs`, `activateTab`, `closeTab`, `isDirty`)	Pure frontend state over CodeMirror `Doc`s with no JS test harness; the structural invariants are covered by `test_webapp_frontend.py` (nothing schedules a save, `saveDraft` reads `tab.doc`, the dot/× glyphs differ). Verified manually: open two files, type in one, switch away and back and confirm the edit and its undo history survive; confirm the unsaved tab shows a dot while the other shows an ×; click the dot and confirm it saves before closing; close the last tab and confirm the editor empties
`jarvis/webapp/static/app.js` inline diff rendering (`showProposalInEditor`, `buildSideBySide`)	Pure frontend DOM/CodeMirror widget work with no JS test harness; the hunk data it renders — including `old_spans`/`new_spans` covering only the changed lines — is covered by `test_drafts.py`. Verified manually against a multi-block removal: confirm only the blocks actually being removed are tinted and the surrounding context is not, in both the inline and the side-by-side layouts, and that narrowing the pane below 720px falls back to stacked
`jarvis/webapp/static/app.js` file context menu (`openFileMenu`)	Pure frontend DOM with no JS test harness; the route it calls is covered by `test_webapp_editor.py` (`/reveal` containment, fixed argv, no shell). Verified manually: right-click a document opens the menu at the pointer, "Show in Finder" reveals the file, "New file in this document…" adds to the same draft folder, and a pending suggestion adds Review/Discard entries
`jarvis/webapp/static/app.js` model picker text box (`useCustomModel`)	Pure frontend DOM with no JS test harness; the server side is covered by `test_webapp_chat.py` (an unlisted spec is accepted, and still refused for a private session). Verified manually: type `openrouter/auto` while on an OpenRouter model and confirm the bare id is prefixed with the current provider and applied
`jarvis/sync/daemon.py` (scheduler loop)	Process-runtime plumbing around APScheduler; the decision functions and job bodies it drives are covered by `test_daemon.py`