Add content index: keywords, tags/categories, search, and XML sitemap

Ships Blog tags/categories, Internal search, and XML sitemap together as
one shared mechanism, plus a new meta keywords request, rather than three
separate ones - the "worth deciding together" call ISSUES.md made when
XML sitemap was first written.

Content stays in files. Per-page metadata is four Twig blocks in the
root layout, the same override mechanism already used for
title/description/og_* - keywords (rendered), tags/changefreq/priority
(not rendered, harvested only). App\ContentIndexer crawls every routable
page (Overlay::listPageDirs(), new) and pulls each block via
Renderer::renderForIndex() (new) calling Twig's own renderBlock() API,
not regex-parsing .twig source, so overrides and layout inheritance
resolve exactly like a real render. Rendered HTML is stripped and
indexed into a SQLite FTS5 table for search.

Off by default (content_index_enabled) - same posture as Matomo/admin
auth, since this is a real SQLite dependency plenty of sites won't want.
Verified true zero footprint when disabled: no data/novaconium.sqlite
gets created, all three consumer routes 404 like they don't exist.
content_index_auto (default true) reindexes lazily on first stale touch
of a consumer route, never on a normal page view; both that path and the
explicit `php novaconium/bin/index-content.php` share one reindex().

Two real bugs caught by testing, not review: a reentrancy bug where
/search's own sidecar calling ensureFresh() during the crawl triggered a
nested reindex() mid-transaction (fixed with a static re-entrancy guard),
and a wrong PDO constant in the search sidecar. Also fixed two unrelated
pre-existing bugs found while building this: an unescaped {{ }} in
admin/docs/sidecars that fataled Twig on any real render of that page,
and a stale "no database layer yet" claim there and in Lib\Input's
docblock, left over from before Lib\Db shipped.

Lib\Db's migrations_dir now accepts an ordered list of roots, not just
one path, so the content index's schema could ship as a framework
migration (novaconium/migrations/) without colliding with project
migrations in App/migrations/ - the two-root extension point flagged in
AGENTS.md when SQLite groundwork shipped. Migrations are tracked by path
relative to the repo root rather than bare filename so two roots with a
same-named file can't shadow each other.

New consumer routes: novaconium/pages/sitemap.xml/ and
novaconium/pages/search/ (framework defaults), App/pages/blog/tag/[tag]/
(project-owned, since blog/ is project content - the existing hand
-written post array in App/pages/blog/index.php is untouched). Added
tags to the 4 existing blog posts as a real demonstration.

Closes the Blog tags/categories, Internal search, and XML sitemap
backlog items in novaconium/ISSUES.md.
This commit is contained in:
code
2026-07-14 17:35:30 +00:00
parent 4862526fa1
commit 37b6764431
30 changed files with 1123 additions and 157 deletions
+70 -14
View File
@@ -146,15 +146,27 @@ parameterized queries as the sole SQL-injection defense. Each connection is
lazy and independent (opened on first `Db::query()`/`Db::connection()` call
naming it, same shape as `Lib\Csrf`'s lazy session start), and migrates
automatically at that point: plain `.sql` files under that connection's own
`migrations_dir`, filename-ordered, tracked in that connection's own
auto-created `schema_migrations` table (each connection's applied
migrations are independent of any other connection's), run once each.
`novaconium/bin/migrate.php` loops every configured connection and runs the
same migration step explicitly (e.g. from a deploy script) without serving
a request first. Only `'sqlite'` and `'mysql'` drivers are implemented; a
connection's `migrations_dir` is optional — omit it to never run migrations
against that connection (e.g. a legacy database this project shouldn't
manage schema for).
`migrations_dir` — a single path, or (since the content index shipped) an
**ordered list of roots**, each root's own files filename-ordered and
roots processed fully in the order given, not interleaved by filename
across roots (so a framework root always finishes before a project root on
the same connection). Tracked by path **relative to the repo root** (e.g.
`novaconium/migrations/0001_x.sql`), not bare filename — two roots can
each contain a same-named file, and tracking by bare filename would make
the second one seen look "already applied" and silently skip it; a
repo-relative path is also portable across environments, unlike a full
absolute path, which would make every migration look new again after a
clone/deploy to a different directory. `realpath()` normalizes any `..` a
`migrations_dir` like `__DIR__ . '/../App/migrations'` would otherwise
leave in the tracked name. Each connection's own auto-created
`schema_migrations` table tracks its migrations independently of any other
connection's; each is only ever run once. `novaconium/bin/migrate.php`
loops every configured connection and runs the same migration step
explicitly (e.g. from a deploy script) without serving a request first.
Only `'sqlite'` and `'mysql'` drivers are implemented; a connection's
`migrations_dir` is optional — omit it to never run migrations against
that connection (e.g. a legacy database this project shouldn't manage
schema for).
**`db_connections` is the one config key in the project that isn't plain
shallow-merge** — `bootstrap.php`/`bin/*.php`'s usual `array_merge($config,
@@ -185,11 +197,12 @@ instead — project-owned like `App/`, gitignored per-file
(`*.sqlite`/`-journal`/`-wal`/`-shm`, with a tracked `.gitkeep` so the
directory exists in a fresh clone) rather than wholesale like
`public/cache/`, since a project might reasonably want other non-DB files
there later. Only `App/migrations/` (the framework default connection's
`migrations_dir`) is scanned by default — the framework ships no core
tables of its own yet — if a future framework feature needs a shipped
migration, extend this to the same App-over-novaconium two-root scan
`Overlay.php` already does for pages/lib, don't invent a second mechanism.
there later. The default connection's `migrations_dir` scans
`novaconium/migrations/` (framework-shipped schema, e.g. the content
index below) before `App/migrations/` (project schema) — see the
`migrations_dir` array-form paragraph above. Any *other* connection a
project adds still defaults to no `migrations_dir` at all unless it sets
one; the two-root default is specific to `default`.
`Lib\Session` (`novaconium/lib/Session.php`) is a thin wrapper around
native PHP sessions (`session_start()`/`$_SESSION`, not a custom store),
@@ -257,6 +270,49 @@ something is gated there, which defeats the point of hiding it. Returns
consistently with the rest of `/admin/*`: wide open until a password is
configured, gated once one is.
`App\ContentIndexer` (`novaconium/src/ContentIndexer.php`) is the shared
crawler behind `/sitemap.xml`, `/search`, and blog tag browsing (see
`/admin/docs/content-index`) — `App\`, not `Lib\`, since it's rendering
infrastructure akin to `Renderer`/`Router`, not project-overridable
content. Content stays in files; only metadata is indexed. Per-page
metadata is four Twig blocks declared in the root layout next to the SEO
blocks (`keywords`, `tags`, `changefreq`, `priority` — the last three
never rendered into the page, only harvested) and pulled via
`Renderer::renderForIndex()`, which calls Twig's own
`TemplateWrapper::renderBlock()` per block rather than parsing `.twig`
source — this is deliberate: it gets App-over-novaconium override and
layout-inheritance resolution for free, the same way a real render does.
**`config['content_index_enabled']` defaults to `false`** — same posture
as `matomo_url`/`admin_password_hash`, since this is a real SQLite
dependency plenty of sites won't want. Every consumer route checks the
flag *before* touching `Lib\Db` and 404s if it's off, so the feature has
zero filesystem footprint (no `data/novaconium.sqlite`) when disabled —
verified end-to-end, not assumed, since "off" silently still creating a
database file would defeat the point.
**Reentrancy hazard, already hit once:** `ContentIndexer::reindex()`
renders every routable page as part of the crawl — including `/search`
itself, which is also a real page and also calls
`ContentIndexer::ensureFresh()` from its own sidecar. Without a guard,
crawling `/search` would trigger a nested `reindex()` call mid-transaction
and fatal on a second `PDO::beginTransaction()`. `ContentIndexer` guards
this with a `private static bool $indexing` flag, checked at the top of
both `ensureFresh()` and `reindex()` — either no-ops while a reindex is
already running on the call stack. Any future consumer route added under
this mechanism inherits the same hazard for free (it'll also get crawled,
and if its sidecar also calls `ensureFresh()`, the guard already covers
it) — don't remove the flag thinking it's unnecessary.
`reindex()` also forces `$_SERVER['REQUEST_METHOD']` to `'GET'` for the
duration of the crawl (restoring whatever it was before, in a `finally`)
— sidecars are expected to be side-effect-free for non-POST requests
anyway (ordinary HTTP-safe-method hygiene), but this guarantees a lazy
reindex triggered from within a POST request can never leak that POST
into an unrelated page's sidecar purely because the crawler happened to
render it. A crawl is a full truncate-and-rebuild inside one transaction,
not incremental — simple and correct at this site's scale; don't add
incremental/diffing logic without a real need for it.
## Running it
```