Add content index: keywords, tags/categories, search, and XML sitemap
Ships Blog tags/categories, Internal search, and XML sitemap together as
one shared mechanism, plus a new meta keywords request, rather than three
separate ones - the "worth deciding together" call ISSUES.md made when
XML sitemap was first written.
Content stays in files. Per-page metadata is four Twig blocks in the
root layout, the same override mechanism already used for
title/description/og_* - keywords (rendered), tags/changefreq/priority
(not rendered, harvested only). App\ContentIndexer crawls every routable
page (Overlay::listPageDirs(), new) and pulls each block via
Renderer::renderForIndex() (new) calling Twig's own renderBlock() API,
not regex-parsing .twig source, so overrides and layout inheritance
resolve exactly like a real render. Rendered HTML is stripped and
indexed into a SQLite FTS5 table for search.
Off by default (content_index_enabled) - same posture as Matomo/admin
auth, since this is a real SQLite dependency plenty of sites won't want.
Verified true zero footprint when disabled: no data/novaconium.sqlite
gets created, all three consumer routes 404 like they don't exist.
content_index_auto (default true) reindexes lazily on first stale touch
of a consumer route, never on a normal page view; both that path and the
explicit `php novaconium/bin/index-content.php` share one reindex().
Two real bugs caught by testing, not review: a reentrancy bug where
/search's own sidecar calling ensureFresh() during the crawl triggered a
nested reindex() mid-transaction (fixed with a static re-entrancy guard),
and a wrong PDO constant in the search sidecar. Also fixed two unrelated
pre-existing bugs found while building this: an unescaped {{ }} in
admin/docs/sidecars that fataled Twig on any real render of that page,
and a stale "no database layer yet" claim there and in Lib\Input's
docblock, left over from before Lib\Db shipped.
Lib\Db's migrations_dir now accepts an ordered list of roots, not just
one path, so the content index's schema could ship as a framework
migration (novaconium/migrations/) without colliding with project
migrations in App/migrations/ - the two-root extension point flagged in
AGENTS.md when SQLite groundwork shipped. Migrations are tracked by path
relative to the repo root rather than bare filename so two roots with a
same-named file can't shadow each other.
New consumer routes: novaconium/pages/sitemap.xml/ and
novaconium/pages/search/ (framework defaults), App/pages/blog/tag/[tag]/
(project-owned, since blog/ is project content - the existing hand
-written post array in App/pages/blog/index.php is untouched). Added
tags to the 4 existing blog posts as a real demonstration.
Closes the Blog tags/categories, Internal search, and XML sitemap
backlog items in novaconium/ISSUES.md.
This commit is contained in:
@@ -146,15 +146,27 @@ parameterized queries as the sole SQL-injection defense. Each connection is
|
||||
lazy and independent (opened on first `Db::query()`/`Db::connection()` call
|
||||
naming it, same shape as `Lib\Csrf`'s lazy session start), and migrates
|
||||
automatically at that point: plain `.sql` files under that connection's own
|
||||
`migrations_dir`, filename-ordered, tracked in that connection's own
|
||||
auto-created `schema_migrations` table (each connection's applied
|
||||
migrations are independent of any other connection's), run once each.
|
||||
`novaconium/bin/migrate.php` loops every configured connection and runs the
|
||||
same migration step explicitly (e.g. from a deploy script) without serving
|
||||
a request first. Only `'sqlite'` and `'mysql'` drivers are implemented; a
|
||||
connection's `migrations_dir` is optional — omit it to never run migrations
|
||||
against that connection (e.g. a legacy database this project shouldn't
|
||||
manage schema for).
|
||||
`migrations_dir` — a single path, or (since the content index shipped) an
|
||||
**ordered list of roots**, each root's own files filename-ordered and
|
||||
roots processed fully in the order given, not interleaved by filename
|
||||
across roots (so a framework root always finishes before a project root on
|
||||
the same connection). Tracked by path **relative to the repo root** (e.g.
|
||||
`novaconium/migrations/0001_x.sql`), not bare filename — two roots can
|
||||
each contain a same-named file, and tracking by bare filename would make
|
||||
the second one seen look "already applied" and silently skip it; a
|
||||
repo-relative path is also portable across environments, unlike a full
|
||||
absolute path, which would make every migration look new again after a
|
||||
clone/deploy to a different directory. `realpath()` normalizes any `..` a
|
||||
`migrations_dir` like `__DIR__ . '/../App/migrations'` would otherwise
|
||||
leave in the tracked name. Each connection's own auto-created
|
||||
`schema_migrations` table tracks its migrations independently of any other
|
||||
connection's; each is only ever run once. `novaconium/bin/migrate.php`
|
||||
loops every configured connection and runs the same migration step
|
||||
explicitly (e.g. from a deploy script) without serving a request first.
|
||||
Only `'sqlite'` and `'mysql'` drivers are implemented; a connection's
|
||||
`migrations_dir` is optional — omit it to never run migrations against
|
||||
that connection (e.g. a legacy database this project shouldn't manage
|
||||
schema for).
|
||||
|
||||
**`db_connections` is the one config key in the project that isn't plain
|
||||
shallow-merge** — `bootstrap.php`/`bin/*.php`'s usual `array_merge($config,
|
||||
@@ -185,11 +197,12 @@ instead — project-owned like `App/`, gitignored per-file
|
||||
(`*.sqlite`/`-journal`/`-wal`/`-shm`, with a tracked `.gitkeep` so the
|
||||
directory exists in a fresh clone) rather than wholesale like
|
||||
`public/cache/`, since a project might reasonably want other non-DB files
|
||||
there later. Only `App/migrations/` (the framework default connection's
|
||||
`migrations_dir`) is scanned by default — the framework ships no core
|
||||
tables of its own yet — if a future framework feature needs a shipped
|
||||
migration, extend this to the same App-over-novaconium two-root scan
|
||||
`Overlay.php` already does for pages/lib, don't invent a second mechanism.
|
||||
there later. The default connection's `migrations_dir` scans
|
||||
`novaconium/migrations/` (framework-shipped schema, e.g. the content
|
||||
index below) before `App/migrations/` (project schema) — see the
|
||||
`migrations_dir` array-form paragraph above. Any *other* connection a
|
||||
project adds still defaults to no `migrations_dir` at all unless it sets
|
||||
one; the two-root default is specific to `default`.
|
||||
|
||||
`Lib\Session` (`novaconium/lib/Session.php`) is a thin wrapper around
|
||||
native PHP sessions (`session_start()`/`$_SESSION`, not a custom store),
|
||||
@@ -257,6 +270,49 @@ something is gated there, which defeats the point of hiding it. Returns
|
||||
consistently with the rest of `/admin/*`: wide open until a password is
|
||||
configured, gated once one is.
|
||||
|
||||
`App\ContentIndexer` (`novaconium/src/ContentIndexer.php`) is the shared
|
||||
crawler behind `/sitemap.xml`, `/search`, and blog tag browsing (see
|
||||
`/admin/docs/content-index`) — `App\`, not `Lib\`, since it's rendering
|
||||
infrastructure akin to `Renderer`/`Router`, not project-overridable
|
||||
content. Content stays in files; only metadata is indexed. Per-page
|
||||
metadata is four Twig blocks declared in the root layout next to the SEO
|
||||
blocks (`keywords`, `tags`, `changefreq`, `priority` — the last three
|
||||
never rendered into the page, only harvested) and pulled via
|
||||
`Renderer::renderForIndex()`, which calls Twig's own
|
||||
`TemplateWrapper::renderBlock()` per block rather than parsing `.twig`
|
||||
source — this is deliberate: it gets App-over-novaconium override and
|
||||
layout-inheritance resolution for free, the same way a real render does.
|
||||
**`config['content_index_enabled']` defaults to `false`** — same posture
|
||||
as `matomo_url`/`admin_password_hash`, since this is a real SQLite
|
||||
dependency plenty of sites won't want. Every consumer route checks the
|
||||
flag *before* touching `Lib\Db` and 404s if it's off, so the feature has
|
||||
zero filesystem footprint (no `data/novaconium.sqlite`) when disabled —
|
||||
verified end-to-end, not assumed, since "off" silently still creating a
|
||||
database file would defeat the point.
|
||||
|
||||
**Reentrancy hazard, already hit once:** `ContentIndexer::reindex()`
|
||||
renders every routable page as part of the crawl — including `/search`
|
||||
itself, which is also a real page and also calls
|
||||
`ContentIndexer::ensureFresh()` from its own sidecar. Without a guard,
|
||||
crawling `/search` would trigger a nested `reindex()` call mid-transaction
|
||||
and fatal on a second `PDO::beginTransaction()`. `ContentIndexer` guards
|
||||
this with a `private static bool $indexing` flag, checked at the top of
|
||||
both `ensureFresh()` and `reindex()` — either no-ops while a reindex is
|
||||
already running on the call stack. Any future consumer route added under
|
||||
this mechanism inherits the same hazard for free (it'll also get crawled,
|
||||
and if its sidecar also calls `ensureFresh()`, the guard already covers
|
||||
it) — don't remove the flag thinking it's unnecessary.
|
||||
|
||||
`reindex()` also forces `$_SERVER['REQUEST_METHOD']` to `'GET'` for the
|
||||
duration of the crawl (restoring whatever it was before, in a `finally`)
|
||||
— sidecars are expected to be side-effect-free for non-POST requests
|
||||
anyway (ordinary HTTP-safe-method hygiene), but this guarantees a lazy
|
||||
reindex triggered from within a POST request can never leak that POST
|
||||
into an unrelated page's sidecar purely because the crawler happened to
|
||||
render it. A crawl is a full truncate-and-rebuild inside one transaction,
|
||||
not incremental — simple and correct at this site's scale; don't add
|
||||
incremental/diffing logic without a real need for it.
|
||||
|
||||
## Running it
|
||||
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user