Add ink integration UI and media playback

This commit is contained in:
2026-05-15 21:23:46 +02:00
parent 44dc64f830
commit f2e786d5bc
89 changed files with 6561 additions and 556 deletions
+22 -12
View File
@@ -12,7 +12,7 @@ The production client must tolerate TTS being unavailable. The safe default TTS
- Done: native ES module loader with dependency graph, module states, progress overlay, cache-busted development loading, and ordered async initialization.
- Done: responsive book layout that scales page, font sizes, and word positions relative to page size.
- Done: story parser for chapters, text blocks, Markdown emphasis, image blocks, sound effect cues, and music cues.
- Done: story parser/protocol bridge for Ink-style `#` tags, chapters, text blocks, Markdown emphasis, image blocks, sound effect cues, and music cues.
- Done: SmartyPants punctuation, language-aware Hyphenopoly integration, and Knuth-Plass paragraph line breaking.
- Done: paragraph rules for normal paragraphs, chapter-first paragraphs, textblock-first paragraphs, drop caps, and first-line indentation.
- Done: sentence queue and playback coordinator for preparing text and TTS before synchronized playback.
@@ -124,6 +124,13 @@ Known inherited implementation note: the old `linked-list.js` analysis identifie
## Story Markup Specification
Canonical structural and media tags use Ink-style `#` syntax:
- Ink engines write native Ink tags such as `# chapter[Title]`, `# image[file.png](landscape)`, or `# music[track.ogg](crossfade loop)`.
- inkjs exposes those tags without the leading `#`; the server parses them into `StoryTag` objects.
- YAML and Zork narrative output use the same leading `#...` syntax, parsed by the server into `StoryTag` objects before the client sees them.
- The browser protocol is structured `TurnResult` objects with structured tags and render blocks, not raw story markup.
Markdown emphasis:
```text
@@ -135,7 +142,7 @@ Markdown emphasis:
Chapter:
```text
::chapter[The Mysterious Mansion]
#chapter[The Mysterious Mansion]
The first paragraph has a drop cap and no first-line indent.
```
@@ -145,35 +152,36 @@ The heading is centered, italic, and uses the body font size. Following ordinary
Section or text block:
```text
::section
#section
The first paragraph is vertically separated from previous content and has no first-line indent.
```
`::textblock` is an alias. Following ordinary paragraphs return to normal indentation.
`#textblock` is an alias. Following ordinary paragraphs return to normal indentation.
Images:
```text
::image[widescreen](file-name.jpg)
::image[portrait](file-name.jpg)
#image[file-name.jpg](landscape)
#image[file-name.jpg](portrait pause=2)
#image[file-name.jpg](square delay=1.5)
```
File names resolve relative to `public/images/`. `widescreen` means full page width and half page height. `portrait` means full page width and full page height. Parsing is implemented; visual rendering is pending.
File names resolve relative to `public/images/`. `widescreen` is still accepted as an alias for `landscape`. Landscape and square images are centered and rendered near full page width with heights snapped to whole text lines. Portrait images float at half page width and following prose is narrowed for the number of lines the image covers. Image pauses accept the same timing style as music (`pause=2`, `delay=2`, `lead=2`, or `2s`); pauses are skippable with click/space and do not prevent the next TTS item from being prepared in the background.
Sound effects:
```text
The old door opens {{sfx:squeaky-door.ogg}} into the dark.
#sfx[squeaky-door.ogg]
The old door opens into the dark.
```
File names resolve relative to `public/sounds/`. The marker is not displayed and is not sent to TTS. Playback starts when animation reaches the marker position.
File names resolve relative to `public/sounds/`. The server parses the tag into a `StoryTag`; the tag is not displayed and is not sent to TTS.
Music:
```text
::music[crossfade, loop, lead=4](track.ogg)
{{music:cut:track.ogg}}
#music[track.ogg](crossfade, loop, lead=4)
```
File names resolve relative to `public/music/`. Modes:
@@ -185,6 +193,8 @@ File names resolve relative to `public/music/`. Modes:
- `once`: do not repeat the track.
- `lead=<seconds>`: for block music, let music play alone before the following text/TTS paragraph starts.
For chapter openings, authors can place `#music[file](..., lead=N)` after `#chapter[...]` and before the first prose paragraph. The heading is rendered/spoken first, then music starts and plays alone for the lead duration, then the dropcapped paragraph continues. Music and image pauses are playback gates only; they must not stop the queue from preparing upcoming TTS in the background.
## TTS And Playback Specification
The playback system must keep text animation and audio synchronized.
@@ -354,7 +364,7 @@ Longer-term goal:
### Pending
- [ ] Implement image rendering for `::image[widescreen]` and `::image[portrait]`.
- [x] Implement image rendering for `#image[file](landscape)`, `#image[file](portrait)`, and `#image[file](square)`.
- [ ] Replace placeholder save implementation with durable save files or server-side save storage.
- [ ] Replace command mirroring with the full LLM/world-model command loop when typography/audio testing no longer needs mirroring.
- [ ] Add optical margin alignment or punctuation protrusion support for line endings.