ai.interactive.fiction/references/CLIENT_TODO_v1.md

Client Architecture Refactoring TODO List

Enhanced SentenceQueueModule Implementation

Phase 1: Core Structure and Design

1. Design the Sentence Object Structure

   • Define comprehensive sentence object with fields for:
     • Unique ID
     • Original text
     • Processed text (hyphenated, typeset)
     • Layout information (breaks, nodes, typography)
     • Audio component (player, duration, data, type)
     • Status tracking (pending, processing, ready, playing, complete)

2. Implement Basic Queue Management

   • Create methods for adding sentences to the queue
   • Implement queue processing logic that maintains order
   • Add status tracking for each sentence in the queue
   • Implement priority handling for urgent sentences

Phase 2: Text Processing Integration

1. Integrate with Paragraph Layout

   • Connect to ParagraphLayoutModule for text processing
   • Implement hyphenation and typesetting in the queue
   • Store layout information in the sentence object
   • Ensure layout processing happens in parallel with audio

2. Text Animation Preparation

   • Calculate animation timing based on text length and settings
   • Prepare animation data for each word in the sentence
   • Store animation timing in the sentence object
   • Create animation player function for the sentence

Phase 3: Audio Processing Integration

1. TTS System Integration

   • Implement audio generation for Kokoro TTS
   • Implement browser TTS handling with duration estimation
   • Implement "none" TTS option with duration calculation
   • Create consistent player interface for all TTS types

2. Audio Data Management

   • Implement audio data storage in sentence objects
   • Connect with TTSFactoryModule's IndexedDB for persistent caching
   • Add audio preloading capabilities
   • Implement audio resource cleanup

Phase 4: Playback Coordination

1. Synchronized Playback

   • Implement coordinated text animation and audio playback
   • Create timing adjustment based on speed settings
   • Add event handling for playback states (start, pause, resume, complete)
   • Implement sentence transition handling

2. User Interaction Handling

   • Add support for fast-forwarding text/audio
   • Implement pause/resume functionality
   • Handle user interruptions gracefully
   • Support skipping to next sentence

Component Consolidation

Phase 1: Identify and Remove Redundancies

1. TTSPlayerModule Refactoring

   • Remove preloadedAudio Map (replaced by sentence objects)
   • Remove preloadQueue (replaced by SentenceQueue)
   • Update speak method to use SentenceQueue
   • Refactor to be a thin wrapper around SentenceQueue

2. UIDisplayHandlerModule Refactoring

   • Remove pendingParagraphs queue (replaced by SentenceQueue)
   • Update displayText to use SentenceQueue
   • Modify processNextParagraph to work with sentence objects
   • Update event handling to work with the new architecture

3. KokoroTTSModule Refactoring

   • Replace pendingGenerations Map with SentenceQueue integration
   • Update generateSpeech to work with sentence objects
   • Modify iframe communication to support the new structure
   • Ensure backward compatibility during transition

4. TextBufferModule Refactoring

   • Move sentence preparation logic to SentenceQueue
   • Update text handling to work with the new architecture
   • Ensure proper integration with SentenceQueue
   • Maintain high-level text management responsibilities

Phase 2: Interface Updates

1. Update Module Interfaces

   • Create consistent interfaces for interacting with SentenceQueue
   • Update event system to work with sentence objects
   • Implement progress reporting for sentence processing
   • Add debugging and monitoring capabilities

2. Documentation and Examples

   • Document the new architecture and interfaces
   • Create usage examples for common scenarios
   • Update developer guidelines
   • Add migration guide for existing code

Testing and Validation

1. Unit Testing

   • Create tests for SentenceQueue core functionality
   • Test text processing integration
   • Test audio processing integration
   • Test playback coordination

2. Integration Testing

   • Test interaction between SentenceQueue and other modules
   • Validate timing and synchronization
   • Test error handling and recovery
   • Verify performance under load

3. User Experience Testing

   • Validate text animation quality
   • Test audio playback quality
   • Verify synchronization from user perspective
   • Test accessibility features

Implementation Strategy

1. Phased Rollout

   • Implement SentenceQueue core structure
   • Add text processing integration
   • Add audio processing integration
   • Implement playback coordination
   • Gradually replace existing components

2. Backward Compatibility

   • Maintain support for existing interfaces during transition
   • Implement adapter patterns where needed
   • Add feature flags for enabling/disabling new architecture
   • Create fallback mechanisms for error recovery

3. Performance Optimization

   • Implement parallel processing where possible
   • Optimize memory usage for sentence objects
   • Add resource management for audio data
   • Implement efficient queue processing algorithms