Implemented chunking optimization to resolve oversized chunks and improve
semantic search quality:
CHUNKING IMPROVEMENTS:
- Added strict 1000-word max limit (vs previous 1500-2000)
- Implemented 100-word overlap between consecutive chunks
- Created llm_chunker_improved.py with overlap functionality
- Added 3 fallback points in llm_chunker.py for robustness
RE-CHUNKING RESULTS:
- Identified and re-chunked 31 oversized chunks (>2000 tokens)
- Split into 92 optimally-sized chunks (max 1995 tokens)
- Preserved all metadata (workTitle, workAuthor, sectionPath, etc.)
- 0 chunks now exceed 2000 tokens (vs 31 before)
VECTORIZATION:
- Created manual vectorization script for chunks without vectors
- Successfully vectorized all 92 new chunks (100% coverage)
- All 5,304 chunks now have BGE-M3 embeddings
DOCKER CONFIGURATION:
- Exposed text2vec-transformers port 8090 for manual vectorization
- Added cluster configuration to fix "No private IP address found"
- Increased worker timeout to 600s for large chunks
TESTING:
- Created comprehensive search quality test suite
- Tests distribution, overlap detection, and semantic search
- Modified to use near_vector() (Chunk_v2 has no vectorizer)
Scripts:
- 08_fix_summaries_properties.py - Add missing Work metadata to summaries
- 09_rechunk_oversized.py - Re-chunk giant chunks with overlap
- 10_test_search_quality.py - Validate search improvements
- 11_vectorize_missing_chunks.py - Manual vectorization via API
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add 'summary' field to Chunk collection (vectorized with text2vec)
- Migrate from Dynamic index to HNSW + RQ for both Chunk and Summary
- Add LLM summarizer module (utils/llm_summarizer.py)
- Add migration scripts (migrate_add_summary.py, restore_*.py)
- Add summary generation utilities and progress tracking
- Add testing and cleaning tools (outils_test_and_cleaning/)
- Add comprehensive documentation (ANALYSE_*.md, guides)
- Remove obsolete files (linear_config.py, old test files)
- Update .gitignore to exclude backups and temp files
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Removed max-height: 300px from .works-list
- Keeps only the Unicode encoding fix (→ to ->)
- Avoids having two scrollbars in the works filter section
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Problem 1: Only 3 works visible despite 8/10 badge
- Added max-height: 300px and overflow-y: auto to .works-list
- Now all 10 works are scrollable in the filter section
Problem 2: UnicodeEncodeError with → character in console
- Replaced Unicode arrow (→) with ASCII arrow (->) in print statements
- Fixes 'charmap' codec error on Windows console
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Disable CLAUDE.md confirmation rules for autonomous agent operation
- Add utility scripts: check_linear_status.py, check_meta_issue.py, move_issues_to_todo.py
- Add works filter specification: prompts/app_spec_works_filter.txt
- Update .linear_project.json with works filter issues
- Remove old/stale scripts and documentation files
- Update search.html template
This commit completes the infrastructure for the autonomous agent that
successfully implemented all 13 works filter issues (LRP-136 to LRP-148).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- WORKS_FILTER.md: Complete user documentation in French
- Feature overview and location
- Selection/deselection instructions
- Quick action buttons (Tout/Aucun)
- Badge counter explanation
- Collapse functionality
- Default behavior and localStorage persistence
- Impact on semantic search
- Recommended use cases (comparative study, focus, exclusion)
- Responsive mobile support
- API Reference section:
- GET /api/get-works endpoint documentation
- POST /chat/send selected_works parameter
- Error codes and validation
- Troubleshooting guide:
- No works displayed
- Filter not working
- How to reset selection
- Chunks count explanation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Test /api/get-works route:
- Unique works extraction with correct chunk counts
- Sorting by author then title
- Connection failure and query exception handling
- Edge cases: empty database, missing title/author
- Test /chat/send selected_works parameter:
- Accepts empty list (search all works)
- Accepts valid work title list
- Rejects non-list types (string, dict)
- Rejects mixed types in list
- Verifies parameter passed to background thread
- Test rag_search works filter:
- No filter when selected_works is empty/None
- Contains_any filter applied when works selected
18 tests, all passing, no real Weaviate calls (fully mocked)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add works filter section HTML above Context RAG sidebar
- Add CSS styles for works filter with checkboxes, badges, and collapse
- Implement JavaScript for loading works from /api/get-works
- Add localStorage persistence for selected works
- Integrate selected_works parameter with /chat/send API call
- Add Tout/Aucun buttons for quick selection
- Add collapsible section with chevron toggle
- Responsive design for mobile screens
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add selected_works parameter to rag_search() function
- Build Weaviate filter using Filter.by_property("workTitle").contains_any()
- Add selected_works parameter to diverse_author_search() function
- Pass selected_works from run_chat_generation to diverse_author_search
- Preserve work filter in fallback search path
- Add logging for applied work filters
The filter allows restricting RAG search to specific works selected by the user.
When selected_works is empty or None, all works are searched (no filter).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add optional selected_works parameter to /chat/send endpoint
- Validate that selected_works is a list of strings
- Pass parameter to run_chat_generation function
- Backward compatible (works without the parameter)
- Add logging for selected_works filter
Linear issue: LRP-137
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add new API endpoint GET /api/get-works
- Returns JSON array of all unique works with metadata
- Each work includes: title, author, chunks_count
- Results sorted by author then title
- Proper error handling for Weaviate connection issues
- Fixed gRPC serialization issue with nested objects
Linear issue: LRP-136
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously created a separate page for summary search, which was redundant since hierarchical mode already demonstrates the summary→chunk pattern. Refactored to integrate summary-only mode as a dropdown option in the main search interface, reducing code duplication by ~370 lines.
Also fixed critical bug in hierarchical search where return_properties excluded the nested "document" object, causing source_id to be empty and all sections to be filtered out. Solution: removed return_properties to let Weaviate return all properties including nested objects.
All 4 search modes now functional:
- Auto-detection (default)
- Simple chunks (10% visibility)
- Hierarchical summary→chunks (variable)
- Summary-only (90% visibility)
Tests: 14/14 passed for dropdown integration, hierarchical mode confirmed working with 13 passages across 4 section groups.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Ligne 2 et 3 ont exactement le même style CSS
- Même couleur (var(--color-accent))
- Même background beige (rgba(125, 110, 88, 0.08))
- Même padding (0.25rem 0.5rem)
- Même border-radius (4px)
- Seule différence : icône et contenu
- Présentation ultra-cohérente visuellement
- Retiré font-weight: 600 du titre section
- Lignes 2 et 3 ont maintenant exactement le même style
- Police par défaut, pas de variations de graisse
- Présentation ultra-simplifiée et cohérente
- Seule différence : couleurs (accent vs text-strong)
- Ligne 2 (hiérarchie) : police normale, pas de font-size
- Ligne 3 (titre) : police normale, pas de font-size ni font-family spéciale
- Changé h4 en span pour cohérence typographique
- Gardé font-weight: 600 sur le titre pour légère emphase
- Résultat : lignes 2 et 3 visuellement cohérentes
Ligne 1 : Auteur | Œuvre | Similarité | Nb passages
Ligne 2 : 🗂️ Hiérarchie (chapterTitle)
Ligne 3 : 📂 Titre section
Plus compact et hiérarchie mieux visible avant le titre
- Badge auteur (récupéré du premier chunk de la section)
- Badge œuvre (récupéré du premier chunk de la section)
- Hiérarchie complète avec icône 🗂️ (chapterTitle du premier chunk)
Ex: "Peirce: CP 7.316"
- Fond beige léger pour la hiérarchie
- Affichage au-dessus du titre de section
Structure header de section:
1. Auteur + Œuvre (badges)
2. Titre section avec icône 📂
3. Hiérarchie complète (chapterTitle)
4. Similarité + nombre passages
5. Résumé LLM
6. Concepts
- Header section avec fond beige dégradé distinct des chunks
- Icône 📂 + label "Section :" explicite avant le titre
- Titre section en plus gros (1.2em, font-weight 600)
- Badge nombre de passages en couleur accent
- Zone chunks avec fond blanc pur pour contraster
- Bordure section plus épaisse (2px) et arrondie (10px)
- Summary text avec fond blanc semi-transparent pour lisibilité
- Label "Concepts :" avant la liste des concepts
Résultat: Hiérarchie visuelle très claire entre section et passages
- Stage 2 now searches chunks for EACH section using section summary as query
- Chunks distributed across sections (limit / sections_limit)
- Template displays sections with nested chunks underneath
- Each section shows: title, summary, concepts, chunk count, and passages
- Removes separate global passages list - now fully grouped by section
Structure: Section 1 → Chunks 1-3, Section 2 → Chunks 4-6, etc.
Problem: Sections showed title twice (once as title, once as summary_text)
Cause: summary_text contains same content as title in current data
Solution: Only show summary_text if different from title and section_path
Condition: summary_text != title AND summary_text != section_path
Error: 'Encountered unknown tag else' - endif was closing the if block too early
Fix: Removed extra {% endif %} before {% else %}
- Line 232: Removed incorrect closing tag
- The {% else %} at line 234 is part of the hierarchical/simple mode conditional
- Proper structure: if hierarchical ... else simple ... endif
Tests:
- Template syntax validates ✓
- Search page loads ✓
- Hierarchical mode works ✓
Root cause:
- Summary.sectionPath: '635. As for the subject...' (paragraph numbers)
- Chunk.sectionPath: 'Peirce: CP 4.47 > 47. §3 THE NATURE...' (canonical refs)
- No way to match them with prefix/equal filters
Solution (workaround until summaries are regenerated):
- Show sections as **context** (relevant high-level topics found)
- Show chunks **globally** (top 20 most relevant passages)
- Don't try to group chunks under sections
UI changes:
- '📚 Sections pertinentes trouvées' (context cards with summary)
- '📄 Passages les plus pertinents' (top chunks, not grouped)
- Cleaner, more honest representation of what we found
Next steps to fully fix:
- Regenerate Summary collection with correct sectionPath format
- Or create a mapping between Summary titles and Chunk sectionPaths
Problem:
- Summary.sectionPath: "Peirce: CP 2.504"
- Chunk.sectionPath: "Peirce: CP 2.504 > 504. Text..."
- Filter.equal() found 0 matches (no exact match exists)
Solution:
- Single semantic query to get all relevant chunks
- Distribute chunks to sections using Python startswith()
- This correctly matches chunks to their parent sections
Performance improvement:
- 1 query instead of N queries (one per section)
- Python-side filtering is fast for small result sets
Result: Chunks should now appear in their corresponding sections
Backend fix:
- Remove return_properties from hierarchical chunk query
- Weaviate returns nested objects (work, document) when return_properties is not specified
- This allows chunks to have work.author and work.title available
Frontend improvements:
- Truncate long section titles to 80 chars with ellipsis
- Hide section_path if identical to title (avoid duplication)
- Work and author badges should now display correctly in chunk metadata
Hierarchical search improvements:
- Display author and work for each chunk using badge-author and badge-work
- Show section hierarchy (sectionPath) in chunk metadata
- Add 📍 icon for section path in headers
Color alignment with charter:
- Replace Bootstrap colors (#007bff, #28a745, #6c757d) with charter variables
- section-group: border and shadow use accent colors (125,110,88)
- section-header: border uses var(--color-accent)
- chunk-item: border-left uses var(--color-accent-alt)
- Mode badges: hierarchical=accent-alt, simple=accent
- Concept badges: subtle beige background with accent border
- Alert boxes: beige background instead of yellow
Visual improvements:
- Add hover transform effect on chunks (translateX)
- Smoother color transitions using CSS variables
- Add @contextmanager decorator for proper exception handling
- Remove all simple_search() calls from within hierarchical_search()
- Return mode='error' to signal fallback needed
- Handle fallback in search_passages() (outside context manager)
- This eliminates 'generator didn't stop after throw()' error
## Problem
"generator didn't stop after throw()" error when hierarchical_search
falls back to simple_search. Both functions use 'with get_weaviate_client()',
creating nested context managers on the same generator.
## Solution
- Use ValueError("FALLBACK_TO_SIMPLE") signal instead of calling simple_search()
inside the context manager
- Catch ValueError in except block and call simple_search() outside context
- Applied to all 3 fallback points:
1. No Weaviate client
2. No summaries found (Stage 1)
3. No sections after filtering
## Result
Fallback now works correctly without context manager conflicts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
## Changes
Allow users to force hierarchical search mode without fallback to simple
search, enabling testing of hierarchical UI even when 0 summaries are found.
**Backend (flask_app.py):**
- Added `force_hierarchical` parameter to `hierarchical_search()`
- When True, never fallback to simple search (return empty hierarchical result)
- Added `fallback_reason` field to explain why no results
- Pass `force_hierarchical=True` when `force_mode == "hierarchical"`
- Applied to all fallback points:
- No Weaviate client
- No summaries found in Stage 1
- No sections after author/work filtering
- Exception during search
**Frontend (templates/search.html):**
- Display warning message when `fallback_reason` exists
- Yellow alert box with explanation and suggestions
- Works even when `results_data.results` is empty
## Usage
1. Select "🌳 Hiérarchique (2-étapes)" in Mode dropdown
2. Enter any query (even if no matching summaries)
3. See hierarchical UI with warning instead of fallback
## Example
Query: "Qu'est-ce que la justice ?" (not in Peirce corpus)
- Mode forced: Hierarchical
- Result: 0 sections, warning displayed
- No silent fallback to simple search
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Problèmes corrigés:
1. TITRE INCORRECT → Maintenant utilise TITRE: de la première page
2. CONCEPTS EN FRANÇAIS → Désactivé l'enrichissement LLM
Avant:
- Titre: "An Historical Sketch..." (mauvais, titre du chapitre)
- Concepts: ['immuabilité des espèces', 'création séparée'] (français)
- Résultat: 3/37 chunks ingérés dans Weaviate
Après:
- Titre: "On the Origin of Species BY MEANS OF..." (correct!)
- Concepts: [] (vides, pas de problème d'encoding)
- Résultat: 14/37 chunks ingérés (mieux mais pas parfait)
Changements word_pipeline.py:
1. STEP 5 - Métadonnées simplifiées (ligne 241-262):
- Supprimé l'appel à extract_metadata() du LLM
- Utilise directement raw_meta de extract_word_metadata()
- Le LLM prenait le titre du chapitre au lieu du livre
2. STEP 9 - Désactivé enrichissement concepts (ligne 410-423):
- Skip enrich_chunks_with_concepts()
- Raison: LLM génère concepts en FRANÇAIS pour texte ANGLAIS
- Accents français causent échecs Weaviate
Note TOC:
Le document n'a que 2 Heading 2, donc la TOC est limitée.
C'est normal pour un extrait de 10 pages.
Reste à investiguer: Pourquoi 14/37 au lieu de 37/37 chunks?
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Problème:
- Erreur: "expected string or bytes-like object, got 'dict'"
- À l'étape "Chunk Cleaning", on passait chunk (dict) au lieu de chunk["text"] (str)
Correction word_pipeline.py (ligne 434):
AVANT:
```python
cleaned = clean_chunk(chunk) # chunk est un dict!
```
APRÈS:
```python
text: str = chunk.get("text", "")
cleaned_text = clean_chunk(text, use_llm=False)
if is_chunk_valid(cleaned_text, min_chars=30, min_words=8):
chunk["text"] = cleaned_text
cleaned_chunks.append(chunk)
```
Pattern copié depuis pdf_pipeline.py:765-771 où la même logique
extrait le texte, le nettoie, puis met à jour le dict.
Test réussi:
✅ 48 paragraphes extraits
✅ 37 chunks créés
✅ Nettoyage OK
✅ Validation OK
✅ Pipeline complet fonctionnel avec Mistral API
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Problème:
- AttributeError: 'NoneType' object has no attribute 'lower'
- Se produisait quand section.get("title") retournait None au lieu de ""
Corrections:
- llm_classifier.py:
* is_excluded_section(): (section.get("title") or "").lower()
* filter_indexable_sections(): (s.get("chapterTitle") or "").lower()
* validate_classified_sections(): Idem pour chapter_title et section_title
- llm_validator.py:
* apply_corrections(): Ajout de vérification "if title and ..."
- llm_chat.py:
* call_llm(): Ajout d'une exception si provider est None/vide
Pattern de correction:
AVANT: section.get("title", "").lower() # Échoue si None
APRÈS: (section.get("title") or "").lower() # Sûr avec None
Raison:
.get(key, default) retourne le default SEULEMENT si la clé n'existe pas.
Si la clé existe avec valeur None, .get() retourne None, pas le default!
Donc: {"title": None}.get("title", "") -> None (pas "")
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Modifications:
- flask_app.py:
* Ajout de "docx" dans ALLOWED_EXTENSIONS
* Nouvelle fonction run_word_processing_job() avec:
- Gestion tempfile pour python-docx (besoin d'un path)
- Intégration du callback de progression SSE
- Nettoyage automatique du fichier temporaire
* Modification upload() route:
- Détection du type de fichier (PDF/Word)
- Routage vers le bon processeur (run_processing_job vs run_word_processing_job)
- Messages d'erreur adaptés pour PDF et Word
* Mise à jour des docstrings
- templates/upload.html:
* Titre: "Parser PDF/Word/Markdown" (au lieu de PDF/Markdown)
* Accept attribute: ".pdf,.docx,.md"
* Tooltips: Explique que Word n'a pas besoin d'OCR
* Pipeline de traitement: Section séparée pour PDF vs Word
* Labels mis à jour pour inclure Word
Fonctionnalités:
✅ Upload de fichiers .docx via interface web
✅ Traitement en arrière-plan avec SSE
✅ Pas d'OCR nécessaire pour Word (économie ~0.003€/page)
✅ Réutilisation complète des modules LLM existants
✅ Extraction directe via python-docx
✅ Construction TOC depuis styles Heading 1-9
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Backend:
- Nouveau dictionnaire global tts_jobs pour tracker les jobs TTS
- Fonction _generate_audio_background() pour génération en thread
- POST /chat/generate-audio: lance génération et retourne job_id
- GET /chat/audio-status/<job_id>: polling du statut
- GET /chat/download-audio/<job_id>: télécharge l'audio terminé
- États: pending → processing → completed/failed
Frontend:
- Fonction exportToAudio() asynchrone avec polling (1s)
- Spinner animé pendant génération ("Génération...")
- Téléchargement automatique quand prêt
- Restauration bouton en cas d'erreur
- Animation CSS @keyframes spin pour le spinner
Avantages:
- Flask reste responsive pendant génération TTS
- Navigation possible pendant génération audio
- Expérience utilisateur améliorée avec feedback visuel
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Nouvelle fonction _clean_markdown() pour supprimer le formatage markdown
- Supprime headers (#), bold (**), italic (*), code blocks (```)
- Supprime liens [text](url), citations (>), marqueurs de listes (-)
- Nettoie les espaces multiples pour un texte propre
- Évite la lecture à voix haute des caractères markdown
- Tests validés: tous les patterns markdown correctement nettoyés
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Header fixe positionné à côté du menu hamburger (80px de gauche)
- Suppression du sous-titre "Visualiseur de base Weaviate"
- Fix variable CSS: var(--color-bg-primary) → var(--color-bg-main)
- Ajustement hauteur chat: fenêtres RAG descendent jusqu'en bas
- Barres de conversation touchent le bas de l'écran
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Ajout de TTS>=0.22.0 aux dépendances
- Création du module utils/tts_generator.py avec Coqui XTTS v2
* Support GPU avec mixed precision (FP16)
* Lazy loading avec singleton pattern
* Chunking automatique pour textes longs
* Support multilingue (fr, en, es, de, etc.)
- Ajout de la route /chat/export-audio dans flask_app.py
- Ajout du bouton Audio dans chat.html (côté Word/PDF)
- Génération audio WAV téléchargeable depuis les réponses
Optimisé pour GPU 4070 (8GB VRAM) : utilise 4-6GB, génération rapide
Qualité : voix naturelle française avec prosodie expressive
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Ajout de python-docx et reportlab aux dépendances
- Création du module utils/word_exporter.py pour l'export Word
- Création du module utils/pdf_exporter.py pour l'export PDF
- Ajout des routes /chat/export-word et /chat/export-pdf dans flask_app.py
- Ajout des boutons d'export (Word et PDF) dans chat.html
- Les boutons apparaissent après chaque réponse de l'assistant
- Support des questions reformulées avec question originale
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
