docs: Update CHANGELOG with GPU embedder migration and documentation overhaul

Added comprehensive entry for 2026-01-09 covering:

GPU Embedder Migration:
- Complete migration to Python GPU embedder (30-70x faster)
- Removed Docker text2vec-transformers service (saves 10GB RAM)
- Manual vectorization for all chunks
- Zero data loss (5,355 chunks preserved)

Testing & Validation:
- Chat RAG test with Puppeteer (hierarchical search confirmed)
- Memories API validated (104 thoughts, 380 messages)
- Performance metrics documented

Documentation Reorganization:
- Created docs/ structure with migration-gpu/ subdirectory
- Moved 6 migration MD files
- Removed 15 obsolete session reports
- Complete README.md rewrite explaining dual RAG system

Docker Configuration:
- Simplified docker-compose.yml
- Weaviate-only deployment

Database Stats:
- Current state: 5,355 chunks, 18 works, 104 thoughts, 12 conversations

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
2026-01-09 12:57:55 +01:00
parent c90864e9f7
commit b1dee3ae5f

View File

@@ -1,5 +1,54 @@
# Changelog - Library RAG Project
## 2026-01-09 - GPU Embedder Migration Complete & Documentation Overhaul
### GPU Embedder Migration (Complete)
- **Full Python GPU embedder**: Migrated from Docker text2vec-transformers to unified Python GPU embedder
- **30-70x faster ingestion**: GPU acceleration (RTX 4070, PyTorch CUDA, FP16)
- **Simplified architecture**: Single embedder for both ingestion and queries
- **Removed Docker service**: text2vec-transformers service removed, saves 10GB RAM
- **Manual vectorization**: All chunks now pre-vectorized before Weaviate insertion
- **Zero data loss**: All 5,355 existing chunks preserved during migration
### Testing & Validation
- **Chat RAG test**: Puppeteer test confirms hierarchical search (11 chunks, 5 sections)
- **Memories test**: Backend API validated (104 thoughts, 380 messages, 12 conversations)
- **Conversations test**: Search functionality working correctly
- **Performance metrics**: ~17ms query vectorization, ~100-500ms Weaviate search
### Documentation Reorganization
- **Created `docs/` structure**: Organized migration docs into `docs/migration-gpu/`
- **Moved 6 migration files**: BUG_REPORT, DIAGNOSTIC, MIGRATION_SUCCESS, TEST reports
- **Moved project_progress.md**: Centralized in `docs/`
- **Removed 15 obsolete files**: Cleaned up temporary session reports and outdated docs
- **README.md rewrite**: Comprehensive documentation of dual RAG system
- 5 Weaviate collections explained (Work, Chunk_v2, Summary_v2, Thought, Conversation)
- Library Philosophique + Memory Ikario architecture
- Quick start guide, usage examples, troubleshooting
- Performance metrics and resource usage
### Docker Configuration
- **Simplified docker-compose.yml**: Removed text2vec-transformers service and modules
- **Weaviate-only deployment**: Manual vectorization, no auto-vectorization modules needed
- **Environment cleanup**: Removed `ENABLE_MODULES` and `DEFAULT_VECTORIZER_MODULE`
### Database Stats (Current)
- **Chunk_v2**: 5,355 chunks with 1024-dim vectors (GPU embedder)
- **Summary_v2**: Hierarchical summaries with GPU vectors
- **Work**: 18 philosophical works
- **Thought**: 104 thoughts
- **Conversation**: 12 conversations with 380 messages
### Git Commits
- feat: Complete GPU embedder migration for ingestion pipeline
- test: Add Puppeteer tests for chat and memories
- refactor: Remove Docker text2vec-transformers service
- chore: Clean up temporary test files
- docs: Reorganize documentation and rewrite README
- docs: Remove obsolete documentation files
---
## 2026-01-08 - Chunking Optimization & Vectorization
### Chunking Improvements