linear-coding-agent/CHANGELOG.md

Changelog - Library RAG Project

2026-01-08 - Chunking Optimization & Vectorization

Chunking Improvements

• Strict chunk size limits: Max 1000 words (down from 1500-2000)
• Overlap implementation: 100-word overlap between consecutive chunks
• Triple fallback system: Ensures robust chunking even on LLM failures
• New module: llm_chunker_improved.py with overlap functionality

Re-chunking Results

• Identified 31 oversized chunks (>2000 tokens, max 7,158)
• Split into 92 optimally-sized chunks
• Result: 0 chunks > 2000 tokens (100% within BGE-M3 limits)
• Preserved all metadata during split (workTitle, workAuthor, sectionPath, orderIndex)

Vectorization

• Created manual vectorization system for Chunk_v2 (no vectorizer configured)
• Successfully vectorized 92 new chunks via text2vec-transformers API
• Result: 5,304/5,304 chunks with vectors (100% coverage)

Docker Configuration

• Exposed text2vec-transformers port (8090:8080) for external vectorization
• Added cluster configuration to fix "No private IP address found" error
• Increased WORKER_TIMEOUT to 600s for very large chunks

Search Quality

• Created comprehensive test suite (10_test_search_quality.py)
• Tests: distribution, overlap detection, semantic search (4 queries)
• Search now uses near_vector() with manual query vectorization
• Issue identified: Collected papers dominates results (95.8% of chunks)

Database Stats (Post-Optimization)

• Total chunks: 5,304
• Average size: 289 tokens (optimal for BGE-M3)
• Distribution: 84.6% < 500 tokens, 11.5% 500-1000, 3.0% 1000-1500
• Works: 8 (Collected papers: 5,080 chunks, Mind Design III: 61, Platon Ménon: 56, etc.)

---

2025-01 - Weaviate v2 Migration & GPU Integration

Phase 1-3: Schema Migration (Complete)

• Migrated from Chunk/Summary/Document to Chunk_v2/Summary_v2/Work
• Removed nested document object, added direct properties (workTitle, workAuthor, year, language)
• Work collection with sourceId for documents
• Fixed 114 summaries missing properties
• Deleted vL-jepa chunks (17), fixed null workTitles

Phase 4: Memory System (Complete)

• Added Thought/Message/Conversation collections to Weaviate
• 9 MCP tools for memory management (add_thought, search_thoughts, etc.)
• GPU embeddings integration (BAAI/bge-m3, RTX 4070)
• Data: 102 Thoughts, 377 Messages, 12 Conversations

Phase 5: Backend Integration (Complete)

• Integrated GPU embedder into Flask app (singleton pattern)
• All search routes now use manual vectorization with near_vector()
• Updated all routes: simple_search, hierarchical_search, summary_only_search, rag_search
• Fixed Work → Chunk/Summary property mapping (v2 schema)

Phase 6-7: Testing & Optimization

• Comprehensive testing of search routes
• MCP tools validation
• Performance optimization with GPU embeddings
• Documentation updates (README.md, CLAUDE.md)

Phase 8: Documentation Cleanup

• Consolidated all phase documentation
• Updated README with Memory MCP tools section
• Cleaned up temporary files and scripts

---

Archive Structure

archive/
├── migration_scripts/        # Migration & optimization scripts (01-11)
│   ├── 01_migrate_document_to_work.py
│   ├── 02_create_schema_v2.py
│   ├── 03_migrate_chunks_v2.py
│   ├── 04_migrate_summaries_v2.py
│   ├── 05_validate_migration.py
│   ├── 07_cleanup.py
│   ├── 08_fix_summaries_properties.py
│   ├── 09_rechunk_oversized.py
│   ├── 10_test_search_quality.py
│   ├── 11_vectorize_missing_chunks.py
│   └── old_scripts/          # ChromaDB migration scripts
├── migration_docs/           # Detailed migration documentation
│   ├── PLAN_MIGRATION_V2_SANS_DOCUMENT.md
│   ├── PHASE5_BACKEND_INTEGRATION.md
│   └── WEAVIATE_RETRIEVAL_ARCHITECTURE.md
├── documentation/            # Phase summaries
│   ├── PHASE_0_PYTORCH_CUDA.md
│   ├── PHASE_2_MIGRATION_SUMMARY.md
│   ├── PHASE_3_CONVERSATIONS_SUMMARY.md
│   ├── PHASE_4_MIGRATION_CHROMADB.md
│   ├── PHASE_5_MCP_TOOLS.md
│   ├── PHASE_6_TESTS_OPTIMISATION.md
│   ├── PHASE_7_INTEGRATION_BACKEND.md
│   ├── PHASE_8_DOCUMENTATION_CLEANUP.md
│   └── MIGRATION_README.md
└── backups/                  # Pre-migration data backups
    └── pre_migration_20260108_152033/

---

Technology Stack

Vector Database: Weaviate 1.34.4 with BAAI/bge-m3 embeddings (1024-dim) Embedder: PyTorch 2.6.0+cu124, GPU RTX 4070 Backend: Flask 3.0 with Server-Sent Events MCP Integration: 9 memory tools + 6 RAG tools for Claude Desktop OCR: Mistral OCR API LLM: Ollama (local) or Mistral API

---

Known Issues

1. Chunk_v2 has no vectorizer: All new chunks require manual vectorization via 11_vectorize_missing_chunks.py
2. Data imbalance: Collected papers represents 95.8% of chunks, dominating search results
3. Mind Design III underrepresented: Only 61 chunks (1.2%) vs 5,080 for Collected papers

Recommendations

1. Add more diverse works to balance corpus
2. Consider re-ranking with per-work boosting for diversity
3. Recreate Chunk_v2 with text2vec-transformers vectorizer for auto-vectorization (requires full data reload)

---

For detailed implementation notes, see .claude/CLAUDE.md and archive/ directories.