Skip to content

CHANGELOG.md

Latest commit

 

History

History
237 lines (200 loc) · 15.8 KB

CHANGELOG.md

File metadata and controls

237 lines (200 loc) · 15.8 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

Changed

  • POSTGRES_USER and POSTGRES_PASSWORD are now required; the compose file and database config no longer provide hardcoded default credentials
  • Removed compose file parsing from database config resolution; credentials must come from environment variables, DATABASE_URL, or CLI arguments
  • Bump runtime dependencies: docling 2.113.0, pgvector 0.5.0, structlog 26.1.0, tqdm 4.68.4, xxhash 3.8.1, ibm-watsonx-ai 1.5.14
  • Bump dev dependencies: datamodel-code-generator 0.68.1, ipython 9.15.0, pyright 1.1.411, pytest 9.1.1, ruff 0.15.21, tox 4.56.4
  • Bump CI actions: actions/checkout v7, actions/setup-python v6 (latest), astral-sh/setup-uv v8.3.2, codeql-action v4.37.0
  • Bump build system: uv_build 0.11.28
  • Added lint, format, typecheck, and clean targets to Makefile
  • Narrowed broad except Exception clauses in database.py to specific types (psycopg.Error, psycopg.errors.UndefinedTable, yaml.YAMLError) and removed unnecessary try/except blocks
  • Replaced subprocess.run() CLI integration tests with Typer CliRunner for in-process invocation — faster execution, no S603/S607 suppressions needed
  • Converted database layer from async psycopg (AsyncConnection) to sync psycopg (Connection), eliminating nested event loop issues with single-threaded batch processing
  • Removed pytest-asyncio and greenlet dev dependencies
  • Bumped ibm-watsonx-ai minimum version to 1.5.12
  • Constrained Renovate to keep typer at <0.22.0 while docling-slim pins it; prevents recurring broken update PRs (#33)

Added

  • GitHub Actions CI workflow (lint + PostgreSQL-backed test suite)
  • OpenSSF Scorecard workflow for security scoring
  • CodeRabbit configuration for automated code review
  • Renovate configuration for automated dependency updates
  • Pull request template
  • AGENTS.md repository knowledge base for AI agents
  • CONTRIBUTING.md development guide
  • SECURITY.md vulnerability disclosure policy

Fixed

  • Deferred heavy imports (transformers, docling, torch) in utils.py, chunks.py, and ingest.py — CLI startup drops from ~3.7s to ~0.2s for commands that don't need ML/document processing
  • Fixed structlog config corruption after single-threaded load — worker logging setup now saves and restores the main process config, even when a batch raises
  • Fixed document path mismatch in up-to-date check — skip-check now uses relative paths matching DB storage, preventing unnecessary re-processing
  • Added per-document savepoint isolation in batch loading — one failed document no longer aborts the entire batch transaction
  • Fixed test_load_documents_force_parameter — test now uses correct fixture layout and exercises both force=False (skip) and force=True (reload) paths
  • Fixed test_load_documents_parameter_validation — removed broad exception swallowing that masked real failures
  • Fixed test_database_functions_interface — removed vacuous assertion (check_database_status returns None by contract)

Removed

  • Removed unimplemented embedding provider stubs (Watson/Slate, SentenceTransformer, NoInstruct, E5) — only Granite was functional

[0.4.4] - 2026-03-16

Fixed

  • Pipeline command now correctly finds source and chunks files
    • Fixed glob pattern bug where **/*.json was being appended with /source.json, creating invalid pattern **/*.json/source.json
    • Changed patterns to ** (directory pattern) since functions automatically append filenames
    • Affects pipeline command only; individual chunk, embed, load commands were already correct
  • Dependency resolution for PyTorch and torchvision
    • Added explicit torchvision>=0.23.0 dependency to ensure it resolves from pytorch-cpu index
    • Fixes fresh install failures where torchvision resolved from PyPI instead of pytorch-cpu
    • Both packages now correctly use +cpu suffix on Linux/Windows
  • Database validation in db-status command
    • Added check for models table existence before querying and provides helpful error message instead of cryptic SQL error if it is missing.

0.4.3 - 2025-01-09

Fixed

  • Embedding deadlock on ARM Linux caused by forking after PyTorch initialization

Added

  • --workers option for docs2db embed command to control parallelism
    • Use --workers 1 for single-threaded mode (avoids fork issues on ARM)
    • Default behavior unchanged (2 workers)

0.4.2 - 2025-11-12

Changed

  • Enhanced PyPI package metadata in pyproject.toml:
    • Added keywords for better discoverability: rag, document-processing, embeddings, docling, contextual-retrieval, semantic-search, vector-database, pgvector, postgresql, llm
    • Added PyPI classifiers for proper categorization (Development Status, License, Python version, Topics)
    • Added explicit Apache-2.0 license declaration
    • Added project URLs (Homepage, Documentation, Repository, Issues, Changelog)

0.4.1 - 2025-11-12

Added

  • New config command to configure RAG settings in the database
    • Settings include: refinement_prompt, refinement, reranking, similarity_threshold, max_chunks, max_tokens_in_context, refinement_questions_count
    • All settings accept string values: booleans as true/false/None, numbers as strings, or "None" to clear
    • Boolean values accept: true, false, t, f, yes, no, y, n, 1, 0
    • Settings are stored in new rag_settings database table
    • Intended for use with docs2db-api, which reads and applies these settings with appropriate priority
  • db-status command now displays RAG settings from the database
  • New rag_settings table in database schema (singleton table with constraint id = 1)
  • New configure_rag_settings() function in database.py for programmatic configuration

0.4.0 - 2025-11-11

Changed

  • BREAKING: Model identification consolidated - model_name and model_id merged into single model term throughout codebase
  • BREAKING: Embedding metadata structure flattened:
    • Old: metadata.model.model_id, metadata.embedding.created_at
    • New: metadata.model, metadata.embedded_at
    • Existing embedding files automatically detected as stale and regenerated
  • Pattern handling standardized across all commands (chunk, embed, load, audit):
    • Commands now accept directory patterns without requiring wildcards
    • Automatically appends /source.json or /chunks.json to patterns
    • Examples: --pattern "docs/subdir" (exact path) or --pattern "external/**" (glob)
    • Old strict wildcard validation removed
  • db-status now displays document paths instead of filenames in "Recent document activity" section for better identification in subdirectory structure
  • Audit command now shows full relative paths from content directory instead of just directory names for better error reporting

Added

  • Environment variable support for debug logging: DEBUG=1 or LOG_LEVEL=DEBUG
  • Improved logging: "Creating LLM session" and batch processing messages moved to DEBUG level
  • README improvements:
    • New "Overview" section explaining processing time, stages, idempotency, and storage
    • Updated "Requirements" section clarifying Ollama's role (optional for contextual chunking)
    • Library usage examples for ingest_file() and ingest_from_content() functions
    • "Serving Your Database" section with docs2db-api integration examples

Fixed

  • BatchProcessor now accepts sorted lists instead of iterators to ensure file processing order across restarts
  • Removed misleading "LLM session created successfully" log message that appeared before actual connection attempts

Internal

  • Embedding and EmbeddingProvider classes simplified with explicit parameters instead of config dictionaries
  • is_embedding_stale() signature updated to match new metadata structure
  • All worker functions updated to use new model parameter naming
  • Test fixtures updated to reflect new metadata format
  • CONTRIBUTING.md updated with correct uv sync --extra watsonx command

0.3.1 - 2025-11-06

Changed

  • BREAKING: load command now uses directory-based patterns (must end with ** or *) instead of file patterns
    • Old: --pattern "**/source.json" or --pattern "**/*.json"
    • New: --pattern "**" or --pattern "external/**"
    • Patterns are automatically converted to match source.json files internally
  • load command default pattern changed from **/*.json to ** (directory-based)
  • Pattern validation now checks for proper glob wildcard structure instead of file extensions

0.3.0 - 2025-11-06

Added

  • document_needs_update() function for external API callers to check if documents need updating without knowing internal storage details
  • Enhanced audit reporting with separate tracking for stale chunks and stale embeddings
  • Zero-chunk document detection in audit (documents that legitimately have no chunks)
  • Orphan directory detection in audit (directories without source.json)
  • Audit command now accepts directory patterns for targeted auditing (e.g., --pattern "external/**" or --pattern "allowed_kcs/*")

Changed

  • BREAKING: File storage now uses subdirectory-based structure where each document gets its own directory:
    • Old: doc_name.json, doc_name.chunks.json, doc_name.gran.json
    • New: doc_name/source.json, doc_name/chunks.json, doc_name/gran.json
  • BREAKING: ingest_from_content() now requires stream_name parameter (e.g., "document.html") for proper format detection
  • BREAKING: ingest_file() and ingest_from_content() now accept directory paths instead of file paths (e.g., content_dir/doc_name not content_dir/doc_name/source.json)
  • BREAKING: Default glob patterns changed to **/source.json (for chunking) and **/chunks.json (for embedding)
  • Audit command default pattern changed from **/*.json to ** (directory-based pattern)
  • Audit now validates that patterns don't include file extensions (must match directories only)
  • Audit results now show separate counts for chunks, embeddings, and zero-chunk documents
  • Moved routine document analysis logging to DEBUG level; summarization events still logged at INFO level
  • performance improvement: LLM API clients (WatsonX, OpenAI) are now reused across documents in the same batch instead of creating new clients for each document
  • LLMSession.__init__() no longer takes doc_text parameter; call set_document(doc_text) after initialization
  • Added LLM_PROVIDER setting (defaults to "openai") to explicitly choose between OpenAI-compatible and WatsonX providers; provider selection now respects Pydantic Settings precedence (CLI/env > .env file > defaults)
  • Added --llm-provider, --openai-url, --watsonx-url, --context-model, and --context-limit CLI flags to chunk and pipeline commands to explicitly control LLM provider settings
  • Provider is inferred from URL flags if not explicitly specified (e.g., --watsonx-url → watsonx, --openai-url → openai)
  • Removed validation that prevented both --openai-url and --watsonx-url from being set; provider selection now explicit via --llm-provider or inferred from flags
  • Settings access is now centralized in public API functions only (generate_chunks(), generate_embeddings(), load_documents(), perform_audit()); all internal functions require explicit parameters and validate inputs strictly (no global settings access)

Fixed

  • Addressed WatsonX rate limiting issues by reusing API clients across documents in worker batches
  • Fixed race condition causing duplicate key errors when multiple workers insert the same embedding model concurrently (now uses INSERT ... ON CONFLICT DO NOTHING)
  • Fixed db-destroy command to correctly parse project name from postgres-compose.yml instead of hardcoding "docs2db" (was failing to remove volumes for projects with different names)
  • Fixed ingest_file() to enforce .json extension regardless of caller-provided path, ensuring downstream chunking/embedding tools always find correct files

0.2.1 - 2025-11-03

Changed

  • Improved warning messages: metadata parsing errors now include the file path for easier debugging
  • Moved WatsonX dependencies from dependency-groups to optional-dependencies for proper PyPI extra support

Fixed

  • WatsonX extra (docs2db[watsonx]) now installable from PyPI

0.2.0 - 2025-11-03

Added

  • Document ingestion using Docling (ingest command) for PDF, DOCX, PPTX, and more
  • Contextual chunking with LLM support (Ollama via OpenAI-compatible API, OpenAI, WatsonX)
  • BM25 full-text search with PostgreSQL tsvector and GIN indexing for hybrid search
  • Database lifecycle commands: db-start, db-stop, db-destroy, db-logs
  • db-restore command for loading SQL dumps
  • pipeline command for end-to-end workflow (ingest → chunk → embed → load → dump)
  • Multi-tier PostgreSQL configuration precedence (CLI > Env Vars > DATABASE_URL > Compose > Defaults)
  • Metadata arguments for pipeline and load commands (--username, --title, --description, --note)
  • Metadata tracking for ingested documents and chunking operations
  • --skip-context flag to bypass LLM contextual chunking
  • --context-model and --openai-url/--watsonx-url flags for LLM provider configuration
  • Persistent LLM sessions with KV cache reuse for improved performance
  • Memory-efficient in-memory document ingestion
  • Comprehensive database configuration tests
  • Pre-commit hooks for code quality enforcement (ruff, pyright, gitleaks)

Changed

  • Default content directory changed from content/ to docs2db_content/
  • Commands now use settings defaults: load, audit, and pipeline fall back to settings.content_base_dir and settings.embedding_model
  • Simplified database lifecycle: removed profile parameter (always uses "prod")
  • Improved error messages: database connection errors now suggest docs2db db-start instead of make db-up
  • Reduced logging verbosity: suppressed verbose docling library output, moved per-file conversion messages to DEBUG
  • Updated .gitignore to exclude generated artifacts (docs2db_content/, ragdb_dump.sql)
  • Improved CLI argument handling with explicit None checks and user-friendly error messages

Fixed

  • Typer required argument handling now provides clear error messages instead of TypeErrors
  • Removed duplicate error logging in database operations
  • Updated compose file password to match default settings (postgres)
  • Corrected ingest command docstring to show docs2db_content/ directory

0.1.0 - 2025-09-29

Added

  • Initial implementation of docs2db
  • Basic document chunking using HybridChunker from docling_core
  • Embedding generation with Granite 30M English model
  • PostgreSQL database with pgvector for vector similarity search
  • CLI commands: chunk, embed, load, audit, db-status, db-dump, cleanup-workers
  • Multiprocessing support for chunking and embedding operations
  • Comprehensive test suite
  • Development tooling: Makefile, Docker Compose setup for PostgreSQL