-
Notifications
You must be signed in to change notification settings - Fork 3
feat(qdrant): epic 4 - complete qdrant semantic cache implementation (tasks #61-77) #24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 18 commits
Commits
Show all changes
41 commits
Select commit
Hold shift + click to select a range
5b1ff1a
feat(qdrant): implement Qdrant client connection manager
claude a881e1a
feat(qdrant): implement Qdrant repository base class
claude 8f241d5
feat(qdrant): implement collection initialization and management
claude 604f52d
feat(qdrant): define collection schema and configuration models
claude 4570ab9
feat(qdrant): implement point models for vector storage
claude 0233dea
feat(qdrant): implement vector storage operations
claude 5cdf4d5
feat(qdrant): implement vector similarity search
claude c8b24d8
feat(similarity): implement similarity score calculator
claude 3935394
feat(similarity): implement vector normalization utilities
claude 21ab124
feat(qdrant): implement filter builder for advanced queries
claude 9da8461
feat(qdrant): implement advanced batch upload operations
claude 345f643
feat(qdrant): implement delete operations for points
claude ebf9778
feat(qdrant): implement point update operations
claude 9a3695c
feat(qdrant): implement metadata handling utilities
claude 6c4ae64
feat(qdrant): implement pagination for large result sets
claude 1568cc0
feat(qdrant): implement comprehensive health check service
claude 12b4d20
feat(qdrant): implement metrics collection models
claude 9b23d56
style: apply black formatting to qdrant implementation
claude d26e1c0
style: fix isort import ordering in qdrant repository
claude fbf2833
style: remove unused imports for flake8 compliance
claude 395c28d
fix(types): resolve mypy type errors across qdrant implementation
claude 13c5804
feat(qdrant): implement comprehensive error handling system
claude 228e8ff
feat(qdrant): implement advanced connection pooling system
claude 68b2c2c
feat(qdrant): implement comprehensive index optimization system
claude 3866fdb
test(qdrant): implement comprehensive unit test suite
claude 26f8a56
feat(qdrant): implement collection backup and restore system
claude 71d2102
style: fix isort import ordering in unit tests
claude 4cc3f27
style: fix isort alphabetical ordering in test_vector_normalizer
claude 0f80235
style: fix flake8 line length errors in qdrant_health
claude 7535193
style: apply black formatting to qdrant_health
claude 6f211c9
fix(types): resolve mypy type errors in qdrant modules
claude 3172db2
fix(tests): add convenience aliases and functions for test compatibility
claude 3440ae7
fix(tests): resolve failing unit tests
claude 96a8733
fix(tests): adjust score interpretation thresholds
claude 6981f68
feat(tests): implement Qdrant integration tests and fix CI workflow
claude e6c4387
feat(benchmarks): implement comprehensive Qdrant performance benchmarks
claude a96ecbe
feat(similarity): implement semantic similarity threshold tuning
claude 44337fd
style: fix flake8 linting errors in benchmarks and threshold tuner
claude bcd6d94
fix(types): resolve mypy type errors in threshold tuner
claude f7cd741
fix: address PR review feedback from Gemini Code Assist
claude 385aa8c
fix(types): resolve mypy type error in qdrant_point.py
claude File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| """ | ||
| Qdrant client connection manager. | ||
|
|
||
| Sandi Metz Principles: | ||
| - Single Responsibility: Qdrant connection management | ||
| - Small methods: Each operation isolated | ||
| - Dependency Injection: Configuration injected | ||
| """ | ||
|
|
||
| from typing import Optional | ||
|
|
||
| from qdrant_client import AsyncQdrantClient | ||
| from qdrant_client.http.exceptions import UnexpectedResponse | ||
|
|
||
| from app.config import config | ||
| from app.utils.logger import get_logger | ||
|
|
||
| logger = get_logger(__name__) | ||
|
|
||
|
|
||
| async def create_qdrant_client() -> AsyncQdrantClient: | ||
| """ | ||
| Create Qdrant async client connection. | ||
|
|
||
| Returns: | ||
| Qdrant async client | ||
|
|
||
| Raises: | ||
| ConnectionError: If connection fails | ||
| """ | ||
| try: | ||
| client = AsyncQdrantClient( | ||
| host=config.qdrant_host, | ||
| port=config.qdrant_port, | ||
| timeout=30.0, | ||
| ) | ||
|
|
||
| # Test connection | ||
| await client.get_collections() | ||
|
|
||
| logger.info( | ||
| "Qdrant client connected", | ||
| host=config.qdrant_host, | ||
| port=config.qdrant_port, | ||
| ) | ||
|
|
||
| return client | ||
|
|
||
| except Exception as e: | ||
| logger.error("Qdrant connection failed", error=str(e)) | ||
| raise ConnectionError(f"Failed to connect to Qdrant: {e}") | ||
|
|
||
|
|
||
| class QdrantConnectionManager: | ||
| """ | ||
| Manages Qdrant client connection lifecycle. | ||
|
|
||
| Handles connection pooling and health checks. | ||
| """ | ||
|
|
||
| def __init__(self): | ||
| """Initialize connection manager.""" | ||
| self._client: Optional[AsyncQdrantClient] = None | ||
|
|
||
| async def get_client(self) -> AsyncQdrantClient: | ||
| """ | ||
| Get or create Qdrant client. | ||
|
|
||
| Returns: | ||
| Qdrant async client | ||
|
|
||
| Raises: | ||
| ConnectionError: If connection fails | ||
| """ | ||
| if self._client is None: | ||
| self._client = await create_qdrant_client() | ||
| return self._client | ||
|
|
||
| async def close(self) -> None: | ||
| """Close Qdrant client connection.""" | ||
| if self._client is not None: | ||
| try: | ||
| await self._client.close() | ||
| logger.info("Qdrant client closed") | ||
| except Exception as e: | ||
| logger.error("Failed to close Qdrant client", error=str(e)) | ||
| finally: | ||
| self._client = None | ||
|
|
||
| async def health_check(self) -> bool: | ||
| """ | ||
| Check Qdrant server health. | ||
|
|
||
| Returns: | ||
| True if healthy, False otherwise | ||
| """ | ||
| try: | ||
| client = await self.get_client() | ||
| await client.get_collections() | ||
| return True | ||
| except Exception as e: | ||
| logger.error("Qdrant health check failed", error=str(e)) | ||
| return False | ||
|
|
||
| async def reconnect(self) -> bool: | ||
| """ | ||
| Reconnect to Qdrant server. | ||
|
|
||
| Returns: | ||
| True if reconnected successfully | ||
| """ | ||
| try: | ||
| await self.close() | ||
| self._client = await create_qdrant_client() | ||
| return True | ||
| except Exception as e: | ||
| logger.error("Qdrant reconnection failed", error=str(e)) | ||
| return False | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,167 @@ | ||
| """ | ||
| Qdrant collection initialization and management. | ||
|
|
||
| Sandi Metz Principles: | ||
| - Single Responsibility: Collection setup and validation | ||
| - Small methods: Each operation isolated | ||
| - Dependency Injection: Repository injected | ||
| """ | ||
|
|
||
| from typing import Optional | ||
|
|
||
| from qdrant_client.models import Distance | ||
|
|
||
| from app.repositories.qdrant_repository import QdrantRepository | ||
| from app.utils.logger import get_logger | ||
|
|
||
| logger = get_logger(__name__) | ||
|
|
||
|
|
||
| class QdrantCollectionManager: | ||
| """ | ||
| Manages Qdrant collection initialization. | ||
|
|
||
| Ensures collection exists and is properly configured. | ||
| """ | ||
|
|
||
| def __init__(self, repository: QdrantRepository): | ||
| """ | ||
| Initialize collection manager. | ||
|
|
||
| Args: | ||
| repository: Qdrant repository | ||
| """ | ||
| self._repository = repository | ||
|
|
||
| async def initialize( | ||
| self, distance: Distance = Distance.COSINE, recreate: bool = False | ||
| ) -> bool: | ||
| """ | ||
| Initialize collection for vector storage. | ||
|
|
||
| Args: | ||
| distance: Distance metric for similarity | ||
| recreate: Whether to recreate existing collection | ||
|
|
||
| Returns: | ||
| True if initialized successfully | ||
| """ | ||
| try: | ||
| if recreate: | ||
| await self._recreate_collection(distance) | ||
| return True | ||
|
|
||
| return await self._ensure_collection_exists(distance) | ||
|
|
||
| except Exception as e: | ||
| logger.error("Collection initialization failed", error=str(e)) | ||
| return False | ||
|
|
||
| async def _ensure_collection_exists(self, distance: Distance) -> bool: | ||
| """ | ||
| Ensure collection exists. | ||
|
|
||
| Args: | ||
| distance: Distance metric | ||
|
|
||
| Returns: | ||
| True if exists or created | ||
| """ | ||
| exists = await self._repository.collection_exists() | ||
|
|
||
| if exists: | ||
| logger.info("Collection verified") | ||
| return True | ||
|
|
||
| return await self._repository.create_collection(distance) | ||
|
|
||
| async def _recreate_collection(self, distance: Distance) -> bool: | ||
| """ | ||
| Recreate collection (delete and create). | ||
|
|
||
| Args: | ||
| distance: Distance metric | ||
|
|
||
| Returns: | ||
| True if recreated successfully | ||
| """ | ||
| logger.warning("Recreating collection - all data will be lost") | ||
|
|
||
| # Delete if exists | ||
| exists = await self._repository.collection_exists() | ||
| if exists: | ||
| await self._repository.delete_collection() | ||
|
|
||
| # Create new collection | ||
| return await self._repository.create_collection(distance) | ||
|
|
||
| async def validate_collection(self) -> dict[str, bool]: | ||
| """ | ||
| Validate collection configuration. | ||
|
|
||
| Returns: | ||
| Validation results dict | ||
| """ | ||
| results = { | ||
| "exists": False, | ||
| "accessible": False, | ||
| "configured": False, | ||
| } | ||
|
|
||
| try: | ||
| # Check existence | ||
| results["exists"] = await self._repository.collection_exists() | ||
| if not results["exists"]: | ||
| return results | ||
|
|
||
| # Check accessibility | ||
| results["accessible"] = await self._repository.ping() | ||
| if not results["accessible"]: | ||
| return results | ||
|
|
||
| # Check configuration | ||
| info = await self._repository.get_collection_info() | ||
| results["configured"] = info is not None | ||
|
|
||
| return results | ||
|
|
||
| except Exception as e: | ||
| logger.error("Collection validation failed", error=str(e)) | ||
| return results | ||
|
|
||
| async def get_status(self) -> Optional[dict]: | ||
| """ | ||
| Get collection status and statistics. | ||
|
|
||
| Returns: | ||
| Status dict if successful | ||
| """ | ||
| try: | ||
| validation = await self.validate_collection() | ||
| if not validation["exists"]: | ||
| return { | ||
| "status": "not_initialized", | ||
| "message": "Collection does not exist", | ||
| } | ||
|
|
||
| info = await self._repository.get_collection_info() | ||
| if not info: | ||
| return { | ||
| "status": "error", | ||
| "message": "Failed to get collection info", | ||
| } | ||
|
|
||
| return { | ||
| "status": "ready", | ||
| "vectors_count": info["vectors_count"], | ||
| "points_count": info["points_count"], | ||
| "collection_status": info["status"], | ||
| "config": info["config"], | ||
| } | ||
|
|
||
| except Exception as e: | ||
| logger.error("Get status failed", error=str(e)) | ||
| return { | ||
| "status": "error", | ||
| "message": str(e), | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
except Exceptionis too broad and can mask programming errors. It's better to catch specific exceptions related to connection issues. You've already importedUnexpectedResponse, which is a good one to catch specifically. You could also consider catching network-related exceptions fromhttpx(whichqdrant-clientuses) if you want to be more precise.