Enhance logging configuration and functionality

- Improved ColoredFormatter to support microseconds in timestamps.
- Enhanced MultiProcessingLog to use a shared lock and prefix for log filenames.
- Updated ColoredLogger to handle new configuration options for log formatting and file handling.
- Added comprehensive tests for log filename configuration from various sources (environment variables, YAML, CLI).
- Refactored tests to remove unused imports and improve clarity.
- Ensured proper handling of log file creation and rotation settings.
- Added priority tests for log filename and date format configurations.

Author: vertok

.github/workflows/prismalog_ci.yml

@@ -32,6 +32,15 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
 
+      # Step 2.5: Cache dependencies
+      - name: Cache pip dependencies
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py', '**/pyproject.toml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+
       # Step 3: Install dependencies (Install dev and docs extras)
       - name: Install dependencies
         run: |
@@ -64,9 +73,9 @@ jobs:
       - name: Test logging performance
         if: matrix.python-version == '3.10'
         run: |
-          python benchmark/performance_test_multiprocessing.py > reports/performance_py${{ matrix.python-version }}.txt
+          python benchmark/performance_test.py -p 3 -t 1 > reports/performance_py${{ matrix.python-version }}.txt
           MSGS_PER_SEC=$(grep -oP '(?<=Messages per second: )[0-9.]+' reports/performance_py${{ matrix.python-version }}.txt || echo "0")
-          echo "## Performance: $MSGS_PER_SEC messages/sec" >> $GITHUB_STEP_SUMMARY
+          echo "## Performance (3p x 1t): $MSGS_PER_SEC messages/sec" >> $GITHUB_STEP_SUMMARY # Updated summary text
 
       # Step 7: Run comprehensive performance comparison (only for Python 3.10 to avoid long run times)
       - name: Run comprehensive performance comparison

benchmark/README.md

@@ -8,10 +8,33 @@ Comprehensive benchmarks were conducted to evaluate the performance characterist
 
 | Test Type        | Processes               | Threads        |
 |------------------|-------------------------|----------------|
-| Multiprocessing  | 4 processes             | 1 thread       |
-| Multithreading   | 1 processe              | 4 threads      |
-| Mixed Mode       | 3 processes             | 2 threads      |
-| Standard Logging | 1 processe              | 4 threads      |
+| Multiprocessing  | 3 processes             | 1 thread       |
+| Multithreading   | 1 processe              | 3 threads      |
+| Mixed Mode       | 2 processes             | 2 threads      |
+| Standard Logging | 1 processe              | 3 threads      |
+
+### Key Observations
+
+1.  **Concurrency Models**:
+    *   The highest throughput was observed in the multiprocessing model, with nearly 27,000 messages processed per second.
+    *   A good balance of throughput and resource utilization was achieved in the mixed mode (processes with threads).
+    *   The multithreading model demonstrated consistent performance at around 10,000 messages per second.
+    *   The standard logging library processed approximately 6,500 messages per second.
+
+2.  **Latency**:
+    *   The lowest per-message latency (0.07-0.08ms) was provided by multiprocessing.
+    *   Medium latency (0.17-0.21ms) was observed in mixed mode.
+    *   Higher latency (0.26-0.42ms) was observed in thread-based approaches.
+    *   Standard logging exhibited the highest latency (0.40-0.46ms) across all log levels.
+
+3.  **Resource Usage**:
+    *   Minimal memory consumption (0.20-0.49MB) was observed across prismalog approaches.
+    *   Standard logging showed the lowest memory increase (0.04MB) but with slower performance.
+    *   Log file sizes remained compact (0.61-1.13MB) across all approaches.
+
+4.  **Timestamp Formatting Impact**:
+    *   **Crucially, the choice of timestamp format significantly impacts performance.** Using `%(created)f` (which logs a raw numeric timestamp) can achieve substantially higher throughput (observed up to **~35,000 msgs/sec** in testing) compared to using `%(asctime)s` (which formats the timestamp into a human-readable string, observed maxing out around **~25,000 msgs/sec**).
+    *   While `%(created)f` requires post-processing to convert timestamps for readability, it drastically reduces logging overhead.
 
 ### Results Summary
 
@@ -41,26 +64,59 @@ Comprehensive benchmarks were conducted to evaluate the performance characterist
    - Standard logging showed the lowest memory increase (0.04MB) but with slower performance.
    - Log file sizes remained compact (0.61-1.13MB) across all approaches.
 
+### Performance Benchmarking
+
+The `benchmark/performance_test.py` script allows you to measure logging performance under different concurrency models.
+
+**Arguments:**
+
+*   `-p N`, `--processes N`: Use N worker processes (default: 2).
+*   `-t M`, `--threads M`: Use M worker threads per process (default: 2).
+
+**Examples:**
+
+*   **Multiprocessing Test (e.g., 3 processes, 1 thread each):**
+    ```bash
+    python benchmark/performance_test.py -p 3 -t 1
+    ```
+
+*   **Multithreading Test (e.g., 1 process, 3 threads):**
+    ```bash
+    python benchmark/performance_test.py -p 1 -t 3
+    ```
+
+*   **Mixed Concurrency Test (e.g., 2 processes, 2 threads each):**
+    ```bash
+    python benchmark/performance_test.py -p 2 -t 2
+    # Or simply run with defaults:
+    # python benchmark/performance_test.py
+    ```
+
+The script also accepts standard `prismalog` arguments like `--log-level`, `--log-format`, etc., to configure the logger during the benchmark. Rotation is automatically disabled during the benchmark run for consistent results.
+
 ### Feature Advantages Over Standard Logging
 
 While performance benchmarks provide valuable insights, several important features are offered by `prismalog` that are not available in the standard logging library:
 
-1. **Color-coded Console Output**:
+1.  **Process-Safe & Thread-Safe File Handling**:
+    *   The included file handlers (`MultiProcessingLog`) are specifically designed to handle concurrent writes from multiple processes and threads safely, preventing log corruption or race conditions. Standard `RotatingFileHandler` is not inherently process-safe without external locking mechanisms.
+
+2. **Color-coded Console Output**:
    - Syntax highlighting for log messages is applied automatically based on their severity level.
    - Customizable color schemes are supported for different environments.
    - Readability is improved by visually distinguishing between different message types.
 
-2. **Special Critical Message Handling**:
+3. **Special Critical Message Handling**:
    - Application termination on critical errors is optionally supported.
    - Configurable callbacks for critical message events are provided.
    - Stack trace preservation is ensured for critical failures.
 
-3. **Advanced Configuration**:
+4. **Advanced Configuration**:
    - Environment variable support is included, with sensible defaults and multiple fallback patterns.
    - Command-line argument integration is supported, with automatic help generation.
    - Configuration file support (YAML) is provided, with automatic detection.
 
-4. **Developer Experience Enhancements**:
+5. **Developer Experience Enhancements**:
    - A simplified API is offered for common logging patterns.
    - Context managers are provided for temporary logging level changes.
    - Convenient decorators are included for function entry/exit logging.

benchmark/performance_compare.py

@@ -10,9 +10,8 @@
 import os
 import re
 import subprocess
-from collections.abc import Mapping  # Add this import
 from datetime import datetime
-from typing import Any, Dict, List, Optional, Tuple, TypedDict, Union, cast
+from typing import Any, Dict, List, Optional, TypedDict, Union
 
 
 # Define explicit types for the nested dictionaries
@@ -31,6 +30,8 @@ class RelativePerfDict(TypedDict):
 from prismalog.argparser import extract_logging_args, get_argument_parser
 from prismalog.log import LoggingConfig, get_logger
 
+# Set up the logging configuration
+os.environ["LOG_FILENAME"] = "performance_compare"
 # Create parser with standard logging arguments
 parser = get_argument_parser(description="prismalog Performance Comparison")
 
@@ -76,36 +77,36 @@ def extract_metrics(output: str) -> Dict[str, Union[str, float]]:
 
     # Extract log file size
     log_size_match = re.search(r"• Size: ([\d.]+) MB", output)
+    log_size_match = re.search(r"• Size: ([\d.]+) MB", output)
     if log_size_match:
         metrics["log_size_mb"] = float(log_size_match.group(1))
 
     return metrics
 
 
-def run_test(script_name: str) -> Optional[str]:
-    """Run a performance test script and capture output while showing logs in real-time."""
+def run_test(script_name: str, args: List[str] = []) -> Optional[str]:
+    """Run a performance test script with arguments and capture output."""
+    args_str = " ".join(args)
     print(f"\n{'-'*60}")
-    print(f"Running {script_name}...")
+    print(f"Running {script_name} {args_str}...")
     print(f"{'-'*60}")
 
-    # Check if the script exists
     script_path = os.path.join(os.path.dirname(__file__), script_name)
     if not os.path.exists(script_path):
         logger.warning(f"Script not found: {script_path}")
         return None
 
     try:
-        # Run the script and stream output to console while also capturing it
+        command = ["python", script_path] + args  # Combine script and args
         output = []
         with subprocess.Popen(
-            ["python", script_path],
+            command,  # Use the combined command
             stdout=subprocess.PIPE,
             stderr=subprocess.STDOUT,
             text=True,
-            bufsize=1,  # Line buffered
+            bufsize=1,
         ) as process:
-            # Read and display output in real-time
-            if process.stdout is not None:  # Add this check
+            if process.stdout is not None:
                 for line in iter(process.stdout.readline, ""):
                     if not line:
                         break
@@ -117,17 +118,17 @@ def run_test(script_name: str) -> Optional[str]:
             return_code = process.wait()
 
         if return_code != 0:
-            logger.error(f"{script_name} exited with code {return_code}")
+            logger.error(f"{script_name} {args_str} exited with code {return_code}")
             return None
 
         print(f"{'-'*60}")
-        print(f"Completed {script_name}")
+        print(f"Completed {script_name} {args_str}")
         print(f"{'-'*60}\n")
 
         return "".join(output)
 
     except Exception as e:
-        logger.error(f"Error running {script_name}: {e}")
+        logger.error(f"Error running {script_name} {args_str}: {e}")
         return None
 
 
@@ -165,34 +166,50 @@ def save_benchmark_results(results: Dict[str, Any], test_type: str) -> str:
 
 
 def main() -> None:
-    """Run all performance tests and compare results."""
+    """Run performance tests for different concurrency models and compare results."""
     print(f"\n{'='*80}")
     print(f"LOGGING PERFORMANCE COMPARISON - {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
     print(f"{'='*80}\n")
 
-    # Track available test results
     test_metrics = {}
 
-    # Run performance tests
     try:
-        # Run all tests and collect metrics
-        test_files = [
-            ("performance_test_multiprocessing.py", "multiproc", "Multiprocessing"),
-            ("performance_test_threading.py", "threading", "Multithreading"),
-            ("performance_test_mixed.py", "mixed", "Mixed Mode"),
-            ("standard_logging_benchmark.py", "std", "Standard Logging"),
+        # Define test configurations for the performance compare script
+        test_configs = [
+            # Config: (arguments_list, result_key, display_name)
+            (["-p", "3", "-t", "1"], "multiproc", "Multiprocessing (3p x 1t)"),
+            (["-p", "1", "-t", "3"], "threading", "Multithreading (1p x 3t)"),
+            (["-p", "2", "-t", "2"], "mixed", "Mixed Mode (2p x 2t)"),
         ]
 
-        for script_name, key, display_name in test_files:
-            output = run_test(script_name)
+        # Script to run for prismalog tests
+        prismalog_script = "performance_test.py"
+
+        for args_list, key, display_name in test_configs:
+            # Add the specific log filename for this test run
+            subprocess_args = args_list + ["--log-filename", key]  # Use the 'key' as the prefix
+
+            # Run the test with specific arguments including the log filename
+            output = run_test(prismalog_script, subprocess_args)  # Pass the modified args
             if output:
                 metrics = extract_metrics(output)
-                metrics["test"] = display_name
+                metrics["test"] = display_name  # Use the descriptive name
                 test_metrics[key] = metrics
                 save_benchmark_results(metrics, f"test_{key}")
 
+        # Optionally, give the standard logging test its own prefix
+        std_script = "standard_logging_benchmark.py"
+        std_output = run_test(std_script)  # Pass the args
+        if std_output:
+            std_metrics = extract_metrics(std_output)
+            std_metrics["test"] = "Standard Logging"
+            test_metrics["std"] = std_metrics
+            save_benchmark_results(std_metrics, "test_std")
+
         if not test_metrics:
-            logger.error("No test metrics collected. Please ensure at least one test script is available.")
+            logger.error(
+                "No test metrics collected. Please ensure benchmark scripts are available and run successfully."
+            )
             return
 
         # Define table structure