Skip to content

REFERENCES.md

Latest commit

 

History

History
204 lines (153 loc) · 10.1 KB

REFERENCES.md

File metadata and controls

204 lines (153 loc) · 10.1 KB
Raw

Research References

Performance Analysis & Benchmarking Methodology

Resilio v6.2 implements methodologies from:

Original Foundations (v6.0)

  • Jain, R. (1991) - Statistical methods for performance measurement
  • Welch, B. L. (1947) - Unequal variance t-test
  • Cohen, J. (1988) - Effect size interpretation
  • ISO/IEC 25010:2011 - Performance efficiency metrics
  • Barford & Crovella (1998) - Workload characterization
  • Gunther, N. J. (2007) - Queueing theory and capacity planning

New in v6.1 & v6.2

  • Mann, H. B., & Whitney, D. R. (1947) - Non-parametric rank-based comparison
  • Wilcoxon, F. (1945) - Rank-sum test theoretical foundation
  • D'Agostino, R. B. (1971) - Normality testing via skewness and kurtosis
  • Kerby, D. S. (2014) - Rank-biserial correlation for effect size
  • Ruxton, G. D. (2006) - The unequal variance t-test is an underused alternative to Student's t-test and the Mann–Whitney U test

Modern Site Reliability Standards

Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media.

Key implementation: Formalization of Service Level Objectives (SLOs), Error Budgets, and the use of Tail Latency percentiles (P95/P99) for reliability measurement.

Menascé, D. A., Almeida, V. A. F., & Dowdy, L. W. (1994). Capacity Planning and Performance Modeling: From Mainframes to Client-Server Systems. Prentice Hall PTR.

  • Chapter 7: Workload Characterization Techniques
  • Multi-Phase Testing Methodology

Lilja, D. J. (2005). Measuring Computer Performance: A Practitioner's Guide. Cambridge University Press.

  • Chapter 4: Statistical Analysis of Performance Data
  • Variance and Standard Deviation Calculations
  • Sample Size Determination Methods

Standards & Specifications

International Organization for Standardization (2011). ISO/IEC 25010:2011 - Systems and Software Quality Requirements and Evaluation (SQuaRE). ISO/IEC.

  • Section 6.2.3: Performance Efficiency Metrics
  • Section 6.2.4: Reliability Measurement Criteria

Internet Engineering Task Force (1999). RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1. IETF.

  • Section 8: Connection Management and Keep-Alive Implementation
  • Section 14: Header Field Definitions for Realistic Protocol Simulation

Empirical Studies & Conference Proceedings

Barford, P., & Crovella, M. (1998). Generating Representative Web Workloads for Network and Server Performance Evaluation. Proceedings of the 1998 ACM SIGMETRICS Joint International Conference on Measurement and Modeling of Computer Systems, 151–160. https://doi.org/10.1145/277851.277897

  • Warm-Up and Ramp-Up Methodology Validation
  • Think Time Modeling for Realistic User Simulation

Hamilton, J. D. (2007). On Designing and Deploying Internet-Scale Services. Proceedings of the 21st Large Installation System Administration Conference (LISA '07), 231–242.

  • Operations-Friendly Service Design Principles
  • Automation and Provisioning Methodology
  • Dependency Management and Failure Isolation
  • Hardware Standardization and Abstraction

Statistical Hypothesis Testing

Parametric Methods

Welch, B. L. (1947). The Generalization of "Student's" Problem when Several Different Population Variances are Involved. Biometrika, 34(1–2), 28–35. https://doi.org/10.1093/biomet/34.1-2.28

  • Implementation in dlt.sh: Welch's t-test for comparing two samples with unequal variances
  • Use case: Applied when both baseline and candidate data are approximately normally distributed
  • Formula: t = (μ₁ - μ₂) / √(σ₁²/n₁ + σ₂²/n₂)
  • Degrees of freedom: Welch-Satterthwaite equation for unequal variances

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates.

  • Implementation in dlt.sh: Cohen's d effect size calculation
  • Interpretation thresholds:
    • |d| < 0.2: Negligible effect
    • 0.2 ≤ |d| < 0.5: Small effect
    • 0.5 ≤ |d| < 0.8: Medium effect
    • |d| ≥ 0.8: Large effect
  • Formula: d = (μ₁ - μ₂) / σ_pooled

Non-Parametric Methods (New in v6.1)

Mann, H. B., & Whitney, D. R. (1947). On a Test of Whether One of Two Random Variables is Stochastically Larger than the Other. Annals of Mathematical Statistics, 18(1), 50–60. https://doi.org/10.1214/aoms/1177730491

  • Implementation in dlt.sh: Mann-Whitney U test for distribution-free comparison
  • Use case: Applied when data is non-normal (high skewness or kurtosis), or when samples have outliers
  • Algorithm:
    1. Combine both samples and assign ranks
    2. Calculate U = n₁n₂ + n₁(n₁+1)/2 - R₁ where R₁ is sum of ranks for sample 1
    3. For large samples (n₁, n₂ > 20): Use normal approximation with continuity correction
    4. Convert to z-score: z = (U - μᵤ) / σᵤ where μᵤ = n₁n₂/2 and σᵤ = √(n₁n₂(n₁+n₂+1)/12)
  • Advantages: Robust to outliers, no normality assumption, handles skewed data
  • Notes:
    • Rank assignment handles ties using average rank method (standard practice)
    • Continuity correction applied for discrete U distribution
    • Z-score approximation valid for n₁, n₂ > 20 (large sample approximation)

Wilcoxon, F. (1945). Individual Comparisons by Ranking Methods. Biometrics Bulletin, 1(6), 80–83. https://doi.org/10.2307/3001968

  • Relationship to Mann-Whitney: The Wilcoxon rank-sum test is mathematically equivalent to the Mann-Whitney U test
  • Theoretical foundation: Establishes the rank-based comparison methodology
  • Historical significance: One of the first non-parametric statistical tests

Kerby, D. S. (2014). The Simple Difference Formula: An Approach to Teaching Nonparametric Correlation. Comprehensive Psychology, 3, Article 11.IT.3.1. https://doi.org/10.2466/11.IT.3.1

  • Implementation in dlt.sh: Rank-biserial correlation as effect size for Mann-Whitney U test
  • Formula: r = 1 - (2U)/(n₁n₂)
  • Interpretation: Same thresholds as Cohen's d (|r| < 0.2: negligible, 0.2-0.5: small, 0.5-0.8: medium, ≥0.8: large)
  • Advantage: Provides standardized effect size comparable to Cohen's d but for non-parametric tests

Normality Testing (New in v6.1)

D'Agostino, R. B. (1971). An Omnibus Test of Normality for Moderate and Large Sample Sizes. Biometrika, 58(2), 341–348. https://doi.org/10.1093/biomet/58.2.341

  • Implementation in dlt.sh: Skewness and kurtosis-based normality assessment
  • Skewness formula: γ₁ = E[(X-μ)³]/σ³
    • Measures asymmetry of distribution
    • |γ₁| < 1.0 considered approximately symmetric
    • High positive skewness indicates long right tail (common in P99 latencies)
  • Kurtosis formula: γ₂ = E[(X-μ)⁴]/σ⁴ - 3
    • Measures tail heaviness relative to normal distribution
    • |γ₂| < 2.0 considered approximately normal tails
    • High kurtosis indicates heavy tails with outliers
  • Decision rule: If |skewness| > 1.0 OR |kurtosis| > 2.0 → Use Mann-Whitney U; otherwise → Use Welch's t-test
  • Minimum sample size: Requires n ≥ 20 for reliable assessment

Technical Documentation & Industry Standards

Apache Software Foundation (2024). ApacheBench Documentation. Retrieved from https://httpd.apache.org/docs/2.4/programs/ab.html

  • Metric Parsing Specifications
  • Statistical Output Format Documentation

Standard Performance Evaluation Corporation (1999). SPECweb99 Benchmark Design Document. SPEC.

  • Industry-Standard Web Server Benchmarking Methodology
  • Validation Procedures for Load Testing Tools
  • Workload Mix Design and Implementation

Implementation Notes for dlt.sh v6.1

Automatic Test Selection Logic

The script implements intelligent test selection based on data characteristics:

┌─────────────────────────────────┐
│   Check Normality of Samples   │
│   (Skewness & Kurtosis)         │
└────────────┬────────────────────┘
             │
     ┌───────┴────────┐
     │                │
Both Normal?    Either Non-Normal?
     │                │
     ▼                ▼
Welch's t-test   Mann-Whitney U
(Parametric)     (Non-Parametric)
More Powerful    More Robust

When Each Test is Used

Welch's t-test (parametric):

  • Applied when: |skewness| < 1.0 AND |kurtosis| < 2.0 for BOTH samples
  • Best for: Mean RPS, average response times, normally distributed metrics
  • Advantages: More statistical power, smaller p-values for true differences
  • Effect size: Cohen's d (standardized mean difference)

Mann-Whitney U test (non-parametric):

  • Applied when: |skewness| ≥ 1.0 OR |kurtosis| ≥ 2.0 for EITHER sample
  • Best for: P95/P99 latencies, tail metrics, error rates, cache hit rates
  • Advantages: Robust to outliers, no distribution assumptions, handles skewed data
  • Effect size: Rank-biserial correlation (analogous to Cohen's d)

Computational Complexity

Operation Time Complexity Space Complexity
Welch's t-test O(n) O(1)
Mann-Whitney U O(n log n) O(n)
Normality check O(n) O(1)

The performance overhead of Mann-Whitney U test is minimal (~200-500ms for 1000 samples) compared to the total test duration (typically 30-60 minutes).

Citation Style

This document follows a hybrid citation format combining elements of APA 7th Edition and IEEE standards, optimized for technical performance analysis research. All references are verified and accessible as of January 2025.

Version History

  • v6.2 (January 2025): Added Site Reliability Engineering standards for SLOs and tail latency measurement
  • v6.1 (January 2025): Added Mann-Whitney U test, automatic test selection, normality checking, rank-biserial correlation
  • v6.0 (January 2025): Added Welch's t-test, Cohen's d, hybrid baseline management, smart locale detection
  • v5.1 (December 2024): Original research-based implementation with 95% confidence intervals

Document prepared for academic and scientific use in performance engineering research.