Skip to content

test: add property-based tests for evaluators using Hypothesis - #37

Merged
zurfjereluhmie merged 1 commit into
mainfrom
test/property-based-tests
Apr 22, 2026
Merged

test: add property-based tests for evaluators using Hypothesis#37
zurfjereluhmie merged 1 commit into
mainfrom
test/property-based-tests

Conversation

@zurfjereluhmie

Copy link
Copy Markdown
Owner

Summary

  • Add 21 property-based tests covering all 7 evaluators (numeric_tolerance, top_k, json_match, regex, exact_match, contains, membership) using Hypothesis
  • Add hypothesis as a dev dependency
  • Guard tests against _parse_expected's ast.literal_eval behavior that converts numeric strings (e.g. "0") into non-string Python objects

Evaluator coverage

Evaluator Properties tested
numeric_tolerance identity, bool return type, within-default-tolerance, non-numeric → False, formula verification
top_k score in [0,1], perfect score, zero score, None output
json_match score in [0,1], identical JSON = 1.0, invalid JSON = 0.0
regex no crash on arbitrary strings, escaped literal matches itself, None output
exact_match identity, different strings don't match
contains substring detection, None output
membership member found, non-member not found

Notable finding

_parse_expected uses ast.literal_eval on string expected values, which silently converts numeric-looking strings ("0", "1.5", "True") into their Python literal types. This causes type mismatches in evaluators that expect string inputs. Tests use assume() guards and a _survives_literal_eval helper to filter these inputs. This is a pre-existing design issue worth addressing separately.

@zurfjereluhmie
zurfjereluhmie force-pushed the test/property-based-tests branch from 3993bde to 2141003 Compare April 22, 2026 12:58
@zurfjereluhmie
zurfjereluhmie merged commit 36b9534 into main Apr 22, 2026
8 checks passed
@zurfjereluhmie
zurfjereluhmie deleted the test/property-based-tests branch May 4, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant