Skip to content

Commit b15e74b

Browse files
svdC1svdC1
committed
Restructure tests into unit/integration and add live integration suite
- Split tests/ into tests/unit/ and tests/integration/, tag modules with unit/integration markers, and move fixtures to per-directory conftest.py - Add tests/integration/test_client.py covering live proxy error detection, data boundaries, and session/RID telemetry against httpbingo.org - Add integration_tests.yml workflow gated on SCRAPE_DO_API_KEY - Scope ci.yml to tests/unit/ and tag uploads with flags: unit - Configure codecov.yml with per-flag project/patch statuses and carryforward - Add [tool.coverage.*] config and register pytest markers in pyproject.toml - Ignore logs/ and *.log - Fix client.py docstring reference (client_timeout -> r_timeout)
1 parent a268074 commit b15e74b

19 files changed

Lines changed: 777 additions & 154 deletions

‎.github/codecov.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
coverage:
2+
status:
3+
project:
4+
unit:
5+
target: auto
6+
threshold: 5
7+
flags: [unit]
8+
integration:
9+
target: auto
10+
threshold: 5
11+
flags: [integration]
12+
patch:
13+
unit:
14+
target: auto
15+
flags: [unit]
16+
integration:
17+
target: auto
18+
flags: [integration]
19+
20+
flags:
21+
unit:
22+
paths: [src/]
23+
carryforward: true
24+
integration:
25+
paths: [src/]
26+
carryforward: true

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,17 +36,19 @@ jobs:
3636
run: mypy src/
3737

3838
- name: Run tests
39-
run: pytest --cov=scrape_do --cov-branch --junitxml=junit.xml
39+
run: pytest tests/unit/ --cov --junitxml=junit.xml
4040

4141
- name: Upload Coverage To Codecov
4242
uses: codecov/codecov-action@v5
4343
with:
44+
flags: unit
4445
token: ${{ secrets.CODECOV_TOKEN }}
4546

4647
- name: Upload Test Results To Codecov
4748
if: ${{ !cancelled() }}
4849
uses: codecov/codecov-action@v5
4950
with:
51+
flags: unit
5052
report_type: test_results
5153
token: ${{ secrets.CODECOV_TOKEN }}
5254

‎.github/workflows/integration_tests.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: Integration Tests
2+
on:
3+
pull_request:
4+
branches: [ "main" ]
5+
release:
6+
types: [ published ]
7+
workflow_dispatch:
8+
9+
jobs:
10+
test:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- name: Check out the repository
14+
uses: actions/checkout@v4
15+
16+
- name: Set up Python
17+
uses: actions/setup-python@v5
18+
with:
19+
python-version: "3.13"
20+
cache: "pip"
21+
22+
- name: Install Dependencies
23+
run: |
24+
python -m pip install --upgrade pip
25+
pip install -e .[dev]
26+
27+
28+
- name: Run Integration Tests
29+
env:
30+
SCRAPE_DO_API_KEY: ${{secrets.SCRAPE_DO_API_KEY}}
31+
run: pytest tests/integration/ --cov --junitxml=junit.xml
32+
33+
34+
- name: Upload Test Logs
35+
uses: actions/upload-artifact@v4
36+
if: always()
37+
with:
38+
name: integration-run-logs
39+
path: logs/*.log
40+
retention-days: 7
41+
42+
- name: Upload Coverage To Codecov
43+
uses: codecov/codecov-action@v5
44+
with:
45+
flags: integration
46+
token: ${{ secrets.CODECOV_TOKEN }}
47+
48+
- name: Upload Test Results To Codecov
49+
if: ${{ !cancelled() }}
50+
uses: codecov/codecov-action@v5
51+
with:
52+
flags: integration
53+
report_type: test_results
54+
token: ${{ secrets.CODECOV_TOKEN }}

‎.gitignore‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,6 @@ __pycache__/
77
.ruff_cache/
88
dist/
99
build/
10-
*.egg-info/
10+
*.egg-info/
11+
logs/
12+
*.log

‎pyproject.toml‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,4 +38,33 @@ docs = [
3838
]
3939

4040
[tool.setuptools.packages.find]
41-
where = ["src"]
41+
where = ["src"]
42+
43+
[tool.pytest.ini_options]
44+
markers = [
45+
"unit: marks tests as unit tests",
46+
"integration: marks tests as integration tests",
47+
]
48+
49+
[tool.coverage.run]
50+
branch = true
51+
source = ["scrape_do"]
52+
relative_files = true
53+
54+
[tool.coverage.paths]
55+
source = [
56+
"src/scrape_do",
57+
"*/site-packages/scrape_do",
58+
]
59+
60+
[tool.coverage.report]
61+
show_missing = true
62+
skip_covered = false
63+
precision = 2
64+
exclude_lines = [
65+
"pragma: no cover",
66+
"raise NotImplementedError",
67+
"if TYPE_CHECKING:",
68+
"if __name__ == .__main__.:",
69+
"@(abc\\.)?abstractmethod",
70+
]

‎src/scrape_do/client.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,7 @@ class ScrapeDoClient:
131131
Additionally, the following `httpx.Client.request` parameters can be
132132
provided as keyword arguments during request execution.
133133
134-
- `timeout` -> `client_timeout`
134+
- `timeout` (`r_timeout`)
135135
- `extensions`
136136
137137
For more information on their behaviour and default values, please

‎tests/conftest.py‎

Lines changed: 1 addition & 150 deletions
Original file line numberDiff line numberDiff line change
@@ -1,110 +1,9 @@
11
"""
2-
Shared fixtures for the SDK's test suite
2+
Shared fixtures for integration and unit tests.
33
"""
44

55
import pytest
6-
import httpx
76
from scrape_do.models import RequestParameters, PreparedScrapeDoRequest
8-
from unittest.mock import MagicMock
9-
from scrape_do.client import ScrapeDoClient
10-
11-
12-
@pytest.fixture
13-
def example_url() -> str:
14-
"""
15-
Provides a valid fake url to be used for model testing
16-
17-
Returns:
18-
A valid fake url
19-
"""
20-
return "https://example.com/"
21-
22-
23-
@pytest.fixture
24-
def mock_json_payload() -> dict:
25-
"""
26-
Provides a fake JSON dictionary including all keys that can be present
27-
in the JSON returned by Scrape.do when `returnJSON=true`.
28-
29-
Returns:
30-
The fake JSON dictionary
31-
"""
32-
return {
33-
"statusCode": 200,
34-
"content": "<html>Target Data</html>",
35-
"networkRequests": [{
36-
"url": "https://example.com/api",
37-
"method": "POST",
38-
"status": 204,
39-
"request_headers": {},
40-
"request_body": "{\"req\": \"data\"}",
41-
"response_body": "",
42-
"response_headers": {}
43-
}],
44-
"websocketRequests": [{
45-
"type": "received",
46-
"event": {
47-
"requestId": "586051.322",
48-
"timestamp": 21815567.089025,
49-
"response": {
50-
"opcode": 1,
51-
"mask": False,
52-
"payloadData": "{\"live_price\": 65000.00}"
53-
}
54-
}
55-
}],
56-
"actionResults": [
57-
{
58-
"action": "Click",
59-
"index": 0,
60-
"success": False,
61-
"error": "Element not found",
62-
"response": None
63-
},
64-
{
65-
"action": "Wait",
66-
"index": 1,
67-
"success": True
68-
}
69-
],
70-
"screenShots": [{
71-
"type": "FullScreenShot",
72-
"image": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HA",
73-
"error": None
74-
}],
75-
"frames": [{
76-
"url": "https://example.com/iframe",
77-
"content": "<html>Iframe Content</html>"
78-
}]
79-
}
80-
81-
82-
@pytest.fixture
83-
def mock_headers() -> httpx.Headers:
84-
"""
85-
Provides a fake httpx.Headers object including all headers returned by
86-
Scrape.do.
87-
88-
Returns:
89-
The fake httpx.Headers object
90-
"""
91-
headers = {
92-
"server": "cloudflare",
93-
"x-frame-options": "DENY",
94-
"transfer-encoding": "chunked",
95-
"scrape.do-auth": "0",
96-
"scrape.do-cookies": "cookie1=value1;cookie2=value2",
97-
"scrape.do-initial-status-code": "200",
98-
"scrape.do-rate": "0:0",
99-
"scrape.do-remaining-credits": "300000",
100-
"scrape.do-request-cost": "25",
101-
"scrape.do-request-id": "123e4567-e89b-12d3-a456-426614174000",
102-
"scrape.do-resolved-url": "https://example.com/final",
103-
"scrape.do-rid": "node-123",
104-
"scrape.do-target-url": "https://example.com"
105-
}
106-
107-
return httpx.Headers(headers)
1087

1098

1109
@pytest.fixture
@@ -120,51 +19,3 @@ def _make(
12019
params = RequestParameters(url=url, **kwargs)
12120
return PreparedScrapeDoRequest(api_params=params, method=method)
12221
return _make
123-
124-
125-
@pytest.fixture
126-
def make_response():
127-
"""
128-
Factory to generate strictly controlled httpx.Response mocks.
129-
"""
130-
def _make(
131-
status_code: int,
132-
json_data: dict = None,
133-
text: str = None,
134-
proxy_status_header: str = None
135-
):
136-
headers = {}
137-
if proxy_status_header is not None:
138-
headers["scrape.do-initial-status-code"] = str(proxy_status_header)
139-
140-
if json_data is not None:
141-
return httpx.Response(status_code, headers=headers, json=json_data)
142-
return httpx.Response(status_code, headers=headers, text=text or "")
143-
return _make
144-
145-
146-
@pytest.fixture
147-
def mock_sync_client():
148-
"""
149-
Yields a cleanly initialized ScrapeDoClient for testing.
150-
"""
151-
with ScrapeDoClient(api_token='dummy_token') as client:
152-
yield client
153-
154-
155-
@pytest.fixture
156-
def mock_env_vars(monkeypatch: pytest.MonkeyPatch):
157-
"""Clears the SCRAPE_DO_API_KEY from the environment for a test function
158-
159-
Guarantees that tests do not accidentally inherit a real
160-
developer token from the local machine's environment variables
161-
"""
162-
monkeypatch.delenv("SCRAPE_DO_API_KEY", raising=False)
163-
164-
165-
@pytest.fixture
166-
def mock_sleep(mocker) -> MagicMock:
167-
"""
168-
Mocks `time.sleep` across an entire test function.
169-
"""
170-
return mocker.patch("time.sleep", return_value=None)

tests/models/__init__.py renamed to tests/integration/__init__.py

File renamed without changes.

0 commit comments

Comments
 (0)