Skip to content

Navigation Menu

Appearance settings

View all features
- BY COMPANY SIZE
  Enterprises
  Small and medium teams
  Startups
  Nonprofits
- BY USE CASE
  App Modernization
  DevSecOps
  DevOps
  CI/CD
  View all use cases
- BY INDUSTRY
  Healthcare
  Financial services
  Manufacturing
  Government
  View all industries
View all solutions
- EXPLORE BY TOPIC
  AI
  Software Development
  DevOps
  Security
  View all topics
- EXPLORE BY TYPE
  Customer stories
  Events & webinars
  Ebooks & reports
  Business insights
  GitHub Skills
- SUPPORT & SERVICES
  Documentation
  Customer support
  Community forum
  Trust center
  Partners
View all resources
- COMMUNITY
  GitHub SponsorsFund open source developers
- PROGRAMS
  Security Lab
  Maintainer Community
  Accelerator
  GitHub Stars
  Archive Program
- REPOSITORIES
  Topics
  Trending
  Collections
- ENTERPRISE SOLUTIONS
  Enterprise platformAI-powered developer platform
- AVAILABLE ADD-ONS
  GitHub Advanced SecurityEnterprise-grade security features
  Copilot for BusinessEnterprise-grade AI features
  Premium SupportEnterprise-grade 24/7 support
Pricing

Search code, repositories, users, issues, pull requests...

Search

Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our documentation.

Appearance settings

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

ArchiveTeam / wpull Public

Notifications You must be signed in to change notification settings
Fork 85
Star 608

Code
Issues 185
Pull requests 24
Actions
Projects
Wiki
Security and quality
Insights

Additional navigation options

Code
Issues
Pull requests
Actions
Projects
Wiki
Security and quality
Insights

URL prioritisation, split meta WARCs, and miscellaneous bug fixes#393

Open

JustAnotherArchivist wants to merge 41 commits into

ArchiveTeam:developArchiveTeam/wpull:developfrom

JustAnotherArchivist:developJustAnotherArchivist/wpull:developCopy head branch name to clipboard

Conversation Commits41 (41)Checks Files changed

Open

URL prioritisation, split meta WARCs, and miscellaneous bug fixes#393
JustAnotherArchivist wants to merge 41 commits into
ArchiveTeam:developfrom
JustAnotherArchivist:develop

Commits

Commits on Jan 16, 2023

Load plugins early for improved flexibility (fixes chfoo/wpull#383)
JustAnotherArchivist
committed
Fix URLRecord not defining all fields on initialisation
JustAnotherArchivist
committed
Initial implementation of URL prioritisation
JustAnotherArchivist
committed
Test suite for prioritisation code
JustAnotherArchivist
committed
Fix pipeline retrieving an item from the ItemSource prematurely
JustAnotherArchivist
committed
Replace the poison pill generic object with a custom class
JustAnotherArchivist
committed
Simplify ItemQueue.put_item_coro
JustAnotherArchivist
committed
Add docstrings to the pipeline module
JustAnotherArchivist
committed
Fix pipeline bug introduced in 2e42b7d9 leading to links discovered on the last
JustAnotherArchivist
committed
Apply --concurrent argument to pipeline series (fixes chfoo/wpull#339)
JustAnotherArchivist
committed
URL prioritisation plugin interface get_priority
JustAnotherArchivist
committed
Tests for get_priority hook
JustAnotherArchivist
committed
Update docs
JustAnotherArchivist
committed
Pin html5lib version to 0.9999999 (seven nines) (fixes chfoo/wpull#332)
JustAnotherArchivist
committed
Docs: Add description of prioritisation
JustAnotherArchivist
committed
Add --warc-split-meta
JustAnotherArchivist
committed
Remove flattening consecutive slashes in URL paths (fixes chfoo/wpull#380)
JustAnotherArchivist
committed
Fix test broken by 44ef3690
JustAnotherArchivist
committed
Typo
JustAnotherArchivist
committed
Handle backslashes in the path of special URLs like forward slashes (fixes chfoo/wpull#377)
JustAnotherArchivist
committed
Handle email.utils.parsedate returning None when parsing fails (fixes chfoo/wpull#376)
JustAnotherArchivist
committed
Fix changelog
JustAnotherArchivist
committed
Strip tab and newline characters from scraped URLs (fixes chfoo/wpull#355)
JustAnotherArchivist
committed
Handle empty ports correctly (fixes chfoo/wpull#340)
JustAnotherArchivist
committed
Update changelog to reflect the new situation regarding the official repository
JustAnotherArchivist
committed
Document --warc-split-meta
JustAnotherArchivist
committed
Print some version information on CI
JustAnotherArchivist
committed
Work around the change in tornadoweb/tornado@84bb2e28
JustAnotherArchivist
committed
Use snake_case for variables in prioritisation argument handling
JustAnotherArchivist
committed
Spaces according to PEP8
JustAnotherArchivist
committed
Wrap docstrings
JustAnotherArchivist
committed
Wrap long lines
JustAnotherArchivist
committed
Remove stray prints
JustAnotherArchivist
committed
Fix long lines in prioritisation plugin
JustAnotherArchivist
committed
Pass the URLInfo and URLRecord objects directly into the get_priority hook
JustAnotherArchivist
committed
s/priorisation/prioritisation/
JustAnotherArchivist
committed
A few minor corrections
JustAnotherArchivist
committed
Fix test broken by a100e5bd
JustAnotherArchivist
committed
Fix order of todo/error with priorities
JustAnotherArchivist
committed
Fix performance regression due to missing index
JustAnotherArchivist
committed
Pin html5lib and SQLAlchemy to working versions in setup.py
JustAnotherArchivist
committed

Footer

© 2026 GitHub, Inc.

Footer navigation

Terms
Privacy
Security
Status
Community
Docs
Contact

You can’t perform that action at this time.