Document Retrieval (actually Knowledge Resource Retriever) is the next step in the Feature Pipeline. Based on a metadata record it needs to retrieve the document for which the record is the 'evidence'. There might be various issues trying to do so, e.g. the document no longer exists, it is behind a paywall, scraping a URL does not lead to a filename, downloading fails or times outs, etc.
Should probably process jobs in parallel, with retries, and log issues.
Document Retrieval (actually Knowledge Resource Retriever) is the next step in the Feature Pipeline. Based on a metadata record it needs to retrieve the document for which the record is the 'evidence'. There might be various issues trying to do so, e.g. the document no longer exists, it is behind a paywall, scraping a URL does not lead to a filename, downloading fails or times outs, etc.
Should probably process jobs in parallel, with retries, and log issues.