✨ Pre-warm shared ebook cache on book listing create and metadata sync#1266
Open
williamchong wants to merge 1 commit into
Open
✨ Pre-warm shared ebook cache on book listing create and metadata sync#1266williamchong wants to merge 1 commit into
williamchong wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a fire-and-forget “cache warming” step that streams ebook files referenced by an NFT book’s on-chain metadata into the shared GCS cache bucket used by the ebook-cors service, so first reads are more likely to hit a warm cache.
Changes:
- Invoke cache warming after NFT book listing creation (store route) and after on-chain metadata refresh (ISCN sync).
- Introduce a new
cacheBookFilesFromNFTClassMetadata()utility that resolves metadata URLs (including LikeCoin/arweave/v2/link/<txHash>indirection) and streams PDF/EPUB bytes directly into GCS.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
src/util/api/likernft/book/index.ts |
Triggers ebook-cache warming during ISCN metadata sync without blocking the sync flow. |
src/util/api/likernft/book/cache.ts |
New implementation for resolving metadata file URLs, fetching with gateway fallbacks, and streaming into the shared GCS cache bucket. |
src/routes/likernft/book/store.ts |
Triggers ebook-cache warming after successful book listing creation without blocking the HTTP response. |
Comment on lines
+48
to
+79
| async function resolveBookFileCacheURL( | ||
| targetURI: string, | ||
| ): Promise<string | undefined> { | ||
| if (!targetURI) return undefined; | ||
| let parsedUrl: URL; | ||
| try { | ||
| parsedUrl = new URL(targetURI); | ||
| } catch { | ||
| return undefined; | ||
| } | ||
|
|
||
| if (targetURI.startsWith(LIKECOIN_API_LINK_PREFIX)) { | ||
| const txHash = targetURI | ||
| .slice(LIKECOIN_API_LINK_PREFIX.length) | ||
| .split('?')[0] | ||
| .split('/')[0]; | ||
| if (!txHash) return undefined; | ||
| const tx = await getArweaveTxInfo(txHash); | ||
| if (!tx?.arweaveId) return undefined; | ||
| const link = new URL(`${ARWEAVE_GATEWAY}/${tx.arweaveId}`); | ||
| if (tx.key) link.searchParams.set('key', tx.key); | ||
| parsedUrl = link; | ||
| } | ||
|
|
||
| switch (parsedUrl.protocol) { | ||
| case 'ar:': | ||
| return `${ARWEAVE_GATEWAY}/${parsedUrl.host}`; | ||
| case 'ipfs:': | ||
| return `https://w3s.link/ipfs/${parsedUrl.host}`; | ||
| default: | ||
| return parsedUrl.toString(); | ||
| } |
| await pipeline(data, fileRef.createWriteStream({ | ||
| metadata: { | ||
| contentType, | ||
| cacheControl: `${PERMANENT_CACHE_TIME_IN_S}`, |
Comment on lines
+171
to
+175
| await Promise.all( | ||
| targetURIs.map((targetURI) => cacheBookFile(classId, targetURI).catch((err) => { | ||
| // eslint-disable-next-line no-console | ||
| console.error(`Failed to cache book file for ${classId} (${targetURI}):`, err); | ||
| })), |
Comment on lines
+168
to
+176
| const targetURIs = Array.from( | ||
| new Set(getTargetURIsFromNFTClassMetadata(metadata)), | ||
| ); | ||
| await Promise.all( | ||
| targetURIs.map((targetURI) => cacheBookFile(classId, targetURI).catch((err) => { | ||
| // eslint-disable-next-line no-console | ||
| console.error(`Failed to cache book file for ${classId} (${targetURI}):`, err); | ||
| })), | ||
| ); |
Comment on lines
+40
to
+80
| /** | ||
| * Reproduces ebook-cors nft/index.js parseNFTMetadataURL() so the cache key we | ||
| * write is byte-for-byte identical to the one ebook-cors will look up. | ||
| * | ||
| * The api/arweave/v2/link/<txHash> case is resolved locally via Firestore | ||
| * (getArweaveTxInfo) instead of an HTTP round-trip — it yields the same | ||
| * `${ARWEAVE_GATEWAY}/${arweaveId}` (+ ?key=) link the endpoint returns. | ||
| */ | ||
| async function resolveBookFileCacheURL( | ||
| targetURI: string, | ||
| ): Promise<string | undefined> { | ||
| if (!targetURI) return undefined; | ||
| let parsedUrl: URL; | ||
| try { | ||
| parsedUrl = new URL(targetURI); | ||
| } catch { | ||
| return undefined; | ||
| } | ||
|
|
||
| if (targetURI.startsWith(LIKECOIN_API_LINK_PREFIX)) { | ||
| const txHash = targetURI | ||
| .slice(LIKECOIN_API_LINK_PREFIX.length) | ||
| .split('?')[0] | ||
| .split('/')[0]; | ||
| if (!txHash) return undefined; | ||
| const tx = await getArweaveTxInfo(txHash); | ||
| if (!tx?.arweaveId) return undefined; | ||
| const link = new URL(`${ARWEAVE_GATEWAY}/${tx.arweaveId}`); | ||
| if (tx.key) link.searchParams.set('key', tx.key); | ||
| parsedUrl = link; | ||
| } | ||
|
|
||
| switch (parsedUrl.protocol) { | ||
| case 'ar:': | ||
| return `${ARWEAVE_GATEWAY}/${parsedUrl.host}`; | ||
| case 'ipfs:': | ||
| return `https://w3s.link/ipfs/${parsedUrl.host}`; | ||
| default: | ||
| return parsedUrl.toString(); | ||
| } | ||
| } |
williamchong
force-pushed
the
feature/book
branch
2 times, most recently
from
ed49e9e to
94a3d8c
Compare
When a book listing is created or its on-chain metadata/file path is refreshed, stream the referenced files into the GCS bucket that the ebook-cors service reads from, so first reads hit a warm cache. The cache key resolution mirrors ebook-cors byte-for-byte (arweave links resolved locally via Firestore) so entries are actually hit. Runs fire-and-forget so it never blocks the response or the ISCN sync.
williamchong
force-pushed
the
feature/book
branch
from
94a3d8c to
05cb501
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a book listing is created or its on-chain metadata/file path is refreshed, stream the referenced files into the GCS bucket that the ebook-cors service reads from, so first reads hit a warm cache.
The cache key resolution mirrors ebook-cors byte-for-byte (arweave links resolved locally via Firestore) so entries are actually hit. Runs fire-and-forget so it never blocks the response or the ISCN sync.