Replace R2 llms-full.txt with fetch from middlecache (build/request time hybrid)#29302
Draft
mvvmm wants to merge 6 commits intoproductionfrom
Draft
Replace R2 llms-full.txt with fetch from middlecache (build/request time hybrid)#29302mvvmm wants to merge 6 commits intoproductionfrom
mvvmm wants to merge 6 commits intoproductionfrom
Conversation
Adds bin/fetch-llms-full.ts that downloads the llms-full.txt tarball from middlecache at prebuild/predev time and extracts it into public/, where Astro serves the files as static assets. This removes the VENDORED_MARKDOWN R2 bucket binding and the worker's llms-full.txt interception — the files are now served directly by the ASSETS binding like any other static file.
Contributor
|
This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:
|
mvvmm
changed the title
…limit) The root llms-full.txt is ~40 MB which exceeds the Workers 25 MiB per-asset size limit. Add it to .assetsignore so it is excluded from the asset manifest, and proxy it from middlecache at request time. Per-product llms-full.txt files (all under 4 MB) continue to be served as static assets.
mvvmm
changed the title
Replace the HTTP proxy for root /llms-full.txt with a direct R2 binding to the middlecache bucket (same account). This eliminates the HTTP overhead (DNS, TLS, connection) and reads directly via internal IPC.
… limit) Instead of extracting the root llms-full.txt into public/ and trying to exclude it via .assetsignore, simply skip it during tar extraction using --exclude. The root file (~40 MB) exceeds the Workers 25 MiB per-asset limit and is served from the middlecache R2 bucket at request time. Only per-product llms-full.txt files (all under 4 MB) are extracted into public/ and served as static assets.
R2 bindings only work for Workers deployed in the same account. External contributors running npm run dev locally would get an empty R2 simulation and 404s for root llms-full.txt. Revert to HTTP fetch from the public middlecache URL which works for everyone.
Contributor
|
Preview URL: https://4bc5dc6b.preview.developers.cloudflare.com |
The root llms-full.txt is served at request time from the middlecache R2 bucket via direct binding (same account, no HTTP overhead). Per-product files remain static assets from the build.
mvvmm
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bin/fetch-llms-full.tsthat fetches the llms-full.txt tarball from middlecache at build time and extracts per-product files intopublic/ASSETSbinding/llms-full.txt(~40 MB) exceeds the Workers 25 MiB per-asset limit, so it is excluded from extraction and read from themiddlecacheR2 bucket at request time via direct binding (same account, no HTTP overhead)VENDORED_MARKDOWNR2 binding with aMIDDLECACHER2 binding pointing to themiddlecachebucketApproach
Build-time static assets + R2 read for root: The
prebuildhook downloadsllms-full.tar.gz(~15MB) from middlecache and extracts only per-product files intopublic/{product}/llms-full.txt(96 products). Astro copies these intodist/and they are served as static assets.The root
llms-full.txt(~40 MB) exceeds the Workers 25 MiB per-asset size limit, so it is excluded from extraction viatar --excludeand the worker reads it from themiddlecacheR2 bucket at request time via direct binding.The fetch script follows the same pattern as
bin/fetch-skills.ts:.tmp/middlecache/with caching (skips if already present)--softflag forpredev(warns on failure instead of blocking)--forceflag to re-fetchThis is one of two approaches being tested — compare with #29301 which reads all files from the middlecache R2 bucket at request time.
Tradeoffs vs request-time approach
Changes
bin/fetch-llms-full.ts— Build-time fetch script (excludes root llms-full.txt from extraction)package.json— Added fetch-llms-full toprebuildandpredevhooksworker/index.ts— ReplacedVENDORED_MARKDOWNR2 read with targetedMIDDLECACHER2 read for root/llms-full.txtonly (per-product files served as static assets)wrangler.toml— ReplacedVENDORED_MARKDOWN→vendored-markdownbinding withMIDDLECACHE→middlecachebindingworker/worker-configuration.d.ts— ReplacedVENDORED_MARKDOWN: R2BucketwithMIDDLECACHE: R2BucketinEnvinterface.gitignore— Added pattern for extracted per-product llms-full.txt files inpublic/