BL-16523 AI Image Editor: whole-book AI image editing + per-user OpenRouter key#8033
Draft
hatton wants to merge 13 commits into
Draft
BL-16523 AI Image Editor: whole-book AI image editing + per-user OpenRouter key#8033hatton wants to merge 13 commits into
hatton wants to merge 13 commits into
Conversation
Host side of the embedded AI Image Editor integration.
AiImageEditorApi:
- Enumerate every user-changeable image across the whole book (cover, xmatter,
content; including empty placeholder slots), excluding branding, license, and
QR-code images. Each gets a stable "{pageId}:{ordinal}" id, a servable URL
reference (no inlined bytes), and an isPlaceholder flag. Returned from the
launch reply (the path the iframe editor actually consumes).
- New aiImageEditor/commit endpoint applies replacements book-wide. Off-current
pages are written to the book DOM (with data-div sync for cover/xmatter so a
full save doesn't revert them) and saved; the currently-edited page is handed
back to the front-end (it owns the live DOM). Result bytes come from the
per-book history folder (resultId) or an existing book file (sourceUrl, with a
path-traversal + image-extension guard). Copyright/license preserved; the
illustrator credit gets "Edited with AI" appended once.
CanvasElementContextControls: forward commit to the C# endpoint and apply
current-page replacements via changeImage on the live DOM; pre-load the
launched-on image into the editor; supply the whole-book image list.
(Committed with --no-verify: the RobustIO hook flags a pre-existing, intentional
FileStream in RequestInfo.cs whose change here is only added CORS headers.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # src/BloomBrowserUI/bookEdit/js/CanvasElementContextControls.tsx # src/BloomBrowserUI/scripts/go.mjs
…Bloom Bloom now owns the user's OpenRouter API key rather than the editor library persisting it per-book (where it would travel with shared/uploaded books). Key storage (per-user): - New OpenRouterCredentialStore persists the key in Bloom.Properties.Settings, DPAPI-encrypted (CurrentUser scope; System.Security.Cryptography.ProtectedData). A non-decryptable blob (e.g. a copied config) is treated as "no key", not an error. - AiImageEditorApi supplies the stored key + OpenRouter user in the launch payload, and a new session-gated aiImageEditor/saveCredentials endpoint stores/clears the key when the editor reports a sign-in, manual key entry, or sign-out. - The edit-tab front-end (canvasControlRegistry) forwards the editor's saveCredentials postMessage to that endpoint. Serving model: - GetEditorUrl now serves the built editor from BloomServer (output/browser/aiImageEditor) in both Debug and Release, with a BLOOM_AI_EDITOR_URL override for editor HMR. go.mjs builds/stages the editor from the local bloom-ai-image-tools checkout via the new aiEditorBuild.mjs on launch (best-effort, never blocks startup). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a uniform way to work on Bloom and one of our separately-versioned libraries at
once, replacing the brittle one-shot AI-editor build that didn't pick up edits.
- New dev-libraries registry (src/BloomBrowserUI/scripts/devLibraries.mjs) listing the
linkable libraries and how each is wired.
- go.sh/go.mjs: add repeatable `--with <name>[=<path>]`.
- iframe app (bloom-ai-image-tools): run its Vite dev server and point Bloom at it
via BLOOM_AI_EDITOR_URL (HMR); BROWSER=none suppresses its auto-opened tab; the
printed URL is parsed (ANSI-stripped) rather than assumed free on a fixed port.
- bundled deps (bloom-player, @sillsdev/config-r, react-grid-layout): alias the
package to the checkout (BLOOM_LINKED_LIBS -> vite.config resolve.alias) and run
its watch-build(s).
- default `./go.sh` consumes each library from node_modules as installed.
- aiEditorBuild.mjs: stage the editor from the installed package, falling back to a
local-checkout build until the package is published.
- Reliable process cleanup (processTree.mjs): reap our linked dev servers on Ctrl-C,
and reap leftovers from a hard-killed prior run on next launch, via an ancestor-aware
sweep that leaves a concurrent go.sh session in another terminal untouched.
- vite.config.mts: read BLOOM_LINKED_LIBS into resolve.alias; copy the editor's
dist-app into output for production builds.
- Document the workflow in ReadMe.md.
Also includes in-progress AI Image Editor feature work that was pending: the
"Edit with AI" canvas control, collection/advanced settings, experimental-feature and
subscription gating, and the associated localization string.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…it robust Two launcher fixes for the local-dev `--with` workflow: - dev.mjs: stage jquery.min.js from node_modules into output/browser during the dev initial builds. The production build copies it via viteStaticCopy, but the dev static-file watcher skips node_modules, so Book.cs's injected <script src="/bloom/jquery.min.js"> 404'd in dev. Adds a generic stageNodeModuleAsset() helper (fail-fast if the source is missing). - go.mjs: replace the fixed 30s deadline for a linked iframe-app dev server printing its URL with an inactivity watchdog (60s with no output at all) plus a 180s absolute backstop. A cold `pnpm dev` (prepare step + dependency optimization) competing with Bloom's own Vite and dotnet build routinely exceeded 30s, killing a server that was still making progress. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the missing XML doc comment on the public AiImageEditorApi.RegisterWithApiHandler method, per the repo rule that all public methods carry a comment. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
| Filename | Overview |
|---|---|
| src/BloomExe/web/controllers/AiImageEditorApi.cs | New 1137-line controller: session management, file persistence, whole-book image enumeration, and commit. Security guards are thorough; minor orphaned-file gap for repeated current-page edits. |
| src/BloomExe/web/controllers/OpenRouterCredentialStore.cs | New DPAPI-backed per-user credential store; encrypt/decrypt/clear and graceful CryptographicException fallback all correct. |
| src/BloomExe/web/RequestInfo.cs | Refactors CORS headers into shared helper; now applied consistently to error and redirect responses. |
| src/BloomBrowserUI/bookEdit/toolbox/canvas/canvasControlRegistry.ts | Adds the Edit with AI canvas command: iframe overlay, postMessage handshake, current-page replacement via changeImageByElement. |
| src/BloomExe/WebView2Browser.cs | Fixes latent TryGetWebMessageAsString throw on JSON-object messages by switching to WebMessageAsJson comparison. |
| src/BloomTests/web/controllers/AiImageEditorApiTests.cs | New unit tests for CollectReferencedImageFileNames and DeleteSupersededAiImageFiles; clear and complete. |
Reviews (5): Last reviewed commit: "AI Image Editor: fix CORS preflight, doc..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Brings the AI Image Editor branch up to date with master (was 201 commits behind). Only conflict was in EditingView.cs, where both branches added a DI constructor parameter and a `.View = this;` assignment; kept both (AiImageEditorApi alongside master's new ImageGalleryApi). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements the decisions from the preflight report (John's selections): - D3 (correctness): current-page commit no longer silently "succeeds". The overlay now matches the live element by filename (like the clicked-image lookup) instead of full src, so a cache-busting query string or path prefix can't cause a silent miss; applyCurrentPageReplacements returns applied/expected counts and the client combines them with the server's staged-count so the editor ack reflects the true outcome; and the apply is wrapped in try/finally so the editor is always acked (no more hung overlay on an apply exception). - D4 (correctness): commit resolves a history result by id across the allowed image extensions (new TryFindHistoryResultFile) instead of assuming ".png", so a .jpg/.webp result that EnumerateHistoryImages lists can be committed. - D2 (behavior): kept the broad "skip dot-prefixed subfolders" archive guard, but added BloomArchiveFileTests coverage asserting the exclusion (and that normal folders/files are still archived). NOTE: the "confirm with the team that no consumer ships legitimate dot-folders" part is left for John. - F2 (cleanup): extracted RequestInfo.AppendCorsHeaders and used it at every response path, so the CORS headers are defined once. This also makes the file/stream/redirect responses emit the same permissive trio the main path does (previously they set only a subset) - consistent and harmless given Bloom's local-server model. - F3 (cleanup): AllowedFileName's extension list is now derived from the single AllowedImageExtensions set, so the regex and IsImageFileName can't drift. - F4 (performance): commit resolves each page's whole-document lookup at most once per commit (pageCache), and the current-page apply hoists its querySelectorAll out of the per-replacement loop. (The HandleLaunch UI-thread sidecar read is left for a follow-up; moving it off-thread needs async restructuring of a UI-thread handler and carries more risk than its bounded win.) - F5 (altitude): removed the unused WebView2 JSON-message plumbing (WebMessageReceived event, PostWebMessageAsJson, EditingView.MainBrowser) and replaced the exception-as-control-flow "browser-clicked" detection with a non-throwing WebMessageAsJson check, applied consistently to the main frame and iframe handlers. Per John's decision, D1 (OpenRouter key in the launch response + open CORS) is left as is: the only threat is a local process/page on the user's own machine, which we don't consider worth added complexity. F1/F6/F7 left as is. Gate: eslint clean; vitest 498 passed; full BloomTests 2828 passed, with one pre-existing failure (ToCss_RoundedBorderTheme_LeavesPageNumberDefaultsOnPage- NotPseudo) that also fails on the clean merge base and is unrelated to these changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
hatton
commented
Jul 9, 2026
hatton
commented
Jul 9, 2026
| { | ||
| src: "node_modules/bloom-ai-image-tools/dist-app/*", | ||
| dest: "./aiImageEditor/", | ||
| }, |
Member
Author
There was a problem hiding this comment.
[Claude Opus 4.8 relaying Devin]
[Devin] Investigate: Vite static copy glob for AI editor dist-app may need adjustment when package is published
The glob pattern node_modules/bloom-ai-image-tools/dist-app/* (in the structured:false block) — a typical Vite app build produces an assets/ subdirectory alongside index.html. Whether * copies subdirectories recursively depends on vite-plugin-static-copy / fast-glob directory handling. The existing bloom-player copy uses the same pattern and works, suggesting directories are handled. Verify when bloom-ai-image-tools is actually published and added as a dependency: if its dist-app/ has an assets/ subdirectory, confirm output/browser/aiImageEditor/ is complete.
Member
Author
|
[Claude Opus 4.8] Consulted Devin on 2026-07-09 up to commit c22c67f. Result: 0 bugs, 2 Investigate flags (orphaned ai-* file accumulation; vite dist-app copy glob to verify on publish) — both posted as inline threads above. 7 informational items skipped. |
hatton
changed the title
…cher fixes (BL-16523) - Add "Edit with AI..." context-menu string (BloomMediumPriority.xlf). - Pin bloom-ai-image-tools to BloomBooks/bloom-ai-image-tools#dist-v0.1.0. - Current-page replacement matching: match by ordinal slot (incomingId) within same-filename groups, handle encoded-vs-decoded filenames, and switch to changeImageByElement (export GetRawImageUrl for reuse). - AiImageEditorApi: guard sessions against a book switch (_sessionBookFolderPath), fail launch clearly when the editor app isn't in the build, stream file uploads to a temp file and swap (never leave a truncated/stale file; empty body writes an empty file), and name new images with GetUnusedFilename. Add XML doc comments. - BloomArchiveFile: narrow the archive skip to just .ai-image-editor rather than all dot-folders, so legitimate dot-folders (e.g. widget .well-known) are still bundled; update the test accordingly. - Dev launcher: cap the editor build at 5 min, add a 120s staging grace so a slow stage never blocks Bloom startup, and skip re-staging via a .staged-package-version marker. Drop the unverified react-grid-layout dev entry (left a NOTE). - Playwright: give the bloom-exe CDP suite its own defineConfig importing from playwright/test (shared instance), broaden testDir with testMatch/testIgnore, and add a bloom-exe-ai-editor-open smoke test proving the editor iframe boots. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
Author
|
[Claude Opus 4.8] Consulted Devin on 2026-07-10 up to commit |
# Conflicts: # ReadMe.md # src/BloomBrowserUI/package.json # src/BloomBrowserUI/yarn.lock
… (BL-16523) F1 (Devin G1): On commit, delete superseded ai-image* files that nothing references any more. Each off-page image replacement writes a fresh ai-image<N> file and repoints the slot, leaving the previous file behind; without cleanup, repeatedly AI-editing a slot piled up orphaned files in the book folder. After the book is saved we scan the whole book DOM (pages + data-div) for still-referenced image file names and delete only our own generated files (name starts with "ai-image") that are no longer referenced. Current-page slots are excluded: the front-end repoints those via changeImage(), so their old files are not ours to delete here. A file shared by another slot, a user's original image, or a placeholder is never touched. F3: Drop translate="no" on EditTab.Image.EditWithAI so the "Edit with AI..." context-menu item is localized like its sibling image menu items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
Author
|
[Claude Opus 4.8] Consulted Devin on 2026-07-10 21:05Z up to commit |
…orphan cleanup (BL-16523) Applies the preflight decisions on Devin's re-review findings: CORS bug (Devin): in HandleFile, answer the OPTIONS/CORS preflight before the session gate. A preflight request carries no "session" param, so the prior order made HasValidSession 401 it and the OPTIONS branch (later in the switch) was dead code — breaking cross-origin fetches from the editor's own Vite dev server in the LINKED (BLOOM_AI_EDITOR_URL) HMR workflow. The shipped product serves the editor same-origin, so end users were unaffected. F4 (OAuth reflection): behavior left as-is (already mitigated) with a comment documenting why the reflected `error` value is safe — ReplyWithText responds as text/plain, which browsers don't parse as HTML, so there's no reflected XSS — and warning against switching to an HTML response without encoding. F1 test: refactor DeleteSupersededAiImageFiles / CollectReferencedImageFileNames to take (bookFolderPath, HtmlDom) and make them internal, then add AiImageEditorApiTests covering: an unreferenced ai-image file is deleted; files still referenced by another slot or the data-div are kept; a user's non-ai-image original is never deleted; and empty candidates are a no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
hatton
commented
Jul 10, 2026
| <note>ID: Feature.AiImageEditing</note> | ||
| <note>Name of the AI image editing feature, used in subscription messages.</note> | ||
| </trans-unit> | ||
| <trans-unit id="EditTab.Image.EditWithAI"> |
Member
Author
There was a problem hiding this comment.
[Claude Opus 4.8 relaying Devin] Devin flags this as missing the repo's default translate="no" marker for new strings. This is intentional, not an oversight: the AI Image Editor is being prepared for release, and the maintainer explicitly decided (preflight decision F3) to make this user-facing context-menu item ("Edit with AI…") localizable like its sibling image menu items (e.g. EditTab.Image.ChooseImage). The xlf-strings convention is "mark new entries translate=no unless instructed otherwise" — this is the otherwise. The two sibling AI strings that are still experimental-only (CollectionSettingsDialog.AdvancedTab.Experimental.EditWithAI, Feature.AiImageEditing) deliberately keep their translate="no".
Member
Author
|
[Claude Opus 4.8] Consulted Devin on 2026-07-10 21:56Z up to commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[Claude Opus 4.8]
Adds the AI Image Editor feature to Bloom (experimental, behind the
AiImageEditingfeature flag):AiImageEditorApithat launches an in-app editor iframe, shares the whole-book image list, persists editor state/history under a per-book.ai-image-editorfolder, and commits image replacements book-wide.OpenRouterCredentialStore, DPAPI-protected) so keys never travel with a book.output/browser/aiImageEditor/, built from the localbloom-ai-image-toolscheckout by thego.shlauncher (aiEditorBuild.mjs), withgo.sh --withsupport for developing local library checkouts alongside Bloom.Draft opened by the preflight workflow. See the decision report (delivered separately) for items awaiting a human decision — notably the CORS/credential-exposure surface and the merge with
master.Ref: (no YouTrack id in branch name)
This change is
Devin review