Image analysis for Mantella Spell - #67
Open
YetAnotherModder wants to merge 3 commits into
Open
Conversation
Add new MCM option to initiate image analysis Add new variables to MantellaRepository to track vision variables Add new checks to MantellaConversation to manage vision and send variables to MantellaSoftware Add a new MantellaVisionScript to execute vision calls (requires SUP_SKSE) Add new MantellaConstants to managevision
New esp dependency for SK desktop if using vision : SUP_SKSE MantellaVisionScript.psc : checks the game version and only performs the screenshot capture if it's a desktop version. Added an option to temporarily remove the menu while taking the screenshot. Mantellarepository: New properties added for new functionalities. New keybind function for the vision hotkey MantellaMCM_VisionSettings.psc : New checks to only display desktop related options if Skyrim desktop is running. New options to keybind the vision hotkey & choose the resize resolution of the image MantellaMCM.psc : new properties, function calls to VisionSetttings and oid onhighlight descriptions. MantellaConversation : sends a new variable to MantellaSoftware to tell it what resolution to use when resizing the image.
General Add support for vision analysis for Skyrim desktop version after 1.6.640 MantellaMCM_VisionSettings Add dynamic version check that will only enable available options specific to the game's version. Add vision 'hints' that allows Skyrim to scan the loaded game cell and send hints on who might be featured in the screenshot Add a distinction between vision analysis (thru SUP_SKSE) and steam vision analysis Add support for a vision Hints hotkey for desktop users that are using Steam. Add a debug mode and a way to force Mantella to consider Skyrim a specific version (for debugging purposes) MantellaVisionScript Add a function to scan cell for actors and reset Hints strings (formatted as arrays). MantellaRepository Add new vision variables to support hints, steam and debug mode. Add new hotkey support for vision hints MantellaMCM Add new support for option IDs for hints, debug mode, forced Skyrim version and hint hotkey selection. MantellaConversation New function to build hint values in custom context : BuildVisionHintsValues() Refactored the custom context code slightly to account for different vision options and game versions MantellaConstants Add new constants for custom context value to support vision analysis Here's what the vision analysis process should look like depending on the game's version : Mantella software should auto-detect which way is the preferred way to proceed, but the user can override that by enabling debug mode. If Skyrim VR 1. Enable Enable Steam vision in Mantella's MCM 2. Users take a steam screenshot using the button combination defined in the Steam preferences 3. When the user is done speaking/writing a flag will be sent to Mantella Software, Mantella Software will automatically check if a screenshot exists in the specific Steam folder and will compare that screenshot to a user specified delay before sending it to the LLM for analysis. The delay is there to avoid sending older screenshots and confusing the user and LLM. If hints are allowed, when the user is done speaking Mantella.esp will automatically scan for probable NPCs in the player's field of view. The scanning step is done when the user is done speaking since there's currently no way to recognize when a steam screenshot gets taken for Skyrim VR directly in the game enging. If Skyrim desktop without SUP_F4SE (so more recent versions of Skyrim than 1.6.640) 1. Enable Enable Steam vision in Mantella's MCM 2. Users take a steam screenshot using the button defined in the Steam preferences. Additionally, the user may defined a button for vision hints and bind it to the same button as the Steam screenshot buttons to help the LLM identify NPCs later in the process. 3. When the user is done speaking/writing a flag will be sent to Mantella Software, Mantella Software will automatically check if a screenshot exists in the specific Steam folder and will compare that screenshot to a user specified delay before sending it to the LLM for analysis. The delay is there to avoid sending older screenshots and confusing the user and LLM. If hints are allowed, Mantella.esp will try to send those to Mantella Software as well if they are present. If Skyrim desktop with SUP_F4SE Automatic vision analysis 1. Enable Enable SUP_SKSE vision in Mantella's MCM 2. When the user is done speaking/writing a flag will be sent to Mantella Software, Mantella Software will automatically check if a screenshot exists in the folder containing the Skyrim executable before sending it to the LLM for analysis. If hints are allowed, Mantella.esp will try to send those to Mantella Software as well if they are present. Hotkey screenshot analysis 1. Users take a steam screenshot using their button defined in the Steam preferences. If hints are allowed, Mantella.esp will try to send those to Mantella Software as well if they are present. Note : there's no need to enable the automatic vision analysis, just taking a screenshot using the MCM hotkey will do the trick. 2. When the user is done speaking/writing a flag will be sent to Mantella Software, Mantella Software will automatically check if a screenshot exists in the folder containing the Skyrim executable before sending it to the LLM for analysis. If hints are allowed, Mantella.esp will try to send those to Mantella Software as well if they are present.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New optional dependency for SK desktop if using vision : SUP_SKSE (1.6.640 and lower only, for other users Steam can be used)
### New MCM options
Mantella software will auto-detect which way is the preferred way to proceed, but the user can override that by enabling debug mode.
Skyrim Desktop (Steam overlay only)
Enable Steam screenshot analysis: This will make conversation.psc send a vision_ready flag when the player is done speaking/writing to Mantella Software so that it searches automatically for a screenshot in the appropriate steam VR folder taken in the last two minutes. The screenshot has to be taken by the player by using the Steam overlay shortcut (usually F12 on desktop). If it doesn't find any nothing happens, if it finds one the screenshot is sent to the LLM for analysis and also flagged for deletion at the end of the conversation.
Enable hints : Give the names and approximate distance of the NPCs displayed in the screenshots sent to the LLM.
Mantella Hints Hotkey : Pressing this hotkey gives the names of the NPCs displayed in the screenshots sent to the LLM, it is recommended to set it to the same key as your screenshot capture key
Resolution : Choose between "auto", "high, "low", these values are sent to the LLM so it either choose itself the resolution to use or force it into "high" mode (less loss, higher token and processing time) or "low" mode (more loss, less processing time and token use) ( See here: https://platform.openai.com/docs/guides/vision
Resize value: Similar to resolution but the resize is done on MantellaSoftware side in python. From my experience it's much faster to do it this way than let the LLM handle it. Even with 512x pixel width image the LLM is still pretty good at spotting details and reading text.
Skyrim Desktop (1.6.640 and other SUP SKSE compatible version only):
Enable automatic vision analysis : Will automatically take a screenshot at the end of player speech or text input and save it in the root folder of Skyrim as Mantella_Vision.jpg
Mantella Screenshot Hotkey : This hotkey will allow the player to take a screenshot during conversation and raise a flag in the repository so that MantellaConversation.psc knows to send custom keys to the LLM to search for a Screenshot.
Hide interface during screenshot : Very briefly hide the interface when using the screenshot hotkey.
Resolution : Choose between "auto", "high, "low", these values are sent to the LLM so it either choose itself the resolution to use or force it into "high" mode (less loss, higher token and processing time) or "low" mode (more loss, less processing time and token use) ( See here: https://platform.openai.com/docs/guides/vision
Resize value: Similar to resolution but the resize is done on MantellaSoftware side in python. From my experience it's much faster to do it this way than let the LLM handle it. Even with 512x pixel width image the LLM is still pretty good at spotting details and reading text.
Enable hints : This will automatically gives the names of the NPCs displayed in the screenshots sent to the LLM.
Skyrim VR Only :
Enable Steam screenshot analysis: This will make conversation.psc send a vision_ready flag when the user is done speaking/writing to Mantella Software so that it searches automatically for a screenshot in the appropriate steam VR folder taken in the last two minutes. The screenshot has to be taken by the player by using the Steam overlay shortcut (usually menu+trigger). If it doesn't find any nothing happens, if it finds one the screenshot is sent to the LLM for analysis and also flagged for deletion at the end of the conversation.
In VR if the hints are used they will only be scanned when the player is done speaking/writing since there’s currently no way to check when a Steam screenshot is taken.
### How conversation.psc changes work:
If conversation.psc detects that there is a pending screenshot or that VR vision is activated, it will then send custom json values to the MantellaMCM that tells it to search for a screenshot in the appropriate place and send it for vision analysis
### Potential upgrades:
Add a way to take screenshots in VR without having to rely on the steam overlay functions.
Harmonize all the different screenshot taking methods by implementing a custom DLL that handle screenshots.