vt-good-image-protocol.tex

% vim:cc=80:cole=0
\documentclass[a4paper]{article}
%\documentclass[a4paper, 20pt]{extreport}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}

\usepackage{colortbl}
\usepackage{todonotes}

\usepackage[hidelinks]{hyperref}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,
    urlcolor=cyan
}

% \usepackage{fancyhdr}
% \usepackage{graphicx}

\usepackage{draftwatermark}
\SetWatermarkText{Draft}
\SetWatermarkScale{4}

\usepackage[english]{babel}

\usepackage{xcolor}
\definecolor{light-gray}{gray}{0.95}

\usepackage{geometry}
\geometry{legalpaper, margin=1in}

\title{VT Good Image Protocol \\
a standardization proposal}
\author{Christian Parpart}
% \date{2020-12-18 (draft, revision 0)}
\date{\today \quad (draft, revision 0)}

\newcommand{\code}[1]{\colorbox{light-gray}{\texttt{#1}}}

\newcommand{\DA}{\code{DA1}}
\newcommand{\DECRQM}[1]{\code{CSI ? #1 \$ p}}
\newcommand{\DECRST}[1]{\code{CSI ? #1 l}}
\newcommand{\DECSET}[1]{\code{CSI ? #1 h}}

\newcommand{\GoodImageProtocol}{\code{Good Image Protocol}}

\begin{document}

\maketitle

\tableofcontents

%\newpage

\section{Motivation} % {{{

For many decades Sixel and ReGIS have been the only image protocols for terminal emulators.
While both are ancient and not even widely implemented, now people are used
to seeing images and even emoji everywhere.
Those people may eventually touch a virtual terminal emulator and expect it to be no different
than what they are used to from other software systems.

There is a growing interest in displaying images in todays
virtual terminal emulators.

Since these protocols are were not designed with todays needs in mind,
virtual terminal emulator developers have started implementing their own
proprietary protocols for displaying images as Sixel and related
protocols were simply not fullfulling today's needs anymore.

While recent efforts to image protocols are positive, it also leads to fragmentation in the ecosystem
because application developers do not know what to support in case they intend
to display images in their client applications, and even new terminal emulator
developers don't know what protocol to implement.

There is also a recent resurrection of the Sixel bitmap protocol
to be implemented by some terminal emulator
\footnote{Alacritty, Contour, gnome-terminal, iTerm2, wezterm, xterm.js}
and library \footnote{notcurses, libsixel} developers,
however, none of them is 100\% conforming to how Sixel used to work 50 years ago,
as there is no easy way to access the hardware they were implemented on
and also xterm does not implement every aspect of Sixel while existing
documentation on sixel is also leaving too much room for varying interpretation.

This specification attempts to unify all those image protocols and addresses
some of the issues, not as a superset, but rather as a largest common denominator,
with usability, simplicity, and implementation adaptability in mind and
is also an attempt to formalize what was written down in \ref{ref:twg-gip},
which can be seen as a conclusion of many discussions that happened scattered
across different forums on the topic of a future image protocol.

% }}}
\section{Terminology} % {{{
\begin{itemize}
    \item \textbf{screen cell}; a rectanglular area of the screen that can contain a character with
        its graphics rendition or an image fragment
    \item \textbf{image fragment}; a rectanglular tile of an image that perfectly fits into a screen cell
    \item \textbf{image storage pool}; a backing store for images received for future operations
    \item \textbf{image layer}; determines how an image fragment relates to the text in the same screen cell:
        \begin{itemize}
            \item \textbf{BehindText} (0) --- the image renders behind text, replacing the cell's SGR background
            \item \textbf{ReplaceText} (1) --- the image replaces the text glyphs (default)
            \item \textbf{AboveText} (2) --- the image renders above text as an overlay
        \end{itemize}
\end{itemize}

% }}}
\section{Prior art and current state} % {{{

Prior art of image protocols for terminals are as follows:

\subsection{Tektronix}

\begin{itemize}
    \item First appeared in 1970s
    \item Tektronix 4010 series was a family of text-and-graphics computer terminals based on storage-tube technology created by Tektronix.
    \item implemented by: xterm
\end{itemize}

\subsection{DEC ReGIS graphics}

\begin{itemize}
    \item First appeared in VT125, in July 1981
    \item vector graphics based
    \item implemented by: xterm
\end{itemize}

\subsection{Sixel graphics}

\begin{itemize}
    \item Sixel grpahics first appeared in LA50 dot-matrix printers (1983) and short after in the VT240 in October 1983.
    \item It is pixel based and positions the graphics cursor in a way it is most optimal for dot printers,
        that could print 6 pixels in height per line.
    \item Specialized image format optimized for 6-pin printers.
    \item Supports scrolling the screen if the image does not fit the current screen.
    \item Supports passing different aspect ratio. This is rarely implemented by virtual terminal emulators though.
    \item Supports endless vertical rendering, e.g. to print seismograph data (rarely implemented by virtual terminal emulators).
    \item Implemented by: xterm, mlterm, contour, wezterm, vte, possibly others.
\end{itemize}

This protocol only understands pixels and thus is dependant on the
configured font size and screen resolution of the terminal, which
terminal emulators seem to have solved differently and therefore cannot
be relied on.

\subsection{Terminology image protocol}

Terminology's image protocol is rather a media transport protocol as it
can also be used to render animations, including videos.

\begin{itemize}
    \item First appeared in june, 2012.
    \item Uses vt sequences inspired by xterm's sequence extensions.
    \item Specification is in the source code repository's \texttt{readme.md} file.
    \item Implemented by: terminology
\end{itemize}

\subsection{iTerm2 image protocol}

\begin{itemize}
    \item First introduced in 2014 with documentation added in 2015.
    \item Can associate a file name (Base64 encoded) to the rendered image.
    \item Image display size can be pixel based as well as cell based.
    \item Also has the ability to address the width and height relative to the terminal window size.
    \item Image data is Base64 encoded.
    \item Any image format that macOS supports is supported by iTerm2's image protocol,
        including PDF, PICT, EPS, or any number of bitmap data formats (PNG, GIF, etc.).
    \item implemented by: iterm, wezterm (partly)
\end{itemize}

The dependency on macOS for supported image formats makes it a little blurry
specification also.

\subsection{DomTerm's inline image support}

\begin{itemize}
    \item Domterm's image support appeared in 2016.
    \item No explicit image protocol but a "insert html"-protocol that does support images.
    \item Specification is in the source code repository's \texttt{readme.md} file.
    \item Implemented by: DomTerm
\end{itemize}

The typical way to insert an image is to insert an HTML \texttt{<img>} element,
with the src attribute using a data: URL scheme,
with the actual data inline as base64.

Any image format supported by the underlying browser engine is allowed.

The \texttt{<img>} element can contain attributes such as width, height, alt
and options to add scrollbars.

The tight integration to the browser makes it hard to be adopted by native
terminals.

\subsection{Kitty image protocol}

\begin{itemize}
    \item First appeared in 2017
    \item Seperates image upload from image display
    \item Can mandate image sizes in pixels as well as in screen cell dimensions.
    \item Images are rendered independantly of text with a specified Z-axis.
    \item Images can be deleted without knowning their names but only by specifying X/Y/Z axis
          and delete all images intersecting these criteria.
    \item Image formats supported are: RGB, RGBA, PNG.
    \item Image data is Base64 encoded.
    \item Supports optionally compressing the image data.
    \item Supports various image data transmission methods:
        \begin{itemize}
            \item inline, as part of the VT sequence
            \item named shared memory
            \item path to local files
        \end{itemize}
    \item Optionally deletes the local file when requested and the file path is in a secure location.
    \item Hard coded image storage pool to 320 MB (per screen buffer).
    \item implemented by: kitty, wezterm (partly)
\end{itemize}

None of the above image protocols are a good candiate. iTerm2's image protocol is very simple and
can be rendered by specifying screen cell sizes instead of relying on pixels, but lacks some
flexibility.

Kitty image protocol on the other hand does provide very good flexibility but is way too complex which
would hinder broader adoptability and conforming implementations, potentially leading to partial
implementations and thus fragmentation.
There is also an added complexity (such as support for Z-axis) and the ability
to send images in a way that does not work trivially when the terminal is
connected to remote clients (such as via SSH). This all seems to hinder
adopting a otherwise very flexible image protocol.

The \GoodImageProtocol specification aims to be future proof with regards to
the young generation of terminal users, as well as being simple enough
to be widely accepted and adopted on the terminal side as well as client side,
yet allowing future extensions to this protocol
without breaking compatibility to existing implementations.

% }}}
\section{Requirements} % {{{

This section lays down the principles and requirements of \GoodImageProtocol.

\begin{enumerate}
    \item \textbf{feature detectability} - the image protocol must be easily detectable with well known techniques.
    \item \textbf{protocol forward-compatibility} - the protocol must be easily extendable in input
        parameters as well as reusability of the image storage pool
        (such as icon-display, desktop-notifications, background images).
    \item \textbf{multiplexer-aware} - multiplexers must be able to fully operate without analyzing the image format.
        it must be possible to blindingly forward the image data to the outputting terminals.
    \item \textbf{deterministic emulation} - the image rendering must not be affected by font size
        or similar properties nor on external api to make productive use of this protocol.
    \item \textbf{synchronous operations by default} - no asynchronous operations unless explicitly requested.
    \item \textbf{preserve aspect ratio by default}
    \item \textbf{remote-terminal capable} - no dependency on the local host, such as the local file system.
    \item \textbf{upload and render seperation} - image upload must be decoupled from image display
    \item \textbf{support for rendering tiles of an image} - adding the ability to only render tiles
        of an image greatly assists client applications when only rendering parts of an image is
        needed (such as in multiplexers).
\end{enumerate}

\paragraph*{}

The document at [\ref{ref:twg-gip}] suggests cell based masking for rendering to help rendering more
complex scenes such as in windowed TUI applications or terminal multiplexers with overlaying contents.
However, the same can be achieved with a sequence of Image-Render commands specifying a
sub-rectangle (tiles) to be rendered of the referenced image.

% }}}
\section{Backwards Compatibility} % {{{

Since all other image protocols are pixel based, the proposed image protocol does not
attempt to retain any backwards compatibility. Instead, the goal is to create
an image protocol that is future proof with todays needs in mind.

% }}}
\section{Future Compatibility and Stability} % {{{

In order to leave room for improvements, the VT sequences should be designed in a way
that allows specifying additional parameters in the future, and simultaneously ensures
that older implementations can still work while safely ignoring any new parameters.

% }}}
\section{Terminal Emulator Requirements} % {{{

This is the list of resource requirements that must be guaranteed
by the virtual terminal emulator.

\begin{enumerate}
    \item at least 64 images concurrently
    \item at least 4 MB per image uncompressed (RGBA8888)
\end{enumerate}

Upper limits must be present for security reasons but can be varying by implementation.

% }}}
\section{Performance Considerations} % {{{

This section provides guidance for implementers on achieving good performance.

\begin{itemize}
    \item \textbf{Base64 decoding} ---
        Base64 expands payload by approximately 33\%.
        Implementations should decode incrementally during DCS passthrough
        rather than buffering the entire encoded body before decoding.

    \item \textbf{Image decoding} ---
        Decoding compressed formats (such as PNG) is CPU-intensive.
        Implementations should consider performing decoding off the main parser thread
        to avoid stalling input processing.

    \item \textbf{Fragment generation} ---
        Generating per-cell RGBA fragments from the source image can be expensive
        for large images spanning many cells.
        Implementations should generate fragments lazily (on demand at render time)
        rather than eagerly during the render command.

    \item \textbf{GPU texture caching} ---
        Rasterized image fragments should be cached as GPU textures,
        keyed by image identity and cell coordinates.
        Avoid re-uploading unchanged textures every frame.

    \item \textbf{Upload/render separation} ---
        The two-phase design (upload then render) is intentionally performance-friendly.
        A single image upload can serve many render commands
        without retransmitting image data.

    \item \textbf{Reference counting} ---
        Reference count operations (increment on cell placement, decrement on cell clear)
        should be O(1) to avoid degradation with large images.

    \item \textbf{Eviction cost} ---
        Eviction should be O(1) amortized.
        Both FIFO and LRU are acceptable strategies.

    \item \textbf{Multiplexer forwarding} ---
        Multiplexers forward DCS sequences verbatim without inspecting image data.
        This keeps multiplexer overhead constant regardless of image payload size.

    \item \textbf{Batch rendering} ---
        Terminals with GPU-accelerated rendering should batch image fragment draw calls
        per frame rather than issuing individual draws per cell.

    \item \textbf{Memory budget} ---
        The minimum requirement (64 images at 4 MB) implies a 256 MB baseline.
        Implementations may keep compressed formats (such as PNG) in compressed form
        until fragment generation to reduce resident memory.
\end{itemize}

% }}}
\section{Security Considerations} % {{{

\begin{itemize}
    \item image upload spamming - should be avoidable by a meaningful limit
        on how many images can be uploaded at the same time.
        Exceeding the limit will result in evicting ideally older images,
        leading to dangling image fragments, as their related image has been
        evicted. A placeholder or an empty screen cell may be displayed instead or no image at all.
    \item huge image uploads - must be avoidable by limiting the number
        of pixels an image can take.
    \item invalid image uploads - a terminal proxy (such as tmux) do not
        implicitly need to validate image formats as they are usually just
        forwarding the blobs to the connected terminal. A non-proxying
        terminal however will validate implicitly by loading the specified
        file format.
    \item invalid image render requests (such as out of bounds in offsetted-renders) -
        invalid named images will not be displayed.
        Overextending dimensions will be displayed but result in cut-of images.
    \item excessive render dimensions - terminals should clamp
        \textbf{screen-rows} and \textbf{screen-cols} to the current page size.
        Render requests exceeding page dimensions are clipped, not rejected.
        This prevents resource exhaustion from requests such as \code{r=10000,c=10000}.
\end{itemize}

% }}}
\section{Semantics} % {{{
\subsection{The lifecycle of an image} % {{{

While classically it should be allowed to just display images,
with having the cursor moved, and forget about that one has rendered that
image, it must also be possible to render the same image multiple times
in an resource efficient time.

Therefore, the concept of image display is broken up into multiple parts,
by separating the image upload from rendering, so
that the same image data can be reused in multiple render commands with
the choice of either drawing the full image or just parts of it.
The latter is important for toolkit developers to efficiently redraw
patially overlapping images.

\subsubsection*{Uploading the image} % {{{

The command for uploading the image simply uploads the image.
In order to make use of that previously uploaded image, a application-specific
fully qualifying (and ideally unique) name is being passed.
This name can be used in future commands to render, rerender the image
either in its whole or parts of it.

The link between the name and the uploaded image can be explicitly released,
and then be reused by other image uploads while prior image uploads to that
name are preserved.

The image upload does not generate any visible output.
If the \textbf{request-status} parameter is set,
the terminal replies with a status code indicating success or failure.

% }}}
\subsubsection*{Rendering the image} % {{{

Rendering the image is done by referencing the previously uploaded
image by its name and then telling the command where to render it
in screen coordinates, with number of columns and number of lines to fill
with that image. This can be seen similar to the `DECFRA` VT sequence
that also fills a rectangular region with a fixed SGR, except
that this command filles the rectangular area with an image instead.

% }}}
\subsubsection*{Releasing the image} % {{{

When the image's referenced name is not needed anymore it must be released.
This does not implicitly erase already rendered screen areas referencing the image.
It purely disassociates the name with the image, eventually leading
to resource eviction when no screen cell is holding a reference to the
uploaded image anymore.

% }}}
\subsubsection*{Rendering the image only once} % {{{

While the above is sufficient to efficiently operate with images in the terminal,
there are usecases where an image needs to be rendered exactly once and
seperating upload, render, and release stages is therefore unnecessary,
such as simply putting the image onto the screen and have the cursor moved
accordingly, so that it perfectly fits into the command flow of your primary screen.

\textbf{iTerm2} is having such a special command, called \code{imgcat},
there is also \code{lsix} to display images in the spirit of \code{ls},
and there are similar tools for kitty's and terminology's protocol, too.

\GoodImageProtocol must provide a command to make it trivial for application
developers to achieve the same.

% }}}

% }}}
\subsection{Feature Detection} % {{{

\DA is already used to advertise terminal features, including Sixel graphics, and thus,
\DA is also used to advertise \GoodImageProtocol support, for consistency.

There is some improved feature detection specification work ongoing,
so there may be other ways to detect \GoodImageProtocol in the future, when that is ready.

% }}}
\subsection{Querying Resource Limits} % {{{

Resource limits must be queryable. Every resource requirement also has a limit
at exactly the value of the requirement or above.

The following resource limits are reported:

\begin{itemize}
    \item Maximum number of concurrently stored images (visible or not visible).
        A recommendation for this limit is 128 on terminals running on desktop computers.
    \item Maximum bytes per image (uncompressed RGBA).
    \item Maximum image width in pixels.
    \item Maximum image height in pixels.
\end{itemize}

The terminal responds to a query with all resource limits in a single reply.
See the Syntax section for the query and response sequence format.

% }}}
\subsection{Upload Image} % {{{

Uploads an image for future render operations.

\subsubsection*{Parameters}

\begin{tabular}{|m{3cm}|m{12cm}|}
  \hline
  \textbf{parameter name} & \textbf{description} \\
  \hline
  \textbf{name}           & a unique identifier for the uploaded image \\
  \textbf{format}         & a value that determines the image format. \\
                          & See section \ref{sec:supported-image-formtats} for available image formats \\
  \textbf{width}          & optional pixel width of the given image
                            (only required if this information is not provided
                            inline via the \textbf{data} field) \\
  \textbf{height}         & optional pixel height of the given image
                            (only required if this information is not provided
                            inline via the \textbf{data} field) \\
  \textbf{data}           & image data in the specified input format \\
  \textbf{request-status} & if set, the terminal sends a response indicating success or failure \\
  \hline
\end{tabular}

\subsubsection{Image Name Constraints}

Image names must consist of ASCII alphanumeric characters (\code{a-z}, \code{A-Z}, \code{0-9})
and underscores (\code{\_}). Names must be between 1 and 512 characters in length.
Names that do not conform to these constraints result in the command being silently ignored.

\subsubsection{Data Validation}

For RGB and RGBA formats, the image data size must equal
\code{width * height * bytes-per-pixel} (3 for RGB, 4 for RGBA).
If the data size does not match the declared dimensions, the upload is invalid.
When \textbf{request-status} is set, the terminal responds with error code 2 (invalid image data).
Otherwise the command is silently ignored.

For Auto format, the terminal first attempts to detect a self-describing format
(such as PNG via its magic bytes). If no known format is detected and width/height
are provided, the data size is used to determine whether the payload is RGB or RGBA.
If the format cannot be determined, the data is treated as invalid.

\subsubsection{Idempotency}

Image upload can be implemented to be idempotent, but doesn't have to,
i.e. the storage pool keeps an internal hash of each image that is
automatically constructed upon image upload.

If the image was already uploaded, that image's reference count is incremented
and in case it is a named resource, that one will point to that existing one.

% }}}
\subsection{Render Image} % {{{

Renders an image that has been previously uploaded.
The image is rendered with its top-left corner at the current cursor position.
Applications should use standard cursor positioning sequences (such as CUP) to place
the cursor before issuing a render command.
By default, the cursor is not moved by this operation.
When the cursor is not moved, it remains at the position it occupied
before the render command. Applications are responsible for repositioning
the cursor (e.g.\ via CUP) to avoid overwriting rendered image cells with
subsequent text output.

If the \textbf{update-cursor} parameter is set,
the cursor is moved to column~1 (or the left margin if \code{DECOM} is set)
of the line immediately below the last rendered image row.

\subsubsection*{Parameters}

\begin{tabular}{|m{3cm}|m{2cm}|m{10cm}|}
    \hline
    \textbf{parameter name}   & \textbf{default} & \textbf{description} \\
    \hline
    \textbf{name}             &                  & unique identifier referencing a previously uploaded image
                                                    (see Upload Image for naming constraints) \\
    \textbf{screen-rows}      &                  & number of screen cells to render vertically \\
    \textbf{screen-cols}      &                  & number of screen cells to render horizontally \\
    \textbf{screen-layer}     & ReplaceText      & render layer: BehindText (0), ReplaceText (1, default), AboveText (2) \\
    \textbf{resize-policy}    & noresize         & optional, mandates how to resize the image within the screen cells \\
    \textbf{alignment-policy} & middlecenter     & optional, mandates how to align the image within the screen cells \\
    \textbf{image-x-offset}   & 0                & pixel x-offset into the source image data (default: 0) \\
    \textbf{image-y-offset}   & 0                & pixel y-offset into the source image data (default: 0) \\
    \textbf{image-width}      & full width       & width in pixels of the sub-region of the source image to display (default: full image width) \\
    \textbf{image-height}     & full height      & height in pixels of the sub-region of the source image to display (default: full image height) \\
    \textbf{auto-scroll}      & false            & boolean, if set and the image would extend below the visible page area,
                                                    the screen scrolls up to accommodate.
                                                    Defaults to false (no scrolling; the image is clipped at the bottom margin). \\
    \textbf{update-cursor}    & false            & optional, tells the terminal to move the text cursor after having the image rendered (image left aligned) below the image \\
    \textbf{request-status}   & false            & request a success/failure status response code from the terminal \\
                              &                  & if value is true, a response is sent back to the application \\
                              &                  & to indicate success or failure. \\
                              &                  & if value is set to false (or parameter is not present), \\
                              &                  & no response is sent back to the application. \\
    \hline
\end{tabular}

\paragraph*{}
At least one of \textbf{screen-rows} or \textbf{screen-cols} must be specified
with a value greater than zero. If one is specified and the other is omitted (or zero),
the missing dimension is automatically determined by preserving the image's aspect ratio.
If both are omitted or zero, the sequence is invalid and silently ignored.

\paragraph*{}
Rendered images that produce a padding due to alignment- and resize-policy
will fill the gap with the currently active sgr background color.

\paragraph*{}
Reverse video mode mode (\code{decscnm}) will affect the padding color but not the image.

\subsubsection*{Resize Policy}

\begin{tabular}{ |c|l|l| }
    \hline
    \textbf{Value} & \textbf{Name}          & \textbf{Description} \\
    \hline
    0 & NoResize      & Does not perform any resize, leading to a padding on one or two sides \\
      &               & as mandated by the alignment-policy (default). \\
    1 & ResizeToFit   & Resizes the image to fit the specified screen width and height, \\
      &               & leading to a padding on zero or one sides as mandated by the alignment-policy. \\
    2 & ResizeToFill  & Resizes the image to fill the specified screen width and height, \\
      &               & preserving the aspect ratio, leading to potential clipping depending on the \\
      &               & alignment-policy. \\
    3 & StretchToFill & Resizes the image to fill the specified screen width and height, \\
      &               & ignoring the aspect ratio, ignoring alignment-policy. \\
    \hline
\end{tabular}

\subsubsection*{Alignment Policy}

\begin{tabular}{ |c|l|l| }
    \hline
    \textbf{Value} & \textbf{Name}          & \textbf{Description} \\
    \hline
    1 & TopStart      & The image is aligned vertically on the top and horizontally on the left. \\
    2 & TopCenter     & The image is aligned vertically on the top and horizontally in the center. \\
    3 & TopEnd        & The image is aligned vertically on the top and horizontally on the right. \\
    \hline
    4 & MiddleStart   & The image is aligned vertically centered and horizontally on the left. \\
    5 & MiddleCenter  & The image is aligned vertically and horizontally in the center (default). \\
    6 & MiddleEnd     & The image is aligned vertically centered and horizontally on the right. \\
    \hline
    7 & BottomStart   & The image is aligned vertically on the bottom and horizontally on the left. \\
    8 & BottomCenter  & The image is aligned vertically on the bottom and horizontally in the center. \\
    9 & BottomEnd     & The image is aligned vertically on the bottom and horizontally on the right. \\
    \hline
\end{tabular}

% }}}
\subsection{Upload and Render Image} % {{{

This uploads and renders the image via a single VT sequence. Therefore no image Id
is required and there won't be any way to reference that image after either.

The parameters of this function is the sum of the image upload and image render parameters excluding the unique
identifier and excluding sub-rectangle rendering parameters.

The image is rendered with its top-left corner at the current cursor position.
By default, the cursor is not moved by this operation.
If the \textbf{update-cursor} parameter is set,
the cursor is moved to the beginning of the line below the last rendered image row.

\subsubsection*{Parameters}

\begin{tabular}{|m{3cm}|m{2cm}|m{11cm}|}
    \hline
    \textbf{parameter name} & \textbf{default} & \textbf{description} \\
    \hline
    \textbf{screen-rows}      &                  & number of screen cells to render vertically \\
    \textbf{screen-cols}      &                  & number of screen cells to render horizontally \\
    \textbf{resize-policy}    & noresize         & optional, mandates how to resize the image within the screen cells \\
    \textbf{alignment-policy} & middlecenter     & optional, mandates how to align the image within the screen cells \\
    \textbf{screen-layer}     & ReplaceText      & render layer: BehindText (0), ReplaceText (1, default), AboveText (2) \\
    \textbf{image-format}     &                  & a value that determines the image format. \\
                              &                  & see section \ref{sec:supported-image-formtats} for available image formats \\
    \textbf{image-width}      & auto-max         & optional, number of pixels of the image's width to display \\
    \textbf{image-height}     & auto-max         & number of pixels of the image's height to display \\
    \textbf{image-data}       &                  & image data in the specified input format \\
    \textbf{update-cursor}    & false            & optional, tells the terminal to move the text cursor after having the image rendered (image left aligned) below the image \\
    \textbf{auto-scroll}      & false            & boolean, if set and the image would extend below the visible page area,
                                                    the screen scrolls up to accommodate.
                                                    Defaults to false (no scrolling; the image is clipped at the bottom margin). \\
    \textbf{request-status}   & false            & request a success/failure status response code from the terminal \\
    \hline
\end{tabular}

The dimension rules from Render Image apply: at least one of \textbf{screen-rows}
or \textbf{screen-cols} must be specified with a value greater than zero.
If one is specified and the other is omitted (or zero), the missing dimension
is determined by preserving the image's aspect ratio.
If both are omitted or zero, the terminal may compute both dimensions
from the source image pixel size and the current cell pixel size (rounding up).
If the terminal does not support this, the sequence is silently ignored.

Scrolling behavior is governed by the \textbf{auto-scroll} parameter.
When set to false (the default), the image is clipped at the bottom of the visible page area.
When set to true, the screen scrolls up as needed to display the full image.
This may cause the top of the image to move into the scrollback area
if the image is taller than the page.

When both \textbf{auto-scroll} and \textbf{update-cursor} are set,
the cursor is positioned after all scrolling has completed.
The cursor moves to the beginning of the line below the last rendered
image row in current screen coordinates.

% }}}
\subsection{Release Image} % {{{

Removes the mapping of the name to the image in a storage pool.
This will cause decrementing the use-count of the previously uploaded image.
Existing rendered images are not affected by this operation.

\subsubsection*{parameters}

\begin{tabular}{|l|l|l|}
    \hline
    \textbf{parameter name} & \textbf{default} & \textbf{description} \\
    \hline
    \textbf{name}           &                  & the unique identifier of the uploaded image. \\
    \hline
\end{tabular}

% }}}
\subsection{Storage Management} % {{{

Uploaded images are reference counted.
Uploading a named image will associate the uploaded image with the specified
name and initialize the image's reference count to 1.

When uploading an image would exceed the storage pool limits,
the host may choose to actively evict images.

The recommendation here is to use FIFO for eviction,
but this is not a mandatory algorithm.
If a terminal author comes with a much smarter eviction policy implementation,
that should not in FIFO-order until the new image fits.

If the image is larger than the allowed storage size, the upload will fail.

Images with a reference count greater than zero (i.e.\ with active screen cell references)
should not be evicted. If eviction of referenced images is unavoidable,
the terminal displays a placeholder or an empty screen cell for affected fragments;
neither representation is mandatory and is free to be chosen by the terminal author.

Displaying an image results in incrementing the reference counter
by the number of screen cells that are holding fragments of the image.

Clearing a screen cell holding an image fragment (e.g. by overwriting
or deleting its contents) will decrement the image reference count of
the prior referenced image.

When no screen cell is holding a reference to the image,
the corresponding reference counter should be either 1 (if uploaded in a separate step)
or 0 (if uploaded within the render instruction).

When the reference count gets down to zero, the image can be safely evicted.

Releasing the image by its name will remove the name-to-image association, and thus,
decrement its reference counter.

Uploading a new image with an already used name will first release the old name-to-image association
as mentioned earlier and then register the new image with its name.
Existing screen cells continue to display fragments of the previously uploaded image.
Only new render commands using the name will reference the newly uploaded image.

% }}}
\subsection{Interoperability with other VT sequences} % {{{

The rendered image can either replace any existing text on the rendered
screen cells, render below or render above of the screen's text.

To retain interop with all existing screen buffer mutating sequences,
any image fragments being associated to a screen cell are strongly bound
to that cell, e.g. when this cell is being moved to the right, the associated
image fragemnt will move to the right, too.

VT sequences that are designed to delete text, will also delete any
associated image fragment data in this screen cell as well.

\subsubsection*{Cursor and margin}

The cursor position respects
\code{DECOM}\footnote{https://www.vt100.net/docs/vt510-rm/DECOM.html}
margin and an image must not leave the current horizontal nor vertical margins
when being rendered.
The image gets clipped if it would render outside the margin.

\subsubsection*{Screen modifying commands}

All existing screen modifying VT sequences preserve their meanings.
That is, commands such as
\code{DECIC}\footnote{https://www.vt100.net/docs/vt510-rm/DECIC.html},
and others will split the image in two halfs.

\subsubsection*{Screen cell characters and image tiles}

Image fragments can be rendered in one of three layers:
BehindText (behind text, replacing the SGR background),
ReplaceText (replacing text glyphs),
or AboveText (above text as an overlay).

When text is written into a screen cell that currently holds
an image fragment in the ReplaceText layer,
the text replaces the image fragment.

\subsubsection*{Image alpha channel and graphics rendition background color}

Images with an alpha channel will blend into the graphics renditions
background color (respecting reverse-video attribute) of that cell analogous
to how padding pixel color is applied for image fragments.

\subsubsection*{Hyperlinks}

If the terminal emulator supports hyperlinks and the application has initiated
a hyperlink, followed by rendering an image, the whole rendered image becomes
hyperlinked, analogous to how hyperlinks are applied to text.

\subsubsection*{Text reflow}

Text reflow is supported only inconsistently across a few terminals.
Therefore this specification does not attempt to address this for images.
Image cells behave just like text cells, potentially wrapping on shrink
and eventually merging back when regrown.

When text reflow is going to be formally specified it must then also address images.

\subsubsection*{Clearing the screen}

VT sequences (such as \code{CSI Ps J}) that reset at least the textual contents
of the screen cells will also clear their image fragment referencing an underlying image,
possibly also releasing the underlying image(s) from the image storage pool
when their reference count has reached zero.

\subsubsection*{Primary and alternate screen}

Alternate screen and primary screen maintain both a shared image storage pool.

% }}}

% }}}
\section{Syntax} % {{{

This section maps each semantic action to actual VT sequences.

All commands use a single \code{DCS} (Device Control String) sequence
with intermediate byte \code{!} (0x21) and final byte \code{g} (0x67):

\begin{center}
    \code{DCS ! g <message> ST}
\end{center}

The operation type is specified via the \code{o} header inside the message:

\begin{tabular}{ |c|l|l| }
    \hline
    \textbf{Value of \code{o}} & \textbf{Operation} & \textbf{Example} \\
    \hline
    \code{u} & Upload          & \code{DCS ! g o=u,n=img,f=4;!<data> ST} \\
    \code{r} & Render          & \code{DCS ! g o=r,n=img,r=10,c=20 ST} \\
    \code{s} & Upload+Render   & \code{DCS ! g o=s,f=1,r=10,c=20;!<data> ST} \\
    \code{d} & Release         & \code{DCS ! g o=d,n=img ST} \\
    \code{q} & Query           & \code{DCS ! g o=q ST} \\
    \hline
\end{tabular}

The two-byte introducer (\code{!g}) avoids conflicts with all known
\code{DCS} sequences. DCS final bytes are a scarce namespace shared
across decades of standards; using a single intermediate-plus-final
combination ensures that the entire protocol consumes exactly one
\code{DCS} slot and leaves room for future operations without
allocating additional sequences.

When the \code{o} header is missing or contains an unrecognized value,
the terminal silently ignores the sequence.

\paragraph*{Error handling.}
When a command is described as ``silently ignored'', the terminal must consume
the entire DCS data stream up to and including the String Terminator (\code{ST})
before resuming normal parsing.
No response is sent to the application unless the \textbf{request-status} flag was set.

\subsection{Message Format} % {{{

This message format helps sending arbitrary parametrized messages
as payload and is suitable for \code{DCS} and \code{OSC} payloads.

The message format is inspired by and trivialized as basic HTTP message like
format with the following basic rules:

\begin{itemize}
    \item A message is devided into two optional parts, i.e. a set of headers and the body part.
    \item Headers and body are seperated by \code{;}.
    \item Header entries are seperated by \code{,}.
    \item Header name and value is seperated by \code{=}.
    \item Duplicate header names will override the previousely declared ones.
    \item Header value and message body can be optionally Base64 encoded by prepending \\
        an exclamation mark (\code{!}) at the beginning of the header's value or body.
        \textbf{Note:} Message bodies containing binary image data (RGB, RGBA, or PNG)
        \textbf{must} be Base64-encoded, because raw binary data may contain bytes
        that would be interpreted as C0 or C1 control characters
        (e.g. \code{0x1B}, the ESC byte), corrupting the DCS parser state.
    \item superfluous commas in header seperation are ignored
    \item header names without a value reflect a boolean truth value,
        its \code{=} can therefore be omitted.
\end{itemize}

When this message format is used as payload for \code{OSC} commands, then
the \code{OSC}'s payload will still need to contain the leading numeric
number followed by a semicolon (\code{;}) in order to retain consistency
with existing \code{OSC} implementations.

\subsubsection*{Resource Requirements}

These are the requirements a message parser must be able to process at least.
There should always be a limit, too, which is implicitly at least as big as
the here specified requirements.

Especially the message body might be useful to extend the limit for.

\begin{itemize}
    \item 64 bytes for header name
    \item 512 bytes for header value
    \item 32 header pairs
    \item 16 Megabyte for the body.
\end{itemize}

\subsubsection*{Examples}

\begin{itemize}
    \item \code{first=Foo,second=Bar;some body here}
    \item \code{,first=Foo,second,,,another=value,also=;some body here}
    \item \code{message=!SGVsbG8gV29ybGQ=} (no body, only one Base64 encoded header)
    \item \code{;!SGVsbG8gV29ybGQ=} (no headers, only one Base64 encoded body)
\end{itemize}

% }}}
\subsection{Feature Detection} % {{{

Syntax: \DA

\DA's response code for detecting support for an implementation of this specification is \code{90}.
Numbers below 50 are avoided to prevent conflicts with existing DA1 extension assignments.

% }}}
\subsection{Query Resource Limits} % {{{

Syntax: \code{DCS ! g o=q ST}

The terminal responds with all resource limits in a single reply
using the same \code{DCS ! g} framing:

\code{DCS ! g s=8,m=Pm,b=Pb,w=Pw,h=Ph ST}

Where:

\begin{tabular}{ |c|l| }
    \hline
    \textbf{Header} & \textbf{Description} \\
    \hline
    \code{s=8} & response type identifier (distinguishes from status codes 0--3) \\
    \code{m}   & maximum number of concurrently stored images \\
    \code{b}   & maximum bytes per image (uncompressed RGBA) \\
    \code{w}   & maximum image width in pixels \\
    \code{h}   & maximum image height in pixels \\
    \hline
\end{tabular}

% }}}
\subsection{Upload Image} % {{{

Syntax: \code{DCS ! g o=u,<headers>;[body] ST}

The payload's format is mandated by the \code{f}'s value. However, since it must not contain
any C0 or C1 escape codes, the transport is further protected by encoding it via Base64.

\subsubsection*{Parameters}

\begin{tabular}{|m{3cm}|m{2cm}|m{11cm}|}
    \hline
    \textbf{header name}   & \textbf{title}   & \textbf{notes} \\
    \hline
    \textbf{n}             & name             & unique name of the image \\
    \textbf{w}             & width            & pixel width of the uploaded image (not needed for Auto or PNG) \\
    \textbf{h}             & height           & pixel height of the uploaded image (not needed for Auto or PNG) \\
    \textbf{f}             & format           & 1 (Auto), 2 (RGB), 3 (RGBA), 4 (PNG). Required. \\
    \textbf{s}             & request-status   & boolean flag; if present, a status response is sent back \\
    \hline
\end{tabular}

\paragraph*{}
The \textbf{format} parameter (\code{f}) is required.
If omitted, the command is silently ignored.

An upload command with an empty body (zero bytes of image data) is invalid.
When \textbf{request-status} is set, the terminal responds with error code~2.
Otherwise the command is silently ignored.

\subsubsection*{Response Sequence}

When the \textbf{request-status} flag is set, the terminal responds with one of the following:

\begin{tabular}{ |r|l| }
    \hline
    \textbf{VT sequence} & \textbf{description} \\
    \hline
    \code{DCS ! g s=0 ST} & Image uploaded successfully \\
    \code{DCS ! g s=2 ST} & Upload failed: invalid image data (bad format, corrupt data, dimension mismatch) \\
    \code{DCS ! g s=3 ST} & Upload failed: resource limit exceeded (image too large or storage full after eviction) \\
    \hline
\end{tabular}

\paragraph*{}

The message body contains the image data.

\subsubsection*{Example}

This is a small Bash shell example that is uploading a PNG image with a fixed ID.

\begin{verbatim}
    # Shell function demonstrating how to upload a PNG image:
    function gip_upload() {
        local Name="$1"
        local ImageData=$(base64 "$2")
        echo -ne "\033P!go=u,n=${Name},f=1;!${ImageData}\033\\"
    }
\end{verbatim}

\pagebreak

% }}}
\subsection{Render Image} % {{{

Syntax: \code{DCS ! g o=r,<headers> ST}

\subsubsection*{Message parameters}

\begin{tabular}{|m{3cm}|m{2cm}|m{11cm}|}
    \hline
    \textbf{message header}   & \textbf{title}   & \textbf{Value} \\
    \hline
    \textbf{n} & name             & unique name referencing a previously uploaded image \\
    \textbf{r} & screen-rows      & number of rows the image will be printed on \\
    \textbf{c} & screen-cols      & number of columns this image will be printed on \\
    \textbf{a} & alignment        & alignment policy: 1 (TopStart) -- 9 (BottomEnd) \\
    \textbf{z} & resize           & resize policy: 0 (NoResize), 1 (ResizeToFit), 2 (ResizeToFill), 3 (StretchToFill) \\
    \textbf{x} & image-x-offset   & pixel x-offset into the source image (default: 0) \\
    \textbf{y} & image-y-offset   & pixel y-offset into the source image (default: 0) \\
    \textbf{w} & image-width      & pixel width of the source image sub-region to display (default: full width) \\
    \textbf{h} & image-height     & pixel height of the source image sub-region to display (default: full height) \\
    \textbf{u} & update-cursor    & boolean flag; if present, moves cursor below the rendered image area \\
    \textbf{s} & request-status   & boolean flag; if present, a response is sent back \\
    \textbf{l} & auto-scroll      & boolean flag; if present, page scrolls up if image doesn't fit \\
    \textbf{L} & screen-layer     & image layer: 0 (BehindText), 1 (ReplaceText, default), 2 (AboveText) \\
    \hline
\end{tabular}

\subsubsection*{Response Sequence}

When a response status code was requested, the terminal responds using the same
\code{DCS ! g} framing. This table consolidates all response codes used across
Upload, Render, and Query operations:

\begin{tabular}{ |c|r|l| }
    \hline
    \textbf{Code} & \textbf{VT sequence} & \textbf{description} \\
    \hline
    0 & \code{DCS ! g s=0 ST} & Operation completed successfully \\
    1 & \code{DCS ! g s=1 ST} & Render failed: no image with the given name found \\
    2 & \code{DCS ! g s=2 ST} & Upload failed: invalid image data \\
    3 & \code{DCS ! g s=3 ST} & Upload failed: resource limit exceeded \\
    8 & \code{DCS ! g s=8,m=...,b=...,w=...,h=... ST} & Resource limit query response (see Query Resource Limits) \\
    \hline
\end{tabular}

\subsubsection*{Example}

\begin{verbatim}
    # Shell function demonstrating how to render an image:
    function render_image() {
        local Name="$1"
        local ScreenColumns="20"
        local ScreenLines="10"
        echo -ne "\033P!go=r,n=${Name},c=${ScreenColumns},r=${ScreenLines}\033\\"
    }
\end{verbatim}

% }}}
\subsection{Upload and Render Image} % {{{

Syntax: \code{DCS ! g o=s,<headers>;[body] ST}

\subsubsection*{Message parameters}

\begin{tabular}{|m{3cm}|m{2cm}|m{11cm}|}
    \hline
    \textbf{message header}   & \textbf{title}   & \textbf{Value} \\
    \hline
    \textbf{f} & format           & 1 (Auto), 2 (RGB), 3 (RGBA), 4 (PNG) \\
    \textbf{w} & width            & pixel width of the image (not needed for Auto or PNG) \\
    \textbf{h} & height           & pixel height of the image (not needed for Auto or PNG) \\
    \textbf{r} & screen-rows      & number of rows the image will be printed on \\
    \textbf{c} & screen-cols      & number of columns this image will be printed on \\
    \textbf{a} & alignment        & alignment policy: 1 (TopStart) -- 9 (BottomEnd) \\
    \textbf{z} & resize           & resize policy: 0 (NoResize), 1 (ResizeToFit), 2 (ResizeToFill), 3 (StretchToFill) \\
    \textbf{u} & update-cursor    & boolean flag; if present, moves cursor below the rendered image area \\
    \textbf{s} & request-status   & boolean flag; if present, a response is sent back \\
    \textbf{l} & auto-scroll      & boolean flag; if present, page scrolls up if image doesn't fit \\
    \textbf{L} & screen-layer     & image layer: 0 (BehindText), 1 (ReplaceText, default), 2 (AboveText) \\
    \hline
\end{tabular}

\subsubsection*{Example}

This is a small Bash shell example that is uploading a PNG image with a fixed ID.

\begin{verbatim}
    # Shell function demonstrating how to render a oneshot image:
    function send_image_once() {
        local ImageData=$(base64 "$1")
        local GridCols="20"
        local GridRows="10"
        echo -ne "\033P!go=s,f=1,c=${GridCols},r=${GridRows};!${ImageData}\033\\"
    }
\end{verbatim}

% }}}
\subsection{Release Image} % {{{

Syntax: \code{DCS ! g o=d,<headers> ST}

\subsubsection*{Message parameters}

\begin{tabular}{ |c|c|l| }
    \hline
    \textbf{header name}   & \textbf{title}   & \textbf{notes} \\
    \hline
    \textbf{n}             & name             & unique name of the image \\
    \hline
\end{tabular}

% }}}
% }}}
\section{Supported Image Formats} % {{{

\label{sec:supported-image-formtats}

\begin{tabular}{c | c | l}
    File Format & Identifier     & Description \\
    \hline
    Auto        & \code{1}       & Automatic format detection from file headers / magic bytes and data size. \\
    RGB         & \code{2}       & raw RGB data with each color component being of size 1 byte. \\
    RGBA        & \code{3}       & raw RGBA data, just like RGB, but with an alpha channel. \\
    PNG         & \code{4}       & PNG file format. Width and height are extracted from the PNG header. \\
    \hline
\end{tabular}

\paragraph*{Auto Format Detection}

When the format is set to Auto (\code{1}), the terminal detects the actual image format
from the data itself.
PNG data is identified by its magic bytes (the first 8 bytes: \code{89 50 4E 47 0D 0A 1A 0A});
width and height are extracted from the PNG header.
For raw pixel data, width and height must be provided, and the data size determines
whether the format is RGB (\code{width * height * 3}) or RGBA (\code{width * height * 4}).
If the format cannot be determined, the data is invalid.
Implementations may extend Auto to support additional self-describing formats in the future.

% }}}
\section{Future Modifications} % {{{

Possible future modifications could (but do not have to) cover:

\begin{itemize}
    \item Set and reset resource limits.
    \item Query allocated resource names.
    \item Query resource utilization.
    \item Direct support of animations.
    \item Different transport encodings (other than Base64).
    \item Extending the use of uploaded images to also be used for application / window / tab icons.
    \item Extending the use of uploaded images to terminal background.
    \item A version negotiation mechanism (e.g.\ extended DA response or query response)
        may be added if protocol extensions require feature detection
        beyond presence/absence of the \DA code~\code{90}.
\end{itemize}

% }}}
\section{References} % {{{

\begin{enumerate}
    \item \label{ref:ctlseqs}\url{https://invisible-island.net/xterm/ctlseqs/ctlseqs.txt}
    \item \label{ref:twg-simple-image-display}Simple Image display, \url{https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/12}
    \item \label{ref:twg-gip}Good Image Protocol, \url{https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/26}
    \item \label{ref:image-sixel}Sixel Image Protocol, \url{https://vt100.net/docs/vt3xx-gp/chapter14.html}
    \item \label{ref:image-item2}iTerm2's image protocol, \url{https://iterm2.com/documentation-images.html}
    \item \label{ref:image-terminology}Terminology's image protocol, \url{https://github.com/borisfaure/terminology/blob/master/README.md#available-commands}
    \item \label{ref:image-kitty}Kitty's image protocol, \url{https://sw.kovidgoyal.net/kitty/graphics-protocol.html}
\end{enumerate}

% }}}
\section{Editorial Notes} % {{{

\begin{itemize}
    \item Maybe use tables instead of lists for parameter listings?
    \item How to implement a cat-like tool for images without breaking determinism for proxy
        terminals (like tmux).
        I.e. how does the proxy terminal know about how many rows will be used when
        a pixel perfect rendering is attempted (resize-policy set to NoResize,
        screen-cols=\$COLUMNS). But what is row count?
        I think the only way to achieve that if that image-cat tool peaks into
        the image dimensions and calculates a maybe perfect number of rows
        based on that and the retrieved pixel height (which is not recommended though).
    \item Do we want to deal with transparent pixels differently?
\end{itemize}

% }}}

\listoftodos

\end{document}

% vim:ts=4:sw=4