Skip to content

Commit 8e13851

Browse files
tharkumtharkum
committed
Add new object and tokens to WEBGL_video_texture
 * Add new HTMLElementTexture object and new tokens for TEXTURE_VIDEO_WEBGL texture target.
1 parent ea5e7d0 commit 8e13851

1 file changed

Lines changed: 242 additions & 51 deletions

File tree

‎extensions/proposals/WEBGL_video_texture/extension.xml‎

Lines changed: 242 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,12 @@
66
working group</a> (public_webgl 'at' khronos.org) </contact>
77

88
<contributors>
9-
<contributor>Byungseon Shin (sun.shin 'at' lge.com)</contributor>
10-
<contributor>Andrey Volykhin (andrey.volykhin 'at' lge.com)</contributor>
9+
<contributor>Byungseon Shin, LG Electronics</contributor>
10+
11+
<contributor>Andrei Volykhin, LG Electronics</contributor>
12+
13+
<contributor>Mark Callow, Edgewise Consulting</contributor>
14+
1115
<contributor>Members of the WebGL working group</contributor>
1216
</contributors>
1317

@@ -18,19 +22,47 @@
1822
</depends>
1923

2024
<overview>
21-
<mirrors href="https://www.khronos.org/registry/gles/extensions/OES/OES_EGL_image_external.txt"
22-
name="OES_EGL_image_external">
23-
<addendum>Defines a new texture target <code>TEXTURE_VIDEO_IMAGE</code>.</addendum>
24-
<addendum>Provides a mechanism for binding <code>HTMLVideoElement</code> stream to video texture targets.</addendum>
25-
<addendum>Provides time of frame, texture width and height of <code>HTMLVideoElement</code>'s texture binding.</addendum>
26-
</mirrors>
25+
<p>This extension defines a new object, the <code>HTMLVideoTexture</code>,
26+
that can be used to efficiently transfer a sequence of image frames from
27+
a producer which is outside the control of WebGL into a WebGL texture.
28+
This is done via a new texture target, <code>TEXTURE_VIDEO_WEBGL</code>
29+
which can only be specified as being the consumer of an image stream from a
30+
producer element.</p>
2731

28-
<features>
29-
<feature>Add support for <code>WEBGL_video_texture</code>
30-
binding of HTMLVideoElement.</feature>
32+
<p>There is no support for most of the functions that manipulate other
33+
texture targets (e.g. you cannot use <code>*[Tt]ex*Image*()</code>
34+
functions with <code>TEXTURE_VIDEO_WEBGL</code>). Also,
35+
<code>TEXTURE_VIDEO_WEBGL</code> targets never have more than a single
36+
level of detail. Because of these restrictions, it is possible to
37+
allow sources which have internal formats not otherwise supported by WebGL,
38+
such as planar or interleaved YUV data to be WebGL texture target siblings.</p>
39+
40+
<p>The extension extends GLSL ES with a new <code>samplerVideoWebGL</code> type
41+
and matching sampling functions that provide a place for an implementation
42+
to inject code for sampling non-RGB data when necessary without degrading performance
43+
for other texture targets. Sampling a <code>TEXTURE_VIDEO_WEBGL</code> via a sampler of
44+
type <code>samplerVideoWebGL</code> always returns RGBA data.</p>
45+
46+
<p>Sampling a WebGL texture which is not associated with any
47+
<code>HTMLVideoTexture</code> object will return a sample value of (0,0,0,1).</p>
48+
49+
<p>Each <code>TEXTURE_VIDEO_WEBGL</code> texture object may require up
50+
to 3 texture image units for each texture unit to which it is bound.
51+
The number of texture image units required by a bound texture object
52+
can be queried using <code>getTexParameter</code> with target set
53+
to the texture target in question, value set to
54+
<code>REQUIRED_TEXTURE_VIDEO_IMAGE_UNITS_WEBGL</code>, and <code>activeTexture</code>
55+
set to the texture unit to which the texture object is bound.</p>
56+
57+
<p><code>HTMLVideoTexture</code> provides a commands for <em>latching</em> an
58+
image frame into the consuming texture as its contents and retrieving additional
59+
information as a timestamp or a transformation matrix.</p>
3160

61+
<features>
3262
<glsl extname="WEBGL_video_texture">
63+
<stage type="vertex"/>
3364
<stage type="fragment"/>
65+
3466
<type name="samplerVideoWEBGL"/>
3567

3668
<function name="texture2D" type="vec4">
@@ -39,87 +71,243 @@
3971
<param name="coord" type="vec2"/>
4072
</function>
4173

74+
<function name="texture2DProj" type="vec4">
75+
<param name="sampler" type="samplerVideoWEBGL"/>
76+
77+
<param name="coord" type="vec3"/>
78+
</function>
79+
80+
<function name="texture2DProj" type="vec4">
81+
<param name="sampler" type="samplerVideoWEBGL"/>
82+
83+
<param name="coord" type="vec4"/>
84+
</function>
4285
</glsl>
4386
</features>
4487
</overview>
4588

4689
<idl xml:space="preserve">
4790
[NoInterfaceObject]
48-
interface WebGLVideoFrameInfo {
49-
readonly attribute double currentTime;
50-
readonly attribute unsigned long textureWidth;
51-
readonly attribute unsigned long textureHeight;
91+
interface HTMLVideoTexture : EventTarget {
92+
void updateTexImage();
93+
94+
double getTimestamp();
95+
void getTransformMatrix(Float32Array matrix);
96+
97+
double getLastAvailableTimestamp();
98+
99+
attribute EventHandler onframeavailable;
100+
attribute EventHandler onerror;
52101
};
53102

54103
[NoInterfaceObject]
55104
interface WEBGL_video_texture {
56-
const GLenum TEXTURE_VIDEO_IMAGE = 0x851D;
57-
const GLenum SAMPLER_VIDEO_IMAGE = 0x8B61;
105+
const GLenum TEXTURE_VIDEO_WEBGL = 0x9248;
106+
const GLenum SAMPLER_VIDEO_WEBGL = 0x9249;
107+
const GLenum TEXTURE_BINDING_VIDEO_WEBGL = 0x924A;
108+
const GLenum REQUIRED_TEXTURE_VIDEO_IMAGE_UNITS_WEBGL = 0x924B;
58109

59-
[RaisesException] WebGLVideoFrameInfo VideoElementTargetVideoTexture(
60-
GLenum target, HTMLVideoElement video);
110+
[RaisesException] HTMLVideoTexture? createVideoTexture(WebGLTexture? texture, HTMLVideoElement video);
61111
};
62112
</idl>
63113

64-
<samplecode xml:space="preserve">
114+
<!-- new functions -->
115+
116+
<newfun>
117+
<p>On <code>HTMLVideoTexture</code>:</p>
118+
119+
<function name="updateTexImage" type="void">
120+
Release the previously-held frame, if any, and update the texture image
121+
to the most recent frame from the image stream.
122+
This may cause some frames of the stream to be skipped.
123+
Sampling the WebGL texture, that is the <code>HTMLVideoTexture</code>'s
124+
<em>consumer</em>, will return values from the latched image.
125+
The image data is guaranteed not to change as long as the image is latched.
126+
It will implicitly binds associated WebGL texture to the active texture unit's
127+
<code>TEXTURE_VIDEO_WEBGL</code> texture target.
128+
</function>
129+
130+
<function name="getTimestamp" type="double">
131+
Retrieve the timestamp associated with the texture image set by the most
132+
recent call to <code>updateTexImage</code>. This timestamp represent
133+
the time of frame relative to start of the producer timeline, the time
134+
when the frame was produced (the Media Stream Counter in OpenML),
135+
not the time when frame was received by application.
136+
This timestamp is in seconds, and is normally monotonically increasing.
137+
It is equivalent of <code>HTMLMediaElement</code>'s <em>currentTime</em> property.
138+
</function>
139+
140+
<function name="getTransformMatrix" type="void">
141+
<param name="matrix" type="Float32Array"/>
142+
Retrieve the 4x4 texture coordinate transform matrix associated with
143+
the texture image set by the most recent call to <code>updateTexImage</code>.
144+
This transform matrix maps 2D homogeneous texture coordinates
145+
of the form (s, t, 0, 1) with s and t in the inclusive range [0, 1]
146+
to the texture coordinate that should be used to sample that location
147+
from the texture. Sampling the texture outside of the range
148+
of this transform is undefined.
149+
The matrix is stored in column-major order.
150+
</function>
151+
152+
<function name="getLastAvailableTimestamp" type="double">
153+
Retrieve the timestamp associated with the most recent texture image
154+
which could be provided by a producer.
155+
</function>
156+
</newfun>
157+
158+
<dl class="methods">
159+
<dt><code class="attribute-name">onframeavailable</code> of type
160+
<code>EventHandler</code></dt>
161+
162+
<dd>The <code>onframeavailable</code> handler is executed when a new frame
163+
could be latched from image stream.</dd>
164+
165+
<dt><code class="attribute-name">onerror</code> of type
166+
<code>EventHandler</code></dt>
167+
168+
<dd>The <code>onerror</code> handler is executed when this object
169+
will be in "incomplete" state in case if a image stream's consumer or
170+
producer will be deleted or due internal errors.</dd>
171+
</dl>
65172

66-
<p> This a fragment shader that samples a video texture.</p>
173+
<!-- new tokens -->
174+
175+
<newtok>
176+
<p>On <code>WEBGL_video_texture</code>:</p>
177+
178+
<p>The meaning and use of these tokens is similar as described in <a
179+
href="https://www.khronos.org/registry/OpenGL/extensions/OES/OES_EGL_image_external.txt">OES_EGL_image_external</a>.</p>
180+
181+
<function name="bindTexture" type="void">
182+
<param name="target" type="GLenum"/>
183+
<param name="texture" type="WebGLTexture?"/>
184+
<code>TEXTURE_VIDEO_WEBGL</code> is accepted as a target by the
185+
<code>target</code> parameter of <code>bindTexture</code>
186+
</function>
187+
188+
<function name="getActiveUniform" type="WebGLActiveInfo?">
189+
<param name="program" type="WebGLProgram?"/>
190+
<param name="index" type="GLuint"/>
191+
<code>SAMPLER_VIDEO_WEBGL</code> can be returned in the
192+
<code>type</code> field of the <code>WebGLActiveInfo</code> returned by
193+
<code>getActiveUniform</code>
194+
</function>
195+
196+
<function name="getParameter" type="any">
197+
<param name="pname" type="GLenum"/>
198+
<code>TEXTURE_BINDING_VIDEO_WEBGL</code> is accepted by
199+
the <code>pname</code> parameter of <code>getParameter</code>
200+
</function>
201+
202+
<function name="getTexParameter*" type="any">
203+
<param name="target" type="GLenum"/>
204+
<param name="pname" type="GLenum"/>
205+
<code>REQUIRED_TEXTURE_VIDEO_IMAGE_UNITS_WEBGL</code> is accepted
206+
as the <code>pname</code> parameter of <code>getTexParameter*</code>
207+
</function>
208+
</newtok>
209+
210+
<newfun>
211+
<p>New <code>HTMLVideoTexture</code> object creation</p>
212+
213+
<function name="createVideoTexture" type="HTMLVideoTexture?">
214+
<param name="texture" type="WebGLTexture?"/>
215+
<param name="video" type="HTMLVideoElement"/>
216+
Create a new <code>HTMLVideoTexture></code> object that can be used to
217+
transfer a sequence of image frames from a <code>HTMLVideoElement</code>
218+
into a WebGL texture.
219+
WebGL texture must be defined as a texture with <code>TEXTURE_VIDEO_WEBGL</code>
220+
target before passing it as <code>texture</code> parameter
221+
(Note: to create a <code>TEXTURE_VIDEO_WEBGL</code> texture and bind it,
222+
call <code>bindTexture</code> with <code>target</code> set to
223+
<code>TEXTURE_VIDEO_WEBGL</code> and <code>texture</code>
224+
set to the name of the new created texture).
225+
If this function is called with <code>HTMLVideoElement</code>
226+
which origin differs from the origin of the containing Document,
227+
a <code>SECURITY_ERR</code> exception must be thrown.
228+
If the WebGL texture is later deleted, connected to a different
229+
<code>HTMLVideoTexture</code>, then this <code>HTMLVideoTexture</code>
230+
will be in "incomplete" state and release any associated resources.
231+
If the <code>HTMLVideoElement</code> or <code>HTMLVideoTexture</code>
232+
is later destroyed then the consuming texture will be "incomplete" and
233+
sampling from it will return a sample value of (0,0,0,1).
234+
</function>
235+
</newfun>
236+
237+
<errors>
238+
<error>
239+
The error <code>INVALID_OPERATION</code> is generated by calling
240+
<code>createVideoTexture</code> with a <code>texture</code> parameter
241+
that does not identify a <code>VIDEO_WEBGL</code> texture.
242+
</error>
243+
</errors>
244+
245+
<samplecode xml:space="preserve">
246+
<p>This a fragment shader that samples a video texture.</p>
67247
<pre>
248+
#version 100 es
68249
#extension GL_WEBGL_video_texture : require
250+
69251
precision mediump float;
70-
varying vec2 v_texCoord;
71252

72253
uniform samplerVideoWEBGL uSampler;
254+
varying vec2 v_texCoord;
73255

74256
void main(void) {
75257
gl_FragColor = texture2D(uSampler, v_texCoord);
76258
}
77259
</pre>
78260

79-
<p> This shows application that renders video using proposed extension. </p>
261+
<p>This shows application that renders an HTMLVideoElement using proposed extension.</p>
80262
<pre>
81-
var videoElement = document.getElementById("video");
82-
var videoTexture = gl.createTexture();
263+
var gl = document.createElement('canvas').getContext('webgl');
264+
var video = document.getElementById("video");
83265

84-
function update() {
85-
var ext = gl.getExtension('WEBGL_video_texture');
86-
if(ext !=== null){
87-
gl.bindTexture(ext.TEXTURE_VIDEO_IMAGE, videoTexture);
88-
ext.VideoElementTargetVideoTexture(ext.TEXTURE_VIDEO_IMAGE, videoElement);
89-
gl.bindTexture(ext.TEXTURE_VIDEO_IMAGE, null);
90-
}
91-
}
92-
93-
function render() {
94-
gl.clearColor(0.0, 0.0, 1.0, 1.0);
95-
gl.clear(gl.COLOR_BUFFER_BIT);
266+
var ext = gl.getExtension('WEBGL_video_texture');
96267

97-
gl.bindBuffer(gl.ARRAY_BUFFER, squareVerticesBuffer);
98-
gl.vertexAttribPointer(vertexPositionAttribute, 3, gl.FLOAT, false, 0, 0);
268+
var texture = gl.createTexture();
269+
gl.bindTexture(ext.TEXTURE_VIDEO_WEBGL, texture); // explicit texture binding
99270

100-
gl.activeTexture(gl.TEXTURE0);
101-
gl.bindTexture(ext.TEXTURE_VIDEO_IMAGE, videoTexture);
102-
gl.uniform1i(gl.getUniformLocation(shaderProgram, "uSampler"), 0);
271+
var videoTexture = ext.createVideoTexture(texture, video);
272+
videoTexture.onframeavailable = function() { ... };
273+
videoTexture.onerror = function() { ... };
103274

104-
gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);
275+
function update() {
276+
videoTexture.updateTexImage(); // implicit texture binding
105277
}
106-
</pre>
107278

108-
<p> Application renders each video frames into WebGL canvas based on game-loop pattern. </p>
109-
<pre>
279+
function render() {
280+
// ... do something
281+
gl.drawArrays(...);
282+
}
110283

111284
while (true) {
112-
update();
113-
processInput();
114-
render();
285+
update();
286+
processInput();
287+
render();
115288
}
116289
</pre>
117-
118290
</samplecode>
119291

120292
<tests/>
121293

122-
<issues/>
294+
<issues>
295+
<ol>
296+
<li>
297+
<p>In which colorspace should be the RGB value returned by sampler?</p>
298+
299+
<p>A. In linear RGB colorspace.</p>
300+
301+
<p>B. In extended sRGB (scRGB-non-linear) colorspace <a
302+
href="https://developer.apple.com/documentation/coregraphics/kcgcolorspaceextendedsrgb?language=objc">kCGColorSpaceExtendedSRGB</a></p>
303+
304+
<p>C. It will be allowed to choose by passed parameter at a
305+
<code>HTMLVideoTexture</code> object creation time.</p>
306+
307+
<p>UNRESOLVED.</p>
308+
</li>
309+
</ol>
310+
</issues>
123311

124312
<history>
125313
<revision date="2016/11/05">
@@ -135,5 +323,8 @@ interface WEBGL_video_texture {
135323
<change>Define new sampler and texture type, TEXTURE_VIDEO_IMAGE and SAMPLER_VIDEO_IMAGE.</change>
136324
<change>Change EGLImageTargetTexture2DOES to VideoElementTargetVideoTexture.</change>
137325
</revision>
326+
<revision date="2017/09/21">
327+
<change>Add new HTMLVideoTexture object and new tokens.</change>
328+
</revision>
138329
</history>
139330
</proposal>

0 commit comments

Comments
 (0)