Thursday, May 17, 2012

Video Manipulation with HTML 5 Canvas

No doubt that the HTML 5 Canvas is currently one of the most popular HTML 5 elements. Its pixel manipulation and drawing APIs provide a broad range of possibilities to create interactive content natively integrated into the web browser. In this blog post I’m going to show you how to implement a simple video filter application using the Canvas Image API and a little bit of JavaScript. Don’t be afraid, it won’t hurt.

The reason why the Canvas element is so popular is its native integration into the web browser. No plugin needed to display animated graphics or interactive content. I guess everyone knows that there are platforms that don’t support Flash or Silverlight, but still provide web access. This is where Canvas comes into play. For sure, it’s not nearly matured like Flash and there are no advanced development environments but it seems very promising and is just at its beginning.

Canvas Image API.
Let’s do some theory first in order to understand what we’re doing later. The Image API provides the following three JavaScript functions:


-          drawImage(Image, dx, dy)
-          drawImage(Image, dx, dy, dw, dh)
-          drawImage(Image, sx, sy, sw, sh, dx, dy, dw, dh)
The first and second functions are very simple. They provide the ability to draw an image at the given position (dx, dy) with the given size (dw, dh). Nothing extraordinary so far. The third function enables you to cut off a piece of the image and place that piece right onto the canvas. The parameters starting with ‘s’ can be used to determine a region in the source image. With the ‘d’ parameters the region can be placed anywhere in the current canvas. For the video sample we will only use the first function. I’m going to demonstrate the use of the third function in a future blog post.

Another functionality provided by the Image API is the ImageData object. This object represents the current content of the Canvas and has only one property, the data property. The data property is a single dimensional array containing all the pixel values of the current image. There are four bytes in the array per pixel in the following order:


So this is exactly what we need to manipulate image or in this case video data. A single frame of a video is nothing else but an image.
There’s one additional hint I’d like to give here. Pay attention when manipulating the image data object while you’re in the development process. For security reasons it is not allowed to get the image data of the canvas when you run a local HTML file. You will receive a DOM Security Exception and your page will not be displayed. Sometimes you won’t see the exception (depends on the browser you are using) and just wonder why you don’t see anything. To avoid this problem start the web site from inside a local web server.

Manipulating the Video.
We’ll have a very simple setting for our web app. We’re just going to display a video and according to a selected filter display the result of the filter operation next to the original. Here is the HTML Body for this setting:

<body>
    <p>
        <select id="filterFunctionSelector">
            <option value="None">None</option>
            <option value="Invert">Color Inversion</option>
            <option value="Edge">Edge Detection</option>
            <option value="Blur">Blur</option>
        </select>
    </p>
    <video id="sourcevid" autoplay="true" loop="true">
        <source src="BigBuckBunny_640x360.mp4" type="video/mp4"/>
        <source src="BigBuckBunny_640x360.ogv" type="video/ogg"/>
    </video>
    <canvas width="640" height="360" id="outputCanvas"></canvas>
    <canvas width="640" height="360" id="sourceCanvas" style="visibility:
hidden"></canvas>
</body>

Notice that I used the new HTML 5 Video element. It’s great because you just need to provide the several video formats and the browser uses the first one it can display. No more nasty object or embed tags. For the demo I used an extract from the big buck bunny video at www.bigbuckbunny.org.
I defined two Canvas elements whereas the source Canvas will not be displayed on the web site. It just takes the current video frame and serves as source for manipulating the image data object. You could also draw directly to the output Canvas but by using an intermediate canvas that is not visible, you save some CPU power while there is no video filter selected.
Let’s have a look at the JavaScript code for displaying the manipulated video:
var destCanvasCtx = document.getElementById('outputCanvas').getContext('2d');
var sourceCanvasCtx = document.getElementById('sourceCanvas').getContext('2d');
var video = document.getElementById('sourcevid'); 

function drawFrame() {
    destCanvasCtx.clearRect(0, 0, destCanvasCtx.canvas.width,
  destCanvasCtx.canvas.height);    sourceCanvasCtx.drawImage(video, 0, 0); 

    var select = document.getElementById("filterFunctionSelector");
    var value = select.value; 

    if (value == 'None')
        return; 

    var input = sourceCanvasCtx.getImageData(0, 0, sourceCanvasCtx.canvas.width,
sourceCanvasCtx.canvas.height);
    var output = destCanvasCtx.createImageData(destCanvasCtx.canvas.width,
destCanvasCtx.canvas.height);
               

    if (value == 'Invert')
        invertColor(input, output);
    else if (value == 'Edge')
        edgeDetection(input, output);
    else if (value == 'Blur') {
        destCanvasCtx.drawImage(video, 0, 0);
        blur(destCanvasCtx, 10);
        return;
    }
    destCanvasCtx.putImageData(output, 0, 0);
}
setInterval(drawFrame, 40);

This is not as bad as it looks. Believe me! First we’re getting a reference to the canvas elements and the displayed video. We then define a function that will be called every 40 milliseconds. This is a little bit more than the 23 frames per second the video has. So we have enough time to apply the filter and draw the result to the output canvas.
Let’s take a closer look at the drawFrame function. First we clear the output canvas and draw the current video frame to the intermediate canvas. Notice the drawImage function. By using the getImageData function from the source canvas we get the pixel data of the current frame. The createImageData function creates an empty pixel array from the output canvas. According to the currently selected filter function, we pass both ImageData objects to the appropriate function. As an example I will show you the invertColor function here:
function invertColor(input, output) {
    var inputData = input.data;
    var outputData = output.data; 

    for (var i = 0; i < inputData.length; i += 4) {
        outputData[i] = 255 - inputData[i];
        outputData[i + 1] = 255 - inputData[i + 1];
        outputData[i + 2] = 255 - inputData[i + 2];
        outputData[i + 3] = 255;
    }
}


The function is very easy. It just iterates over the input pixel data and sets the manipulated values to the output array. When the function returns to the drawFrame function, the output array contains the complete image frame and can be drawn to the output canvas by using the putImageData function. Here is an image of the result in the browser with the applied edge detection filter:

Just one more hint: Applying image filters with software rendering has mostly not a good performance. For color inversion and edge detection it was no problem. But there is a blurring filter too. First I tried to implement a real blurring filter but it turned out to be very slow in the several browsers. After searching the web I found that others had the same problem. You can find a very elegant solution here: http://www.flother.com/blog/2010/image-blur-html5-canvas/. The approach is to just draw several copies of the current image whereas each copy is offset one pixel around the original one and is displayed with some opacity. So it’s not a real filter algorithm but it gives the impression of a blurred image and provides the needed speed for a video.

Summary
The Canvas element is very powerful and it is exciting to see all the possibilities lying within it. As I mentioned at the beginning of this post it has just started and there is still a lot of work to do when it comes to browser support and appropriate development environments. Just give it a try and find out for yourself if Canvas has something to offer for you.

No comments:

Post a Comment