Note – the current versions of the libraries referenced may not work properly with the examples in this post. We’ll update the post soon to make sure it works properly.
So not that long ago, if you wanted an app to take a photo, it had to be a native app — such as a Windows/Mac app or a native mobile application. But HTML5 has brought a number of new APIs that allow not only taking photos, but analyzing and manipulating them all within a browser.
Sadly, there is still inconsistent support for these features across browsers (both desktop and mobile). So in practice, the best approach may be to target the APIs supported by iOS 6, which is more or less the minimal feature set across browsers that support any of this at all.
The three pieces we’ll need are:
- The Canvas
- File I/O
- Changes to the File Upload input and Media Capture
Note that Media Capture has been superceded by getUserMedia (more info) but we don’t use that here because iOS doesn’t support getUserMedia yet
The Canvas is a space on the page to draw into, using either 2-D or 3-D APIs. Conveniently, it also lets you draw images into it, inspect and alter the pixel data, and export its content as an image. Canvas is supported by IE 9 and modern versions of Firefox, WebKit/Safari/iOS 6, Opera, and Chrome. However, not all browsers support 3-D (via WebGL), and browsers don’t generally support ALL of the Canvas spec (which has continued to evolve even after initial browser support). Still, the parts we need are supported across browsers.
The additional “accept” and “capture” attributes on the file upload input element let us specify that the user should be able to select an image file, getting it from an attached camera if possible. Crucially, on iOS you are given the option to select a photo from the photo library or take a photo on the spot (though some desktop browsers only let you browse for an image file, even if a Webcam is attached). Then the “files” property on the input element lets us access the selected files (unfortunately, not supported in IE < 10).
So the formula for taking and manipulating a photo is:
- Configure a file input using the new attributes
- When an image file is selected (or photo taken), access the image file
- Check the image EXIF data for rotation using File I/O
- Decide what size to scale the image to (if needed)
- Configure a Canvas for the selected image dimensions
- Draw the image to the Canvas, rotating and scaling as appropriate
- Access and manipulate the pixel data for the Canvas
- Write the pixel data back to the Canvas
- Export the image on the Canvas as an image file
Take these one step at a time (and note that I use jQuery for convenience, but it is not needed for this to work):
1. Configure a file input using the new attributes.
There are two ways to configure the input (and it makes no difference to desktop browsers). An input configured like this will prompt the user to take a photo or select a saved impage on either iOS or Android:
<input id="PhotoPicker" type="file" accept="image/*" />
With the additional capture=”camera”attribute, Android goes directly to the camera without giving the option to use a saved photo (though the iOS behavior is the same):
Note that if you don’t like the default appearance, you can conceal the input widget and connect a button or link (styled any way you like) to activate it. The input doesn’t work if set to display: 'none', but we can put it inside a div of size 0 to effectively hide it:
$('#PhotoButton').click(function() {
$('#PhotoPicker').trigger('click');
return false;
});
2. When an image file is selected (or photo taken), access the image file.
We just use the new files property on the input:
$('#PhotoPicker').on('change', function(e) {
e.preventDefault();
if(this.files.length === 0) return;
var imageFile = this.files[0];
...
});
3. Check the image EXIF data for rotation using File I/O:
All images may have a rotation recorded by the camera, such that no matter how the camera was oriented when the photo was taken, the image can be displayed appropriately on screen. In particular, an iPad considers the landscape orientation to be normal (though it can still be upside-down), and the portrait orientation takes photos with a 90 degree rotation. The EXIF header in an image records this rotation, so we can read it in order to display the image appropriately.
This part uses two open source (MPL) EXIF-reading scripts:
<script src="http://www.nihilogic.dk/labs/exif/exif.js"
type="text/javascript"></script>
<script src="http://www.nihilogic.dk/labs/binaryajax/binaryajax.js"
type="text/javascript"></script>
Then we use the new File API to read the image data (and there’s a nice image heredescribing the EXIF Orientation codes):
var width;
var height;
var binaryReader = new FileReader();
binaryReader.onloadend=function(d) {
var exif, transform = "none";
exif=EXIF.readFromBinaryFile(createBinaryFile(d.target.result));
if (exif.Orientation === 8) {
width = img.height;
height = img.width;
transform = "left";
} else if (exif.Orientation === 6) {
width = img.height;
height = img.width;
transform = "right";
}
...
};
binaryReader.readAsArrayBuffer(imageFile);
4. Decide what size to scale the image to (if needed).
If you want to limit the size of the image, you can scale the height and width accordingly. With mobile devices with high-resolution cameras, this may be necessary to limit memory usage:
var MAX_WIDTH = 1024;
var MAX_HEIGHT = 768;
if (width/MAX_WIDTH > height/MAX_HEIGHT) {
if (width > MAX_WIDTH) {
height *= MAX_WIDTH / width;
width = MAX_WIDTH;
}
} else {
if (height > MAX_HEIGHT) {
width *= MAX_HEIGHT / height;
height = MAX_HEIGHT;
}
}
5. Configure a Canvas for the selected image dimensions.
If the canvas is on the page, you can just grab it:
var canvas = $('#PhotoEdit')[0];
Otherwise, you can use an offscreen canvas:
var canvas = document.createElement('canvas');
…and then size it accordingly:
canvas.width = width; canvas.height = height;
6. Draw the image to the Canvas, rotating and scaling as appropriate.
In order to draw the image to a Canvas, first we have to load it in an , and a convenient way is to create a URL to assign as the img.src from the selected file (using part of the File API):
var img = new Image();
var url = window.URL ? window.URL : window.webkitURL;
img.src = url.createObjectURL(imageFile);
img.onload = function(e) {
url.revokeObjectURL(this.src);
...
};
Then inside the onload handler, we can draw the image to the canvas. (In practice, I put most of the logic we’re discussing inside the onload handler.) A transformation matrix on the context handles rotating the image (if needed) and shifting it back into the viewable area.
var ctx = canvas.getContext("2d");
ctx.fillStyle = 'white';
ctx.fillRect(0, 0, 700, 600);
if(transform === 'left') {
ctx.setTransform(0, -1, 1, 0, 0, height);
ctx.drawImage(img, 0, 0, height, width);
} else if(transform === 'right') {
ctx.setTransform(0, 1, -1, 0, width, 0);
ctx.drawImage(img, 0, 0, height, width);
} else if(transform === 'flip') {
ctx.setTransform(1, 0, 0, -1, 0, height);
ctx.drawImage(img, 0, 0, width, height);
} else {
ctx.setTransform(1, 0, 0, 1, 0, 0);
ctx.drawImage(img, 0, 0, width, height);
}
ctx.setTransform(1, 0, 0, 1, 0, 0);
7. Access and manipulate the pixel data for the Canvas
The getImageData method returns an array of pixel data, with one byte each for red, green, blue, and alpha. This example applies a “green screen” effect, setting pixels to transparent (alpha = 0) if they are mainly green.
var pixels = ctx.getImageData(0, 0, canvas.width, canvas.height);
var r, g, b, i;
for (var py = 0; py < pixels.height; py += 1) {
for (var px = 0; px < pixels.width; px += 1) {
i = (py*pixels.width + px)*4;
r = pixels.data[i];
g = pixels.data[i+1];
b = pixels.data[i+2];
if(g > 100 &&
g > r*1.35 &&
g > b*1.6) pixels.data[i+3] = 0;
}
}
There are of course other possibilities for filtering the image.
8. Write the pixel data back to the canvas.
Then we use putImageData to write the array of pixel data back to the canvas.
ctx.putImageData(pixels, 0, 0);
9. Export the image on the Canvas as an image file
Finally, the whole content of the canvas can be exported as an image. There may be a way to do this through the browser (in Firefox, for instance, right-clicking on any Canvas lets you view it as an image), but there is a canvas API call as well. Every browser supports saving the image as a PNG, and some may support other formats:
var data = canvas.toDataURL('image/png');
The resulting data URL looks like this:
data:image/png;base64,...
…where the “…” is the base 64 encoded version of the binary image file. So you can, for instance, upload the data URL to a server as a form field, and the server can strip off the data:image/png;base64, prefix and base 64 decode the rest and save it to a file like mycanvas.png.
Summary
So there you have it — a Web page that takes a photograph, scales and orients it as appropriate, processes the image as desired, and then displays it to the user and/or uploads it to a server.
Unfortunately, there is not yet a way to present a “Save As” dialog to the user to save the file directly to their local disk — the FileSystem and FileWriter APIs are not widely supported across browsers. But you can of course upload the image to the server and then redirect the user to a server URL to save it…
Consolidated Sample Code
Here’s a single page with a working example using the code we’ve gone through here. You can also use this sample image to see the green screen replacement effect.
Platform Support
Bold entries work fully:
Desktop Browsers
- Firefox 18
- Safari 6
- Chrome 25
- IE 10
- IE 9 (does not support getting selected files from file input)
Mobile Browsers:
- iOS 6
- iOS 5 (file input doesn’t work)
- BlackBerry 10
- Android 4.1 (built-in browser)
- Android 4.1 with Chrome from Play store
- Android 2 (native browser; does not take photo)
- Windows Phone 8 (file input doesn’t work)
- Surface RT (doesn’t work)