Alan Roberts has been testing cameras for many years, first at the BBC, and now as an independent consultant. His independence and accuracy is unquestioned. Camera manufacturers who turn to him for tests do not have the chance to alter or selectively edit Alan’s findings if the report is published. Unlike other testers he does not do direct comparisons with other cameras. Each test stands on its own and comes with a set of recommendations for how users can set up the camera for good results. He tries “to think like a normal user would think: What am I going to use this camera for? And how am I going to use it?”
For these reasons Alan’s reports have become required reading across the industry and over the years he has become the go-to authority for advice on the suitability of a camera’s imagery for different broadcast tasks.
Alan is largely responsible for defining the HD camera test regime for the European Broadcast Union (EBU) and the standards established are what he now uses to classify cameras.
The tests focus on how a camera would perform in a broadcast environment where the image needs to survive editing, grading (sometimes there isn’t any), encoding and transmission, before reaching a viewer’s TV. Measured resolution and dynamic range are important here. Alan singles out aliasing as being a particular issue because it wreaks havoc with the “smarty pants parts of the final compressors” before transmission for broadcast.
“If you can’t calculate where the motion is, you can’t calculate a really good high ratio compression. A camera that produces high aliasing is not suitable for grade one broadcasting,” he says.
Simply put, the tests determine if a camera is suitable for broadcasting in 4K and HD.
The EBU groups cameras into specific tiers for both UHD and HD. In HD, Tier 1 is the strictest. A camera that qualifies is one that can be used without restriction on any high definition programme.
For UHD Tier 1, a camera must have a minimum sensor pixel count of 3840 x 2160 in each of R & G & B. Every camera tested except the Sony F65 has thus far failed to reach this – mainly due to the limitations of using a Bayer-patterned sensor. The next rung down is UHD Tier 2 and to qualify cameras must be at least 10-bit and have a sensor pixel count greater than 2880 x 1620. The camera must also have timecode genlock.
Alan recently put the Canon C300 MkII through the test. His results are exhaustive and can be downloaded from the EBU here.
I spoke to him at length about his conclusions and he was full of praise for the C300 MkII’s imagery.
His key finding is that the camera has about 15 stops of dynamic range in Canon Log2. His method is based on standardised testing of the signal-to-noise ratio within patches of a colour chart under tungsten light. Alan explained to me that you can’t measure noise by simply covering the lens and shooting black (something that people sometimes do in online tests). This is because much of the noise in modern camera sensors is something called shot noise – random fluctuations in the numbers of photons hitting the sensor at a given illumination level. His method of testing is based on varying the exposure of the tungsten lit colour chart by adjusting the aperture, internal ND filters and shutter speed if needed.
In regular Canon Log he measured the camera at 13.5 stops of dynamic range, and in BT.709 the range was 11.5 stops.
Log 2 using the 12-bit recording format
(RGB 444). Adding 6dB and 12dB gain confirmed that the dynamic range is indeed about 15 stops.
I took the time to compare Alan’s C300 MkII findings to his earlier published results for the ARRI Amira in Log-C. The C300 MkII Canon Log2 noise levels appear to be very similar in the mid-range using his testing methods. Indeed, 50% of peak white is the point which the EBU use for certification. The C300 MkII passes here with no problems. At closer to peak white the Canon is actually quite a bit less noisy than the Amira (although film is actually noisy in the whites which is where the negative is at its densest). At the lower end of the curve, at below 10% peak white, things get noisier with the C300 MkII. The ARRI does something clever here which makes these lower levels less noisy.
Alan also tested the resolution of the camera and found that in 4K 3840×2160 10-bit 4:2:2 (in I-frame 410Mb/s) recording the camera resolved around 3300 lines horizontally and 1750 vertically. Like other Bayer pattern sensors there is always a loss of resolution due to the way it works. There was aliasing present, but he notes that it is not particularly high. These numbers are pretty good from a 4K single chip CMOS sensor.
In 1920×1080 Alan found the the C300 MkII resolution “reaches the limits for HDTV, and the level of aliases within the limits quite low.”
As to be expected, the Canon Log2 images had a slightly lower perceived resolution than the standard 709 gamma (although in post you should be able to bring Log images back near to the 709 version). Perhaps more interesting is that aliasing in the image was also reduced when switching from BT.709 to Canon Log2. These factors essentially make the Log images look more filmic than the BT.709 versions. Alan explains opting for Canon Log2 like this: “You don’t really lose anything; what you get is a different feel to the picture.”
The Canon Cinema EOS line is widely regarded as having pleasing images, even out of the box. Historically EBU reports have had detailed settings for gamma, knee, colour and other settings. With the C300 MkII Alan felt it was unnecessary to spend much time detailing these as the camera’s presets already “do a fantastic job”. He adds: “The presets are all doing precisely what they should do. The average user is never going to need to go into those (complex gamma, knee and slope) menus.” This should provide reassurance for operators wondering if they need to tinker more with the camera’s base settings.
He also maintains that to get the best colour performance with Canon Log2, the C300 MkII’s HD/2K 12-bit 4:4:4 mode should be used. This presents an interesting choice for shooters who need a HD/2K final output. Do you opt to shoot in lower bit depth 4K with the benefit of being able to pan and crop the image later in post? Or do you shoot in HD and opt for the better colour? The answer will of course depend on your circumstances.
Alan also tested the camera’s built-in noise reduction capabilities. He explains that the C300 MkII uses a spatial filter to do the noise reduction. He found that turning it on results in a loss of resolution, which should be expected. He says that the noise reduction does a good job and that if you want you can get 5db of noise improvement out of the camera, but at that point you’ve “lost quite a lot of resolution”. Between maximum noise reduction and off, Alan says, “clearly there is a compromise that is going to work rather nicely”, and setting the camera for mild noise reduction “between 3 and 5 (out of 12) would be a good result to have”. This maintains most of the detail while reducing the noise effectively. It also reduces the level of aliasing nicely.
Asked whether it is better to do mild noise reduction in post, instead of in camera, Alan says that in-camera is better if possible: “The less noise there is in the image, the easier job the encoder has putting pictures onto a card”. Stronger levels of noise reduction would be better done at a post facility, he told me, but this is “not the sort of thing you do on your own laptop”.
Based on his testing, Alan had no problems in awarding the C300 MkII Tier 1 for HD production, and Tier 2 for 4K. This rating should be highly reassuring for owners of the camera and programme makers looking at using it for upcoming projects.
Alan was keen to point out that his recommendations for settings are only a starting point from which operators and DPs can then experiment. He tells me:“My guess is that most users are going to be owners, and that they are going to experiment anyway.
“All I am going to do is test whether the camera is suitable for the main set of jobs that it is expected to do, and give some advice on how you should do it. Then the user is on his own….This is only a starting point. Off you go, explore it.”
