The headline specifications of Sony’s tiny a6300 make great reading: S35 sensor, internal 4K recording, internal HD recording at 50 Mbps, S-Log 2 and 3 gammas, plus the option of adding a XLR audio adapter to use professional audio equipment.
Specs are one thing – but just how good an image is it possible to get out of a $1000 camera? Are the pictures good enough for the kind of broadcast work that many of our readers do? At Newsshooter we were curious enough to send the diminutive camera to test guru Alan Roberts to find out.
Regular readers will be aware of Alan’s camera tests – they inform the EBU’s influential system of camera ‘tiers’ and are widely viewed in the television industry as the gold standard when assessing the quality of a camera’s output. He’s also sometimes commissioned by manufacturers themselves to put their latest releases through their paces as an independent assessor of image quality. (Companies that commission these tests can withhold the whole report if they choose, but cannot make changes to, or selectively publish, the results).
Our test differs from previous Alan’s previous EBU tests. We wanted to focus almost exclusively on the image quality. For this, it uses the same exacting measurements as the full test. We are only testing if the image of a camera lives up to the EBU requirements. It does not examine the other parts of the camera specification, its operation, or the menu. To us this makes sense because operators often live with or work around a camera’s physical shortcomings, but you can’t easily fix image issues.
The test focusses heavily on aspects of a camera’s image that will affect the final image after TV transmission. Aliasing, noise, dynamic range, rolling shutter, sensitivity and colour are all carefully examined. Although Alan’s test setup is relatively simple he uses his own specially created software to analyse the images. For the test a Zeiss ZE 100mm f2 macro lens was used. This lens is capable of producing extremely sharp images and should easily out-resolve any 4K sensor.
These results are interesting to all users, but will be critical to those of us who work for broadcasters. If you work primarily for web, then it’s worth bearing in mind that your footage won’t necessarily need to meet the same criteria, as video platforms like Vimeo and Youtube compress differently to TV. Basically, if your web video looks good enough to you and your clients, then it is good enough. It is entirely possible that some aliasing in your image might actually be perceived as making a video look ‘better’ online. Still, even if you do work mainly for online, the report will provide a useful way to assess the qualities of the camera.
The EBU groups cameras into specific tiers for both UHD and HD.
In HD there are four Tiers with several subdivisions. A Tier 1 camera can be used for any HD production and the Canon C300 mkII and XC10 recently made this grade. Below that are Tier 2L, which caters for Long-form TV, and also Tier 2J, which relaxes some of the criteria for news programming. Tier 2L cameras include the likes of Sony’s PMW-200 and PMW-150.
For UHD Tier 1, a camera must have a minimum sensor pixel count of 3840 x 2160 in each of R & G & B. Every camera tested except the Sony F65 has thus far failed to reach this – mainly due to the limitations of using a Bayer-patterned sensor.
The next rung down is UHD Tier 2 and to qualify cameras must have a sensor pixel count greater than 2880 x 1620. A Tier 2 camera should process in at least 10-bit and can be 4:2:0, but 4:4:4 is preferred. Strictly speaking it must also have timecode genlock and internal recording at 200 Mbps or higher for 4K. Cameras that have met this include the ARRI Alexa, Amira and the Canon C300 mkII. These criteria would automatically disqualify the a6300 and most other smaller cameras for Tier 2 acquisition in the full EBU test, but given the hype around the a6300 we were really curious to see how its image stands up against the higher end cine cameras. Could it get anywhere close to meeting this level of image?
You can see the full EBU Tier criteria here.
For the test we used an a6300 loaned by Sony. Test images were recorded internally to SD card, and also externally to an Atomos Shogun 4K recorder.
Resolution and aliasing
So to the results. First up, Alan looked at the resolution and aliasing in HD mode. He recorded both internally and externally in PAL mode and straight away the camera began to exhibit unexpected behaviour. Alan explains that “resolution does not reach the system limits and there is coloured aliasing both vertically and horizontally. There is some diagonal luma aliasing.”
Alan then tested with differing levels of detail (sharpening) applied. He noted that a level of between -2 and -4 should be a ‘good compromise’, but that even set like this it will not eliminate the aliasing. Its also worth noting that a less sharp lens like a kit zoom might reduce the severity of the aliasing as well.
Using S-Log things appear to improve, but Alan noted that when the flat log image is made more contrasty in post production it “will return the aliasing, so this is not a solution.”
He suspected that “the down-scaling from 3840×2160 to HD has used a simple algorithm, probably linear or cubic interpolation.” Simply put, the HD images from the camera are not really suitable for broadcast.
Next, Alan examined the UHD 4K record mode. Here things started a little better. He noted that “the camera is resolving detail all the way up to the limits (of UHD resolution)”, but then discovered that there is no 4K optimised optical low pass filter in front of the sensor and as a result aliasing occurs. He said: “Detail at higher frequencies is reaching the sensor and causing more out-of-band aliasing.”
Alan then went on to test the different detail settings and found that a lower value between -3 and -5 may be acceptable, but just as with the HD image, “aliasing remains a problem” even so.
Setting the camera into S-Log gave effects “very similar to [the] effects in HD, flattening the image, softening it, lowering the level of aliases, but with the high probability that post-production operations will reinstate the aliasing along with the resolution.”
Colour performance
Alan’s test for colour accuracy was to see how the camera represented a standard Colorchecker chart lit with a known tungsten source. He said the a6300 performance was generally good, “although the orange and yellow patches are a little over-bright, and the red and light skin tone a little over-saturated”.
Dynamic range
This is one area of performance where the different manufacturers do not have a set standard for measurement. As a result, one manufacturer may claim 15 stops of dynamic range (DR) for a camera that has less DR than another manufacturer’s 13 stop camera. To avoid this confusion Alan uses a standardised scientific test that readers can use to do their own comparisons if they wish. It is worth noting that even if a camera scores well in dynamic range this does not mean its images are free of excess noise. Alan makes no attempt to define what part of the dynamic range is ‘usable’ in post production – simply what is measurable. This is because different uses have different thresholds for acceptability and different post workflows have varying levels of success in combatting or reducing noise in the shadows and highlights of an image.
To make the measurements Alan again exposed the Colorchecker chart with tungsten lighting. Multiple shots were then taken and shutter speed and aperture was used to vary the exposure (as there are no built-in ND filters). Curves were then plotted using the data from the greyscale patches.
The curves show that in ITU 709 the a6300 has “a dynamic range of not more than 9 stops”. This is to be expected for a 709 image. In S-Log 2 Alan found the camera had a range of “about 11 to 11.5 stops”. In S-Log 3 it “appears to capture about 13 to 13.5 stops”, which is in line with expectations.
That isn’t the end of the story, though. Many users, including myself, avoid using S-Log 3 with Sony’s lower-end cameras and Alan was quick to point out “the problems with using these curves is that the recording is only to 8-bit depth, and so there is a significant risk of contours appearing due to the low slope of the curves. S-Log 2 may well perform better than S-Log 3 because of this.”
Noise tests
The results of this section of the test are very interesting. Alan illuminated a six step grayscale chart with tungsten light. The EBU Tiering system is more concerned about noise at mid-grey (50% video) than in the highlights, or towards black. That said, the noise charts do indicate what’s going on across the range and you can look at the other EBU reports to see how rival cameras compare.
The a6300 was set to 800 ISO in UHD mode and noise reduction turned off.
Starting in ITU 709 gamma he found that while Sony had taken steps to address noise it was not enough to ensure that it stayed below the -50dB level at mid-grey – an EBU requirement for UHD Tiers 1 and 2. It seems that Sony have done clever stuff to reduce the noise near to black, which can be seen in the shape of the curve. However, this has not helped the overall level. Even though I suspected the a6300 to have a higher level of base noise than a camera with larger photo sites like the a7S II, this high a level is somewhat surprising.
When he moved to S-Log he found the curves have a more traditional shape. The noise level is proportional to the slope of the curve with more noise evident the nearer you get to black – as you would expect. Alan noted that “although the curves look similar, their range is quite different; S-Log 3 is more noisy than S-Log 2 near black, by up to 6dB.” As a result, the EBU -50 dB noise threshold at mid-grey is not met by S-Log 2, only by S-Log 3. Even so, Alan suggested that “in general the S-Log 2 curve appears to be the better for this camera. However, only the S-Log 3 curve meets the threshold level of -50dB for UHD, but would require significant post-production work to produce good pictures, which is likely to raise the noise levels.”
Alan also tested the noise levels at higher ISOs. Using the ITU 709 gamma he found that the noise levels did not increase in a linear manner as you might expect as the gain increased. In fact, noise levels dropped between 1600 and 3200 ISO instead of increasing.
Alan suggested that this “probably indicates some noise reduction to achieve target noise levels”, even though noise reduction was turned off in the camera settings. Increasing or decreasing the gain settings did not help the a6300 to get near to passing the EBU -50dB threshold in ITU 709.
Turning in-camera noise reduction on seemed to have little effect on the overall level. At best there may have been a negligible 1dB improvement.
Rolling shutter and motion
Alan’s standard test for rolling shutter is to film a small desk fan that can be adjusted in speed. Then, “if the sensor is being scanned, the downward-moving blade will be widened and the upward-moving blade narrowed. The effect is made much more visible by using a short shutter.” Alan says.
You can see that the effect is extremely pronounced in the images and Alan classes the motion distortion as “severe” and “worse in UHD than in HD”. This is in line with what many other users have observed.
Conclusions
Here are Alan’s conclusions straight from the report:
Testing the camera was made more difficult than expected due to the changes of image size when changing frame rate and output format. It seems that differently-sized parts of the sensor are used for HD and UHD, and for 25 and 30fps shooting. Also, setting the HDMI output did not always produce the expected effect, which has serious implications for the use of external recordings. Since the HDMI output is only 8-bit, there seems to be little advantage in using an external recorder, it only makes life more complex.
The sensor has about 6000×4000 photo-sites, which ought to be enough for good performance even at UHD. The dimensions are 23.5×15.6mm, so the photo-sites are spaced at 3.9μm, meaning that the photo-site area cannot be greater than 15.3μm2, compared with 5μm and 25μm2 for a conventional 2⁄3” camera.
However, the camera fails expectations in several ways. Although the sensor dimensions considerably exceed the requirements of both HD and UHD, the scaling to produce HD and UHD images is unsatisfactory. Resolution at UHD is only adequate, but aliasing is unacceptably high. Similarly, at HD the aliasing is sufficient to cause problems in low bit-rate coding.
The dynamic range is about 9 stops in standard ITU 709 mode (without knee), 11 stops in S-Log 2, 13 in S-Log 3. This is unsurprising since the camera uses only 8-bit recording.
Noise levels are consistently higher than acceptable, and noise reduction is too ineffective to improve things adequately. Only at ISO100 or ISO200 could the camera qualify for HD Tier SP, using EBU R.118 requirements.
The sensor is evidently scanned, the so-called ‘rolling shutter’ effect. Rotating motion is very poor.
This camera is not suitable for use in general broadcasting, apart from Special Purposes where its physical size or shape makes it uniquely suited to a particular type of shot.
My take
Although there have been a lot of very positive things said of the a6300 – including by us – Alan’s findings were rather sobering, to say the least.
And although the camera is a great feat of miniaturisation, the results aren’t great if you want to use the a6300 for broadcast. The principal barriers are the high levels of aliasing, rolling shutter and noise (at 50% grey).
The issues are visible whether shooting in UHD or HD and, interestingly, Alan suggests an external recorder doesn’t improve picture quality greatly as the camera output is only 8-bit – although it does, of course, bring workflow and monitoring enhancements.
The results are likely to be comforting to owners of more expensive cameras like the C300mkII or Amira. The a6300 still has a long way to go to catch the higher end models. However, it is not a simple case of getting what you pay for. Canon’s XC10, which is not much more expensive, qualified for Tier 1 HD production,and although it didn’t meet the 4K standards due to the hardware, its noise performance did meet the criteria for both EBU Tiers of UHD.
Reading the report certainly made me hesitate to use the a6300 in the way I had planned to – as a B-camera to an FS7 or F5. I will now reserve use to the kind of special purposes work Alan sees it as suitable for – where the size and AF benefits will get you a shot you couldn’t have got another way.
‘Let’s Dance’ – Shot entirely using autofocus on the Sony a6300 in 4K from Dan Chung on Vimeo.
Sony’s tiny S35 shooter is still an impressive piece of packaging – and a showcase for the AF systems that are surely going to become standard on prosumer video cameras. If your work is mainly for online and you are happy with the results the a6300 gives then you can choose to largely ignore this report. If you work for TV then don’t expect the a6300 to replace your C300 or F55 just yet…
The full Alan Roberts Newsshooter report is available to download from the EBU website.
