Guest post by Geoff Boyle:
You can view the test here.
Background to the tests
The origin of these tests goes back a long way. I started testing film stocks in 1975 to see what their dynamic range was and what their real Exposure Index (EI) was, as manufacturers don’t always use measuring conditions that reflect a real-world shooting situation.
After I established CML in 1996 I started to publish my findings with films and also list GRR’s (Geoff’s Recommended Ratings) for various stocks. These ratings came from my measurements of the stocks in the real world.
Initially we had a fairly controlled environment: the lab processed in a predictable way and they had standard printer lights. These would have been 25-25-25 when the batch was new but would change over time. As long as I knew what the Kodak reference strips printed at a particular lab looked like I would know if my exposure was correct. Also if I preferred the look of a stock slightly overexposed and printed down I could specify that they printed at say four points darker than normal. This gave a very consistent, predictable result.
When we went to scanning from the neg I could still get the lab to check the neg density and I could also ask for the telecine or scanner to be set up using TAF (Telecine Alignment Film) which was a reference neg from Kodak. If everything was set up so that TAF looked right then if my neg was exposed properly it would look right.
Once we went to digital origination and digital distribution all reference points vanished. Anyone could alter my pictures at any stage and it was very hard to establish just what it was that I’d shot.
I tried using LUTs to control what happened to my images but this was totally reliant on people in the post process actually using them. Often they wouldn’t.
ACES: a reference system for digital
ACES is a control system that finally brings some order into the madness of digital image creation.
There is a common ACES colour space, a common RRT ( Reference Rendering Transform ) that converts the scene-referred colorimetry to display-referred, and resembles traditional film image rendering with an S-shaped curve. It has a larger gamut and dynamic range available to allow for rendering to any output device (even ones not yet in existence).
The RRT is not something to think or worry about: it’s just there!
What matters to us is the IDT (that transforms the captured image into ACES colour space) and the ODT (that transforms the graded image into your desired viewing colour space). It frees you to grade in say P3 but output in rec709 and get a predictable result.
So, finally a fixed reference system. If I shoot digitally and use the relevant IDT and ODT then my result should be totally predictable.
My initial personal tests using old rushes were encouraging so I decided to run the next CML-CMIR test series totally in ACES.
Optimistic manufacturers, a unique chart, unexpected results
The results were interesting: they revealed that some manufacturers were being a little optimistic in their EI ratings. There’s actually no standard way of measuring this so who can fault a manufacturer for using a system that makes their camera look good?
However, I’m not interested in what looks good on paper, I want to know how to expose the camera to produce the best pictures I can get from it.
We had serious issues with one of the cameras: the Varicam didn’t produce the results that I expected and it has taken four months and the assistance of Nick Shaw to establish what the problem was. We now know that Panasonic and Resolve don’t mix properly: it looks like the rushes are in video range when they are in fact in extended range. A manual adjustment in Resolve fixes this.
We also established that some cameras have not only very different colour responses in tungsten and daylight but that they also have very different EI’s.
The chart that DSC labs made to my specification that was used for these tests does not conform to any accepted colour space, neither does the real world, a chart that limits itself to what can be reproduced in rec709 or P3 doesn’t reflect the real environment. I wanted a chart that had extremely high saturations because I was fairly sure that all single sensor cameras would have problems with them.
The results are all at http://www.cinematography.net/CML-UWE-tech.html and my comments on the Varicam will be updated to reflect the new information we have. Essentially the highlight response remains as we first published but the shadow response is now at least as good as an Alexa.
Results:
Sony F65 Tungsten
+3.5 Red is not clipping at +4 but Blue and Green are.
Unity + 0.5 this would indicate an EI of 500 to 640
-1 noise base
DR is therefore 11.5
Alexa Tungsten
+4 although there is no clipping compression does come in from +3
Unity 0 so an EI of 800
-3 noise base
DR is therefore 14
Alexa Daylight
+4 with the same highlight compression as tungsten
Unity 1 so EI of 1600, I have double checked this as I always use 400
which gives best SNR
-4
DR is therefore 15
Varicam PL Tungsten
+3.5 actually a little above this but not +4 say +3.66
Unity 0
-2 Noise base
DR is 12.5
Varicam PL Daylight
+3.5 with the same comment as tungsten
Unity 0.5 so EO of 1000 to 1200
-2 noise base with Varicam IDT
-3.5 noise base with Slog3 IDT
DR is 12.5 with Panasonic IDT or 14 with Sony IDT(!)
Dragon LLD Tungsten
+4 noting a curve change above +3
Unity +1 so an EO of 400
0 noise base, very noisy as measured but visually less so
DR measured at 11
Dragon LLD Daylight
+3.5
Unity +.5 so EI 500 to 640
-2
DR therefore 12.5
Dragon Skin Tungsten
+3
Unity +1.3 so an EI of 160
0
DR therefore 10
Dragon Skin Daylight
+3
Unity +.5 so EI of 250 to 320
-2.5
DR is therefore 12.5
Canon C500 Tungsten
+2
Unity 0 so EI 800
-2
DR is therefore 11
Canon C500 Daylight
+2
Unity 0.5 so EI of 1000 to 1200 this matches my shooting experience of 1200
-3 maybe -3.5 it’s very close
DR therefore 12
Details of the test:
The tests were run by CML with the assistance of CMIR in January 2015 to try and establish how the most commonly used higher-end cameras responded to ACES.
All cameras were set to the manufacturers recommended EI.
Readings were taken with a Minolta 6, a Pentax Digital Spot and a recent Sekonic lightmeter.
Lighting was with Trucolor XH (2 * HS) with 3200 and 5600 panels, backlight was from matching Matchstix.
The levels for the Dragon skin tungsten tests should all be read bearing in mind a one stop offset, so plus one is really zero and plus four is really plus three. This was due to a problem getting enough level for 400 EI Tungsten. The daylight tests are as stated.
CT readings were taken with a Minolta CT3 and a Sekonic, the Minolta gave 3200 and no M/G shift and 5600 again with no M/G shift, the Sekonic agreed with 3200 but read 5900 for the 5600 light.
The scene brightness range is seven stops so taking that seven and adding the plus and minus values will give us the usable filming range. Please note that this is very different to the values produced by a Xyla chart which as far as I’m concerned is only good for manufacturers’ PR blurbs and has no use in real filming situations.
All material was processed in Resolve 11.3 with the appropriate IDT, adjustments were made so that the middle grey patch read 512, contrast was reduced to 0.9 overall to contain extreme peaks of colour.
White clip levels were easy to find as were the ‘unity’ levels i.e. the levels where no gain alteration was required, this gives us the ‘real’ EI of the camera.
Shadows were a different matter, with most it was possible to discern the small chip within the black on the chart down to levels that gave so much noise they were unusable.
I therefore established a point that showed clearly using the vector-scope in magnify mode and when a camera reached that noise point that was the level I have published. The pictures may well be usable below this point but there has to be some kind of level playing field. In fact what happens after this point varies hugely from camera to camera and is totally down to personal preference.
The Varicam had serious issues with shadows blocking up and just for the hell of it I tried using the Sony Slog3 IDT at levels below the point that the Varicam IDT had failed. This gave very surprising results.
I have listed the cameras in order of cost.
The Sony F65 is Tungsten only because my DIT managed to destroy the daylight rushes, he was distracted playing with the pictures from all the cameras. Maybe this is a clear argument for an unthinking data monkey who just copies and checks that it has copied…
