UPDATED at bottom!

DISCLAIMER:

Much in this post is speculation, especially the theories about how the Canon CMOS sensors record video and what is theoretically feasible to achieve with a firmware upgrade. These speculations are of course not unfounded and rely on information given by other users and technology experts. I am not a technology expert or engineer, and I speculate purely from my – maybe naïve – standpoint as a DoP and filmmaker that is keen on the things to come.

Thanks for checking back! Last time, in Part 1 of this two-part series, I looked at the disadvantages of shooting video with these shiny new DSLRs we adore so much. I focused mainly on the problems that are caused by the CMOS sensors that are optimized for still image shooting.

This time, I will focus on the good bits. To be more precise, I will focus on what THEORETICALLY would be possible with DSLRs if the manufacturers or some software crack like Trammell Hudson (with his Magic Lantern alternative firmware for the 5D Mark II) would come up with a new firmware. So let’s call it a little vision. It would be a way to put some of those still image pixels to good use for video shooting.

Video acquisition on a DSLR in practice:

I scribbled up this little graphic to show you how the acquisition of 1080p video works on the example of the Canon EOS 5D Mark II (it works the same way in the 7D or 550D/T2i, with the only difference being that their CMOS sensors have a slightly smaller resolution and are smaller, APS-C sized).

Canon doesn’t have official spec sheets that explain what exactly they are doing to acquire video with these cameras. However, many people who are much smarter than me have come up that this must be the way how they do it. I’ve summed up all the facts I could find on various web forums and posts and summed it up in these graphical interpretations. Please be aware of the fact that much of this is still speculation, but this is the most likely way video seems to be recorded on DSLRs.

When shooting 1080p video on the 5D Mark II, only every third line of the 3,159 lines of the 16:9 area of the sensor is recorded, which results in 1,053 lines (in total, the sensor consists of 3,744 lines, but the additional lines are only used for the different aspect ratio of still images).

Every line consist of 5,616 pixels each and is downsampled to 1,920 pixels, while the 1,053 lines are interpolated to 1,080 pixels. The final 1,920*1,080 raw video is immediately compressed into H.264 in a QT-MOV container and then written onto the CompactFlash memory card.

This constant twisting and wrenching of the images results in awful aliasing artifacts and increase the effect of the rolling shutter, both of which I explained in my last posting.

So let’s look at what this high-resolution CMOS sensors might actually be good for.

The 640*480 pixels crop mode on the Canon EOS Rebel T2i / 550D:

Let’s start with the obvious things. As all you T2i/550D owners out there will already know, our new little DSLR camera sports a new video mode that is lacking from its bigger siblings: the 640 x 480 crop mode.

It just takes the 1.920  x 1.080 resolution and crops everything around it to leave us with 640 x 480 pixels. (At half the data rate, about 23 MBit/s.)

Now you might say this is not an option that is viable for a filmmaker, and you are right. But it could also be seen as a step in the right direction.

I recorded a test clip in the 640 crop mode and found it uses half the data rate of the 1080p mode (23 MBit instead of 44-45 MBit). It might have been useful if it used 44 MBit as well as this would have increased the quality of such a pseudo-telephoto mode quite a bit.

Unfortunately, it also uses line skipping to acquire these images. In fact, it seems to use the same line skipping mode as the 1080p mode, with the only difference being that the majority of lines are disposed during the downsampling process.

The theory: 1080p crop mode – no line skipping, no aliasing, massively increased “focal lengths”

Think about the alternative: Why not make an OPTIONAL crop mode without using line skipping in the process? Line skipping wouldn’t be necessary if you used only part of the sensor anyway. It would even be possible to use the exact same pixels without involving any downsampling or interpolation whatsoever. Of course that could not replace the line-skipping as we still need the whole sensor real estate, but it could be employed as an additional mode and provide us with a “free” additional fixed “focal length” / decreased field of view.

NOTE (this has kept coming up in the comment section): I see this as an OPTIONAL additional feature that would not at all replace the way video images are currently acquired with these cameras. But it could be a feature for sports and other situations where such a “focal length” / small field of view could be extremely helpful.

The recorded image would quite simply be only a 1,920*1,080 pixels area in the middle of the much larger 5,616*3,744 pixels CMOS sensor of the 5D Mark II.

As a result, we wouldn’t suffer from many of the issues deriving from the twisted way HD video is usually recorded on a hybrid DSLR camera. No line-skipping means less aliasing and also less ‘skew’, aka ‘rolling shutter’.

Using a 200mm telephoto lens would instantly give you access to effective 600mm on a 5D Mark II, and getting rid of all the disadvantages of HDSLR video as we know it now.

It would even be thinkable to introduce an “in-between-mode” with line-skipping only every second line. This could give DSLR filmmakers access to two additional telephoto stages with every lens. Effectively, you would cover a much longer area of focal lengths with a single lens.

So the only question remains: Is this technologically feasible? Maybe, maybe not. Geniuses like Trammell Hudson, who developed the alternative 5D Mark II firmware which allows audio monitoring and other goodies usually not available in the 5D Mark II, have said that it might be an option. It might simply not possible with the current generation of cameras because of lacking CPU horsepower, but it might be feasible with future generations – even if the manufacturers themselves won’t allow it, because they’d probably fear to sell less lenses.

Just imagine: One 70-200mm telephoto lens to rule them all, going up to effective 600mm on a 5D Mark II (and of course even further on crop-sensor cameras like the 7D or 550D/T2i).

I would really love to have a discussion about these ideas, I’d love to hear your thoughts!

Also please check out this Cinema5D forum entry that I just discovered – they came to similar conclusions like me, maybe some of these guys want to talk about it right here.

EDIT:
Somebody in a forum brought up that the DOF would change and that it would look like the image from a 1/3″ camcorder. That’s just plain wrong.
The DOF will stay EXACTLY the same. Think about it: The projected image is still the same, therefore the DOF is the same. You only take the middle part of this image and acquire it in 1080p, which results in nothing more than a 3x decrease in field of view > a small portion of the entire sensor area acquired in Full HD resolution without line skipping (1:1 pixel ratio)!

DOF would only be different if you would somehow change the projected image onto a smaller area, but that’s not what we do here.

It’s true that it wouldn’t be as shallow as on a real 600mm lens if you use a 200mm lens with 3 x magnification factor, but it wouldn’t be less shallow either. It would just stay the same of course. As an additional advantage, our “fake 600mm” would be much more light sensitive than your average “real” 600mm lens that would likely cost thousands of Euros (or dollars).

Thanks to fellow filmmaker and photographer Peter Hainzl (I know him from Twitter) who really gave me the idea about a “digital zoom for DSLRs” in the first place.

On a completely unrelated note: I was shooting with him at a local skate park in Vienna recently, check out the video that he cut from the footage he shot – I’m in it, too:

EDIT 5th May 2010:

Some people in the comments keep claiming that we would lose the beloved shallow depth of field if we were to employ an optional window mode. That is just plain wrong though, because the projected image of the lens wouldn’t change a bit. We would only take the middle part of it and that’s about it.

Commenter “Blendin” explains it best:

“The rule is, same lens, same DOF. Nothing more, Nothing less.. nothing to do with the sensor (size, or size of the pixel). so, with 200mm you got the DOF of 200mm. the crop fator only make the image bigger or smaller.
With the same glass, how come people think the DOF of the image can be different ?”

He might not have phrased it entirely correct, but what he means is a simple fact.

Think about it:
YES, we take a smaller sensor area, but that does not at all mean it would look the same as on a small sensor.
WHY? Simply because we also only take SMALL SECTION OF THE IMAGE that exact same lens is projecting. IT IS THE SAME IMAGE, we just take a high resolution “window” right out of the middle of it. Nothing changes about the physics of the sensor OR the lens – which would be necessary to constitute a change in depth of field.

I hope now everybody gets it 🙂