An HD Primer
Douglas Spotted Eagle/VASST Instructor ©2007 Sundance Media Group

HDV:What You NEED to Know 2nd Edition

 

 

 


Never before has the video industry seen such an explosion of formats as we're experiencing today, and along with the plethora of formats comes confusion about what HD is and what it isn't. Rarely does a day go by that we don't hear the phrase "True HD" or "Full HD"  in the context of comparing camcorders and/or formats.
 
If the camcorder displays 1280 x 720, or 1920 x 1080 on OUTPUT, it is indeed a "High Definition" camcorder. Truly!

Hopefully this simple statement clarifies HD. I believe the confusion stems from marketing campaigns in the early days of low-cost HD camcorders and discussions of imager sizes and recorded image sizes.

Although some manufacturers, evangelists, or editors might suggest that the sensor/imager of a camcorder determines whether a camera is an HD camcorder or not, the imager/sensor size isn't directly related to whether a picture is HD or not. DSP may be employed to create an HD output from the camcorder. In theory, a camcorder manufacturer might take a Standard Definition imager and use DSP (Digital Signal Processing) to upsample the SD image output from the imager, and store the image on tape, hard drive, DVD, or memory card as an HD image. Of course, recording a VHS signal to HDCAM might make it HD, but the picture quality obviously comes into question.

HD camcorders range in price from $800.00 to $800,000.00; this article is  predominantly focused on the lower and mid-level camcorders.

Imagers

Various camcorders offer different sizes of imagers. The HDV format for example, is always a 1280 x 720 or 1920 x 1080 display, but various sized imagers may be employed to reach the display size. For example, the Canon XLH1 camcorder uses a 1920 x 1080 imager to generate a 1440 x 1080 recorded image. The JVC HD200 uses a 1280 x 720 imager to create a 1280 x 720 recorded image. The Sony Z1 uses a 960 x 1080 imager to deliver a 1440 x 1080 recorded image. The Panasonic HVX200 offers the smallest imager of any HD camcorder at 960 x 540, slightly smaller (vertically) than a PAL SD imager, recorded to a 960 x 720 frame size (in 1280 x 720 modes). All of these camcorders upsample or downsample the recorded image, excepting JVC's HDV camcorders and Sony XDCAM EX. Their format is a full-raster format (meaning imager matches recorded frame size).  XDCAM HD offers a 1440 x 1080 recorded image derived from a 1920 x 1080 imager, while XDCAM EX offers a 1920 x 1080 recorded and displayed image.

Imagers range in size from 1/4" to 2/3", depending on the camcorder model. The Sony HVR V1U features 1/4" imager set on the diagonal, providing a sample equal to 1440 x 1080, but using unique (and oft-debated) technology, the image is sampled at 1920 x 1080. XDCAM HD features a 1/2 imager, while Varicam and HDCAM products offer a 2/3 imager. The larger imagers are not only more light sensitive, but they also allow for shallow depth of field and a smoother image in most shooting situations.

Smaller imagers packed with many more pixels than their SD counterparts are prone to being low-light challenged. More pixels in the same space comes at the cost of sensitivity, although new imagers and DSP are making tremendous strides in low-light situations.

SxS™ memory card prototype

SxS card

Storage

HD images are stored on a variety of storage formats. The Sony HVR Z7U stores HDV on a CF card and/or HDV tape simultaneously, while the XDCAM EX stores media on the new SxS (pronounced "S by S") cards that read incredibly fast. AVCHD is stored on memory sticks, DVD, HDD, and SDHD cards. XDCAM is stored on Sony Professional Discs, which uses Blu-ray technology to store HD media on optical storage. Panasonic uses the P2 card system, which is a PCMCIA card format. Grass Valley Infinitycam stores on Iomega REV discs, while RED and SI record to hard drive units. Higher end camcorders may record to tape, or use dual-channel SDI output to large hard drive arrays.

Compression

When dealing with the various flavors of MPEG (MPEG 2/AVCHD), the primary consideration is the bitrate at which the video file/frames are encoded. Ranging from 11Mbps to 50Mbps and differing ratios of compression, it can easily become confusing. Don't confuse compression with color sampling.
AVCHD varies the bitrate from 11Mbps to 16Mbps, depending on the manufacturer and model of the camera. HDV offers two bitrates that are constant bitrates (CBR). 720p offers a bitrate of 19Mbps, while 1080i/p is encoded at 25Mbps. XDCAM offers a variable bitrate (VBR) of 35Mbps and 50Mbps (depending on the model). XDCAM EX offers a VBR of 35 Mbps or a CBR of 25Mbps. MPEG is the same format as delivered over satellite and cable, DVD, Blu-ray, or HD DVD. It's possible with some camcorders to shoot to disc or card and play back the footage immediately on a BD system.
HDV compresses media at a rate of approximately 20:1, while AVCHD compresses at nearly double the HDV ratio. HDV may be used for broadcast in many applications, whereas currently, AVCHD is predominantly a consumer format, and likely wXDCAM EXill be for some time to come. The benefit of higher compression is that it allows for storage of high resolution media on small or slow format devices such as memory stick, DVD, or hard drive. The drawback to higher compression is that color and detail may be compromised, particularly in scenes containing complex color and complex motion. An AVCHD camcorder may manage broad latitude images (lots of dark black and bright whites) while holding static, and may manage low latitude images with some motion, but combining wide latitude with high motion is very challenging for highly compressed formats. HDV also suffers in some circumstances of high motion and extreme chroma/color.

MXF

This is the acronym for Material eXchange Format. XDCAM, XDCAM EX, Panasonic HVX series, and other camcorders use the MXF format to package video files complete with essence markers/notes from the shoot ranging from geographic locations, preferred shots, shutter speed, framerate, exposure, etc.

MXF is an object-based file format that wraps video, audio, and other bitstreams ("essences"), optimized for the interchange or archiving by the content creator, and intended for implementation in devices ranging from cameras and videotape recorders to computer systems. If for no other use, it's a terrific format for exchanging media from one NLE to another when EDLs aren't easily shared, and multiple transcodes should be avoided.

Anamorphic

Most of the HD formats are anamorphic, meaning that pixels are not square in the recorded image, and must be stretched in the NLE or display device. For example, the 1080 HDV format uses short, wide pixels at an aspect ratio of 1.333, which turns a 1440 pixel wide image into a 1920 displayed image (1440 times 1.333=1919.52). XDCAM uses the same anamorphic format, as does HDCAM (this is the inspiration for this primer, as someone recently emailed me saying that their V1U wasn't "full HD because it recorded a 1440 x 1080 image" and he wanted it "true HD like HDCAM").
Additionally, it's important to note that all HD is widescreen. There is no 4:3 display of high definition media.

Color Sampling

Most of the various MPEG-based camcorders sample color at 4:2:0. There are camcorders using MPEG that sample at 4:2:2, but they're not shipping yet. 4:2:0 provides a more robust color sample than does DV. Some feel that 4:2:0 color sampling isn't robust enough for chromakey work, where others feel it's adequate for a good key. The primary benefit of 4:2:0 sampling is that it does not require a chroma re-sample when authoring a DVD or putting media on the web. It's the same color sampling scheme used for broadcasting HD signals as well.

Color sampling courtesy of Wikimedia

Most uncompressed systems sample at 4:2:2. Sony HDCAM samples the image a 3:1:1. Technically anything that isn't 4:4:4 means that the image is compressed, as the full color sample isn't taken. Additionally, very light compression is often referred to as "uncompressed." Uncompressed HD signals require significant bandwidth and tremendous storage space. Getting the uncompressed signal from the camcorder to the computer/hard drive generally requires an HD SDI card such as the Black Magic Design Decklink or Intensity, AJA Kona/Xena, or Bluefish 4:4:4 cards.

Image courtesy Black Magic Design

So what's up with all the "Full HD" and/or "True HD" discussions?

"FullHD" is a marketing term related to a 1920 x 1080 image. That's all. Outside of indicating 1920 x 1080 video output/display, it's meaningless.
"True HD" is often used to describe uncompressed HD vs compressed formats. Typically it's used to distinguish HDV from other formats, but overall, it's a silly term as it is so amorphous. What is "true HD" to one person isn't the same "true HD" to another. For example, given the imager size and recorded frame size of the Panasonic HVX 200 camcorder, the color sample is so small, it can barely be considered the 4:2:2 that is advertised. HDV offers a more robust color sample with an identical 1/3 imager, but is sometimes referred to as not being "true HD" when compared to the DVCPro HD that the HVX uses simply because of the 4:2:2 color sample applied to the lower resolution camcorder. On the other end of the spectrum, the Infinitycam is referred to as "true HD" in the same sentence/context as a Panavision Genesis or Grass Valley Viper. All of these are silly comparisons and tremendously confusing to camera operators seeking information about HD camcorders. Some aspects are very difficult to comprehend at a lower level anyway. For example, a Panasonic Varicam samples color at 4:2:2 on a much smaller imager than Sony HDCAM (which records a 3:1:1 image) so the user is left with a better color sample at a lower resolution. HDCAM samples 3:1:1 with a 1920 x 1080 imager, recorded at 1440 x 1080. Varicam starts with a 4:2:2 color sample, 1220 x 720 imager recorded to a 960 x 720 resolution, and then upconverts the image on playback to 1920 x 1080. Which is better? It's a moot discussion, but this goes towards my original point of "what is "true" HD.

The issue is somewhat related to imagers, color sampling, etc, but the bigger divider of the various levels of HD is the compression used in the recording of the picture, not the size of the imager or color sampling.

Today, the issue of greater importance is comprehension of codecs and bitrates. The Compression-Decompression of an image is far more important than the actual imager on most camcorders. This is where a discussion of various codecs and bitrates come into play, as the codec will more or less describe the category from which the image quality is derived.

These bitrates can be loosely divided into the following categories:
1-16Mpbs=consumer/soccer mom/very low end pro-sumer, education
17-25Mbps=Pro-sumer/low end professional, corporate, religious
25-50Mbps=Professional, ENG, Independent film, commercial production
50Mbps-Uncompressed=High end, professional only

However, a high bitrate does not assure a quality picture. Various formats may "cheat" by using high compression at a high bitrate, reduce spatial or temporal resolution, or other DSP techniques.

There are essentially four formats currently used for HD production;

DVCPro HD (Very old format based on DV technology, small frame format)

AVCHD (many variants in terms of compression, size, still evolving, extremely high compression)

MPEG 2 (older technology, still evolving/developing, standard of broadcast, low bitrates are challenged)

Uncompressed/lightly compressed (expensive, but standard of high end production, requires tremendous storage space)

There are of course, others, but the majority fall into one of these four categories. The major cameras that don't would include:
Grass Valley Infinitycam (JPEG 2000 codec)
RED One (REDcode  codec)
SI2K (CineForm codec)
Panavision Genesis

Expect to see many, many more codecs come into play as storage moves away from tape. DPX is one of several upcoming codec flavors that may well find their way into camcorders at some future stage. Codecs are where the money is at, licensing them, keeping them proprietary, etc. It's in every developer's best interest to keep a codec proprietary, while it's in the consumer's best interests to have access to all codecs. One example is DivX recently acquiring Main Concept; it will be a very interesting future for all of us using Main Concept indeed.

A lot could be said about MPEG (as it's by far the most common acquisition, editing, and delivery format), but the primary benefit (and drawback) of MPEG is the GOP or "Group of Pictures" that gives MPEG the extreme efficiency. Made up of three frame types, the stream includes "I" frames, or "Inter" frames which include the entire content of the picture. "B" frames, or "Backwards Predictive" frames look backwards at the I frames and P frames, and the "P" frame or Predictive frame that looks forward at the frame, constantly comparing frame content.

Three different MPEG streams, the top version has no P or B frames, and is the least efficient format. The second stream records I and P frames only, and the bottom stream records I, B, and P frames.

There are different methods in which the GOP may be recorded. The number of frames between the I frames may vary. The longer the length between I frames, the more efficient the stream will be, but at the same time, this can compromise the image quality. I&P frame recording is efficient, yet the most efficient stream while yielding the best image is the IBP frame sequence.

The first encoded picture in a GOP is always an I picture. Typical 1080 GOP length is 15 pictures with the following structure (in display order):

I B B P B B P B B P B B P B B I (as seen above)

JVC offers a 6 and 12 image I frame sequence, while Sony and Canon use a 15 frame sequence in their HDV products. For further research, Google the term "DCT" frames. One small comment, B & P frames can act as DCT or partial DCT frames, thus allowing the format to deal better with rapidly shifting color, movement, and latitude. In any event, what makes MPEG work so well/efficiently, is the GOP format. In times past, NLE's couldn't identify the I frames, which is the only place an MPEG stream could be cut. Technology of today allows extremely accurate editing of MPEG, and the concerns of editing on I frames-only are a thing of the past.

Codecs both in the NLE and in the camera will constantly evolve and shift over the course of time, providing higher quality images over lower bitrates. MPEG variants will stay the standard for a long time to come, with virtually every camcorder manufacturer developing MPEG-based products. This shouldn't scare you, but rather provide an exciting opportunity to learn what works best for your production needs.

A few codecs of outstanding quality:
CineForm
Sheer Video/Bitjazz
REDcode Raw
Avid DNX
DivX
Canopus HQ
Sony YUV

 

10 bit vs 8 bit vs....

By and large, most of the HD camcorders are an 8 bit format. 10 bit recording/codecs come into play at the capture/Xfer stage, or after the Xfer stage during a transcode, depending on the NLE being used. 10 bits offer greater color accuracy and saturation. 10 bit codecs also offer reduced or eliminated "banding" or gradients often seen in slight color variations. Yes, even though video is acquired at 8 bits, there are significant benefits in most workflows to working with a 10 bit file format, particularly if a lot of color correction is part of the workflow.

Some NLE's and compositing tools offer 32bit floating point processing (or higher). This isn't relevant to the 10 bit question, but rather the internal workings of the NLE and how capable the accuracy of the processing may be. There is no 32bit delivery format. In fact, at the end of the day, the majority of your work (assuming a small production studio) will be delivered in an 8 bit format. However, editing with the higher 10 bit files generally assures a better picture integrity when it comes time to output an 8bit master.

NTSC and PAL

NTSC and PAL designations are effectively dead so far as HD is concerned. No longer do we have different resolutions for NTSC and PAL. All HD, regardless of anything else, falls into the two resolutions previously mentioned. Frame rates for "PAL" are 25p, 50i, and 50p. Framerates for "NTSC" are 24p, 30p, 60i, and 60p, but the resolutions are the same. The colorspace is the same, and the color sampling format is the same. HD falls under the view of the ATSC or Advanced Television Standards Committee.

Blu-ray, HD DVD, and other formats

HD currently is indeed difficult to deliver to the masses, not because the technology isn't there, but because the masses don't currently all have playback devices for HD content. Blu-ray and the Toshiba HD DVD format are battling it out very hard, and the 2007 holiday season will play a big role in determining the winner of the "war" between the formats. The smart money is on players that play both formats, because at the moment it looks like everyone is a loser in this battle royale. Both formats support identical codecs. BD (Blu-ray) supports MPEG 2, AVCHD, and VC1, as does HD DVD. BD does offer significantly larger discs and more importantly, 60p output. BD is easy to author in tools such as Adobe Encore CS3 and other popular DVD authoring tools. BD burners are available at speeds of up to 4x, and at affordable prices. HD DVD burners are hard to find. Delivery of HD at this moment, is predominantly through the web, on hard drive, or on flash-based players.

Canon AVCHD Canon XH G1 XDCAM EX Sony XDCAM F350 Grass Valley Viper Filmstream

Eventually, you'll likely be moving into HD, whether it's AVCHD, HDV, XDCAM, Infinitycam, Varicam, or HDCAM. The learning curve can be steep if you pay attention to the numbers and the manufacturer hype, or it can be reasonably smooth if you use your eyes and common sense. There are many HD pioneers out there, talk to them, talk to producers actually USING HD in whatever their work may be, regardless of their delivery. The noise ratio of those playing with HD versus those genuinely using HD is incredibly high, and it's difficult to wade through the morass of conjecture and measurebating. Rent, borrow, or hang out with various HD camera formats. In the sub 7K market, it's cut-throat, with most of the cameras offering very similar qualities. It becomes even more murky once one rises above the 7K/below 20K mark, with excellent choices from all of the manufacturers. Who ever is newest to market usually has the best tools, while others are typically trying to do the "Take out" thing by saying "Yeah, their product is cool, but wait'll you see OUR new technology." At the end of the day, it boils down to this; "Is not having an HD camcorder hurting your business right now?" Or, "Can I increase my revenues by making the investment in HD right now?"

Our industry has reached a point where it makes little sense in my opinion, to continue shooting SD. Although the end product may be SD, the HD footage in archive, as well as learning the format and its relevance to various workflows, is a good thing. It's all moving forward very fast, and the last thing anyone wants, is to be left behind, so I'd recommend you start the journey sooner, rather than later.
 


-douglas spotted eagle

For further reading:

~HDV: What You NEED to Know (VASST)
~Digital Video and HDTV (Poynton)

BOOKS from VASST

Instant Digital Audio

Instant ACID

 

 
    Search the VASST site for other project files, tutorials, reviews, and more!
 View other VASST training tools, FREE! Click the link below to be transported to the VASST resources pages.
VegFiles
Plugins
Scripts
Hardware

 

Tutorials
Themes
Forums/Groups
HDVPortal

 

Books/DVDs
Reviews
FreeStuff
VegasFAQ's