4K Imaging In Augmented Reality

The buzz surrounding 4K, or Ultra High Definition, has played a leading role at every key industry trade show during the last twelve months, writes Andrew Butler, applications manager, Vitec Videocom.

There has been a raft of new products released to support the 4K trend at every stage of the broadcast workflow. At point of capture, the latest professional Ultra HD camera ranges from the likes of Sony, Red and Blackmagic, have further fuelled the interest.
The 4K spotlight has also been shining on the need for the highest quality lenses capable of delivering enough resolution to project a crisp and error-free 4K image on to the camera sensor. There is also the matter of needing adequate bandwidth in cabling and the latest wireless technology to transfer these higher resolution images.
While an increasing number of broadcasters and telcos currently have the infrastructure in place to make 4K a mass market proposition, advances in encoding such as HEVC and possibly even Google's VP9 are set to be key enablers for making UHD more practical. HEVC's compression rate will probably see UHD needing only between 10-20 per cent more bandwidth than full HD, making it entirely feasible for the mass market. In addition, when it comes to infrastructure the wider roll-out of fibre is expected to iron out these issues in the medium term.
As this occurs, rights holders and content distributors will be looking for additional revenue streams to offset the costs of upgrading productions and infrastructures to 4K standard. One method open to them is the use of augmented reality advertising during live events, notably sport where it has become a common feature.
High precision movement control and the performance of the lenses is critical to success for any 4K application, however there is an additional consideration to factor in when adding augmented reality to the mix. This is to provide sufficient resolution of pan and tilt positional data for the UHD frame when using shots at the extreme tight end of the zoom range.
The aim of any augmented reality system, such as those used for inserting a logo for a sponsor onto the pitch during an international rugby match, is to seamlessly composite an image of the real background with a computer generated view of a 3D model to the extent that it appears physically 'real'. This requires highly accurate pan and tilt positional information for the camera, which is typically provided by an encoded head such as the Vinten i-Series encoded heads.
Encoders in the support fluid head measure the direction of the camera and the encoders in the lens measure the position of the lens barrels or motors. Any minor delay or backlash between optics and encoders results in an error in the data sent to the rendering engine. The erroneous data will define the viewing vector and field of the virtual camera viewing the 3D model, which in turn will damage the illusion of both images existing on the same plane.
In developing the Vinten i-series products, the standard resolution required was calculated to meet the requirements of a 1080p high definition production using one of the longer lenses, such as a Fujinon XA101x8.9 BESM, that a typical sports production might deploy.
When the lens is zoomed all the way in, its horizontal angle of view might be just 36 minutes, or 1/600th of the 360 degrees the camera head could pan. On a 1080 HD frame, the horizontal pixel count would be 1920. By multiplying the horizontal pixel count of 1920 by 600 that informs how many data points would be needed in the head's 360 degree rotation in order to resolve 1920 pixels in the 36 minute of horizontal view, the highest telephoto point of the zoom lens.
The overall result is it would take 1.15 million data points through 360 degrees of rotation in order to adequately describe the pan angle of that camera, zoomed all the way in, at pixel level. To put that in perspective, Vinten's i-series of encoded heads provide a minimum of 1.48 million data point per 360 degrees of panning rotation, which surpasses the level of performance necessary to resolve each individual pixel.
When the positional data resolution in an augmented reality system is below one data point per pixel, if an operator pans the graphics that are composited with the video it could potentially oscillate between pixel one and pixel two, and the rendering computer will not be able to resolve the precise angle. The worst case scenario could involve the rendering computer becoming so confused as to where the graphic should be composited that it would show either a fuzzy graphic image when the camera pans or tilts, or a the graphic floating slightly over the composited video.
For 4K augmented reality applications, double the number of data points would be required, equating to 2.35 million through 360 degrees of rotation of the pan and tilt axis. This would appear to be a higher resolution than the Vinten heads appear to output, however the encoder technology in the i-series can deliver up to 12m counts per 360 degrees. As the heads and the VRI interface box architecture are designed to enable this upgrade with no modification, it can be exchanged by an engineer in a matter of minutes.
This ultra-high precision measurement of camera viewing vector is vital in augmented reality 4K productions to ensure the graphics are perfectly located, move smoothly and therefore look part of the real scene to viewers.
In terms of the zoom and focus data, it is unlikely that lens manufacturers will have to change the encoding performance when designing new 4K lenses. An HD lens typically has around 55,000 encoder counts between telephoto and full wide angle, the two stops on the zoom. Shooting in 4K with a 4K lens, sensor size being equal, it is still the same focal length difference between the largest and shortest focal length of that lens. Therefore the 55,000 encoder counts should be adequate enough for the rendering engine to use a mathematic model to determine what the field of view is, and the magnitude of any distortion in the image to a degree that the compositing looks real.
As 4K requires higher bandwidth video transport systems, it would be reasonable to think that the camera tracking information for 4K would follow suit requiring a four-fold increase in data flow between the VRI interface box and rendering computer, potentially overloading the bandwidth of the serial connection between them. Not so with the Vinten system, which was architected so that all of the processing and scaling can be completed in the VRI box and the data packet transmitted has the same format and size, regardless of the resolution of the image and therefore the encoders.
An increase in the frame rates (which could be beneficial to motion quality in 4K) would result in a higher volume of data transmitted between the VRI and rendering engine. However, in the VRI system there is sufficient processing power and bandwidth to handle the data required at up to double current frame rates.
The excitement and testing activities in recent months have indicated that, augmented and virtual applications in 4K are becoming a reality for increasing numbers of broadcasters and independent production/OB companies and that any future 4K productions will rely not only on the latest camera equipment, lenses and transmission, but the highest performing, precise capabilities of the camera support equipment.

The article is also available in BFV online.
(IT/JP)