Compression Considerations
Posted on Fri, Feb 11, 2011
By Kevin Klemmer, CISSP PSP for Pivot3, Gold Sponsor at the ASG Security Summit & Expo
The IP surveillance market has boomed over the past decade with dozens, if not hundreds, of new players emerging and traditional CCTV manufacturers expanding into networked technologies. With the avalanche of new companies and technologies cascading over the industry, there is no wonder there is confusion surrounding the proper configuration of networked-based security technologies, and what is necessary for a successful and reliable deployment.
This blog post is focused on the technical factors involved. A good integrator will provide a business and risk assessment prior to recommending a technology approach. That assessment will underline the performance considerations outlined in this post.
With that said, resolution, compression, frame rate, retention time and motion in the field-of-view are some of the main factors to consider when configuring an IP-based system. While most of these factors are pretty straightforward, compression is the most confusing and daunting to address. What is MJPEG, MPEG4, H.264 or J2K? Lossy or lossless?
Defining Compression
Video compression is very complex but described simply, it removes redundant data from the field-of-view to make video file sizes as small as possible to enable users to store as much data as possible. Furthermore, video compression technologies save time and money associated with storage and servers.
Each compression scheme is an algorithm used to accomplish data reduction. Each algorithm delivers a different amount of stored data for the exact same raw video. There are two primary types of compression: Lossless and lossy compression.
Determine Video Needs
How can you choose which compression is best for a particular application? Two factors come to mind and both ultimately come down to costs. The first cost factor is insurance and is difficult to determine. This insurance refers to the data recorded and if it can be used for evidentiary purposes. If a user requires the original video data for legal reasons then lossless compression is what is needed. If not, lossy compression is an ideal fit.
Lossless data compression is an algorithm that enables the exact original data to be reconstructed from the compressed data. An example of lossless compression is JPEG2000, although there are some versions of JPEG2000 that are lossy compression standards as well.
Lossy data compression actually discards some of the data during encoding to deliver a much smaller file size. This is very useful from a storage perspective, but can be an issue when recreating the video stream. Most of the commonly used compression algorithms are lossy and do a fine job of recreating the data for general purpose surveillance needs. Examples include MPEG4 and H.264.
The second cost factor is the storage and servers required. There is a substantial trade off in terms of storage costs associated with different compression types. Lossless compression requires more storage than lossy compression. A VGA (640 x 480 pixels) video stream for a single camera at 10 frames per second for a single day of retention, uncompressed, requires more than 700GB of storage. The same configuration with MJPEG requires approximately 75GB; MPEG4 equates to 10GB; and H.264 needs 4GB.
Cost Effectiveness
The cost per gigabyte has decreased significantly over time and can be less than $1 per GB for the average end user. This is due to innovations in software-based RAID protection and use of commodity hardware to accomplish what previously required proprietary hardware and fixed RAID controllers. The flexibility offered to end users with next generation storage servers is significant, and is helping to provide exceptional data and software protection at a fraction of the price.
For the server side, there are two other factors to consider when deploying lossy compression. They are scene motion and object speed. Lossy compression will certainly save money on storage, but may require significantly more processing power (more processors, more money) to perform the task. The more motion in the field-of-view and the speed of the objects, the more processing is required when using a lossy compression algorithm. The algorithm is basically estimating the location of those objects moving within the field-of-view and their speed when compressing (encoding). It then must recreate the stream when accessing data (decoding). In areas of significant changes from static images to high motion/high speed, processing requirements become overwhelming. There are also visual effects that occur when systems are not configured to take higher processing needs into account including ghost images, artifacts, storage spikes, bandwidth overconsumption and video corruption.
It is interesting to discover that each camera manufacturer performs differently depending on the type of video management software and the compression algorithm. The differences can be vast. In order to make certain there is an accurate server and storage configuration, you need to verify these details with not only the video management software provider, but the camera manufacturer. Storage server suppliers are also a great resource as theses companies test and benchmark the various open architecture video management systems available to determine the most effective hardware platform. These vendors typically have more expertise in data payload to disk delivery and processing requirements.
With all of the options available for compression, with price as the primary consideration, the lossy schemes are most common. The surveillance environment helps determine the best compression fit for the needs of the user. Lossless compression provides a higher quality image and better evidentiary data than lossy compression and should be considered in appropriate environments.
Most of the Article contents were originally posted in Security Dealer & Integrator Magazine
ASG Publisher Note (Background Notes) Definition: Lossless and lossy compression are terms that describe whether or not, in the compression of a file, all original data can be recovered when the file is uncompressed. With lossless compression, every single bit of data that was originally in the file remains after the file is uncompressed. All of the information is completely restored. This is generally the technique of choice for text or spreadsheet files, where losing words or financial data could pose a problem. The Graphics Interchange File (GIF) is an image format used on the Web that provides lossless compression.
On the other hand, lossy compression reduces a file by permanently eliminating certain information, especially redundant information. When the file is uncompressed, only a part of the original information is still there (although the user may not notice it). Lossy compression is generally used for video and sound, where a certain amount of information loss will not be detected by most users. The JPEG image file, commonly used for photographs and other complex still images on the Web, is an image that has lossy compression. Using JPEG compression, the creator can decide how much loss to introduce and make a trade-off between file size and image quality.