A QUALITY MEASURE FOR COMPRESSED IMAGE SEQUENCES BASED ON AN EYEMOVEMENT COMPENSATED SPATIO

A Q UALITY M EASURE FOR C OMPRESSED I MAGE S EQUENCESB ASED ON AN E YE -M OVEMENTC OMPENSATED S PATIO -T EMPORAL M ODEL *S TEFAN J.P. W ESTEN , R EGINALD L. L AGENDIJK , AND J AN B IEMONDDelft University of TechnologyDepartment of Electrical Engineering, Information Theory GroupP.O. Box 5031, 2600 GA Delft, The Netherlands E-mail: {stefan,inald,biemond}@it.et.tudelft.nl* This work was supported in part by Philips Netherlands N.V. under contract RWC-061-PS-96014-ps (POEM project).A BSTRACTWe propose a vision model for predicting the perceptibility of coding errors in digital image sequences.Eye movements have a significant effect on distortion visibility, and hence they form an important part of the model. A modified motion estimator is used to predict the eye movements of a viewer when watching an image sequence. The spatio-temporal frequency sensitivity and masking effects are included in the model. The model is based on a decomposition of the signal into frequency and orientation bands to allow accurate modeling of spatial masking by a contrast transducer. We describe how the proposed quality measure can be used to compress image sequences using a constrained MPEG-2 encoder.1. I NTRODUCTIONAlthough the Mean Squared Error (MSE) is a widely used distortion measure for the optimization of image compression, it is not a very good measure for the perceived image quality. More appropriate quality measures incorporate characteristics of the visual system.Most models that have been proposed for predicting the visibility of compression errors in digital image sequences are based on spatial properties of the human visual system [1-3]. Of the few models that incorporate both spatial and temporal characteristics of the human visual system [4-7],only [7] recognizes the importance of the capability of the human eyes to carry out almost perfect compensation of object motion in image sequences. This compensation process is known as “smooth pursuit eye movements”(SPEM), and has as consequence that blur and noise in moving objects in image sequences have the same visibility as in non-moving objects. The existence of SPEM must therefore be incorporate into models that exploit differences in visual sensitivity for different spatio-temporal frequencies.In Section 2 of this paper we propose a quality measure for compressed image sequences based on SPEM-compensated spatio-temporal sensitivities and masking properties of the human visual system. To this end, in Section 3 we propose a modified motion estimation algorithm for the estimation of the SPEM [8] such that the retinal image sequence, obtained after SPEM-compensation of the displayed image sequences, can be realistically determined. In Section 4 we discuss how the proposed quality measure can be used to optimize the compression of image sequences using a constrained MPEG-2 encoder (fixed rate per frame, I frames only).2. M ODEL S TRUCTUREThe structure of our spatio-temporal quality model is shown in Figure 1. The model consists of the following components:1. the calculation of the retinal image sequence by SPEM estimation, followed by SPEM compensation,2. the spatio-temporal contrast sensitivity filter to account for varying sensitivities as a function of spatial and temporal frequencies,3. the spatial frequency decomposition, so that frequencyFigure 1: General model structure.dependent in-band spatial masking, modeled by a contrast transducer, can be accounted for,4. the summation of the differences between thenormalized contrasts calculated from the original and the compressed image sequence.There are various ways in which the varying sensitivities for different spatio-temporal frequencies can be incorporated into a quality model. In our earlier work that proposed a spatial quality model [3], the varying spatial sensitivities were incorporated by frequency band dependent contrast sensitivity multipliers. In this model the image is first spatially decomposed into 21 directional frequency bands, followed by the frequency band dependent contrast sensitivity multiplier and the frequency band dependent masking functions.The straightforward extension of the spatial approach to frequency dependent temporal sensitivities is typically done through a two-channel temporal decomposition [6]. Such decomposition, although being supported by some psycho-visual evidence, does not allow to model the varying temporal sensitivities accurately enough. Note that this is in contrast to the spatial frequency decomposition, where far more bands are used. For the above reason we have chosen to separate the modeling of the spatio-temporal frequency dependent contrast sensitivity (namely by the above mentioned step 2), and the spatio-temporal in-band masking (by the above mentioned step 3).The spatio-temporal contrast sensitivity filter is based on the visual filter by Lukas and Budrikis [4]. It consists of a spatio-temporal excitation response E(x,y,t) and inhibition response I(x,y,t), nonlinearly combined as follows:C x y t A E x y t k I x y t(,,).(,,)(,,)=+(1)Here C(x,y,t) is the contrast at the spatial location (x,y) in frame t, and A and k are model adaptation constants that depend on the viewing conditions. The advantage of (1) is that the excitation and inhibition responses are separable in a spatial part (Gaussian filter) and a temporal part (second order exponential filter). The overall response is, however, nonseparable in spatial and temporal components as prescribed by many psycho-visual experiments, e.g., [10].The contrast calculated by (1) is spatially decomposed into various orientations and frequencies. The filter is a variation [3] on the steerable pyramid by Simoncelli et al.[9]. The decomposition gives a radial bandwidth of 1 octave and an orientation bandwidth of 45 degrees. Unlike critically sampled decompositions the steerable pyramid is completely shift-invariant, which is required in a quality model because shifting the image should not affect the calculated image quality. The spatial decomposition per frame of the contrast C(x,y,t) results in 1 low-pass and 20band-pass images.For each frequency and orientation band, in-band masking is incorporated by a contrast transducer similar to Lubin [2]. For the oriented frequency band (k,l) at time t, the masked contrast C out(x,y,t,k,l) at coordinates (x,y) is given by:C x y t k lC x y t k lC x y t k lout(,,,,).(,,,,)(,,,,)=+αγβ2(2)Here αβγ,, are model constants that are fitted to experimentally obtained curves. Finally, the masked contrast is computed for the original image sequence and the compressed sequence, and the difference is taken. The differences are combined over the various oriented frequency bands such that a spatially varying distortion measure is obtained for each frame. Though the process of error summation is largely an open question and involves not only signal processing but also cognitive processes, we have chosen a simple quadratic measure:()M x y t C x y t k l C x y t k lk l(,,)(,,,,)(,,,,),=−∑orig compr2 (3)The overall result is a per frame mask M(x,y,t) that indicates the relative spatio-temporal distortion visibility at each spatial position.3. SPEM E STIMATION AND C OMPENSATIONWhen an eye movement is made, the image that is projected onto the retina (“retinal image”) makes a translation movement. This movement alters the spatio-temporal frequency spectrum of the retinal image, and hence the perception of compression artifacts. Though it is difficult to predict which SPEMs will be made by viewers, the most critical movements are the ones due to tracking of moving objects. We therefore carry out the calculation of the spatio-temporal sensitivities in a SPEM-compensated way, assuming that each moving object can be tracked perfectly. Although the spatio-temporal contrast sensitivity filter (1) could indeed be operated along the estimated object’s motion trajectory, we have chosen to first locally compensate the frames for the estimated SPEMs and then operate (1) on the compensated frames.The estimation and compensation of SPEM is somewhat similar to global motion estimation and compensation, since eye movements act as a global operation on the image sequences. The compensation operation can be fairly well modeled by a translational model. However, whereas for global motion estimation we need to estimate a single translation vector, in SPEM estimation the translation vector depends on which moving part of the sequence the viewer decides to concentrateupon. It is therefore necessary in SPEM estimation and compensation to track all dominant motion vectors in the sequence, since the viewer may decide to concentrate, and consequently compensate, for each of these.Block-based estimation algorithms are in principle suitable for the estimation of SPEMs, particularly since we are interested in finding only translational dominant vectors that model the smooth pursuit eye movements. We propose to use a modified hierarchical motion estimator that consists of 2 phases. In the first phase, fairly large blocks (e.g., 64x64 pixels for 4CIF frames) are used to determine candidate motion vectors (full search using large blocks). In the second phase, at smaller block size levels no new motion vectors are computed, but only the earlier computed candidate vectors in a close neighborhood of the block under consideration are tested for their suitability.Since we avoid the computation of new motion vectors and essentially only locally propagate vectors from a pool of candidates, the resulting vector field is very smooth.Though for motion compensated compression such a vector field would be unsuitable, the field does show the gross effect of SPEM when tracking a local object. The resulting vectors are used to calculate an SPEM compensated retinal image sequence, from which the mask M (x ,y ,t ) is then determined.4. O PTIMAL C OMPRESSION OF MPEG-IThe objective of bit rate control and adaptive quantization strategies in, for instance MPEG, is to minimize a distortion measure given a certain bit ing the mask M (x ,y ,t ), and assuming that we wish to optimize only the local quantization Q (i ,t ) for (macro)block i in frame t , this optimization problem can be formulated as:min ((,,))(,)(,)Q i t i tf M x y t R i t given constant ,∀∑=(4)where R (i ,t ) is the number of bits used to code (macro)block i in frame t . An important choice in (4) is the function f (.) that operates on the mask. Two different properties of f (.) should be distinguished, namely· the type of function, such as sum-of-squares, sum-of-absolutes, and maximum,· the domain over which f (.) is calculated. The most elaborate form in this respect is to let f (.) operate on the entire image sequence, yielding for instance for the sum-of-squares:f M x y t M x y t x y t ((,,))(,,)(,)=∈∈∑∑2all pixelsall frames (5)Various papers have addressed optimization problems of the form (4) in the context of MPEG compression [11].In order not to let the computational complexity dominate the efforts in using and evaluating the proposed spatio-temporal quality measure, we impose the following constraints:· the coder uses I frames only, so we do not have to deal with propagation of compression artifacts. Of course,the temporal component in our model will still make the quantization of successive I frames dependent,· the bit budget per frame is constant,· the optimization process is sequential such that future frames cannot influence the quantization of the current frame,· the optimization process is conditioned on the previously compressed frames.With the above constraints, Eqs. (4) and (5) can be reformulated as follows for the compression of frame k :min(,,)()(,)(,)(,)Q i k x y iM x y k R i k k t k21∈∑∑=<all pixelsconstant for each (2) all compressed frames given (6)Essentially (6) is a fixed bit rate adaptive quantization problem per frame, knowing the previously compressed frames and without knowing the future frames. Note that the calculation of M (x ,y ,k ) using proposed spatio-temporal model does not need information from future frames due to the recursive temporal filter used in (1).The most elegant way of solving (6) would be through the linearization of the spatio-temporal model, followed by the calculation of a relative distortion measure per DCTFigure 2: Iterative optimization procedure.coefficient and per (macro) block, similar to our approach in [3]. Since due to the high nonlinearity of model linearization seems questionable, we have followed the computationally more expensive approach of iterative compression. The objective of the iteration is to find the setting of the quantization parameter MQUANT(i) in MPEG such that the resulting bit rate is within the bit budget and the error criterion is minimized. The structure of the iterations is shown in Figure 2.5. E XPERIMENTSAs a demonstration of our model, we coded the CAR_CIF sequence with MPEG-1 at 1.15 Mb/s. The original sequence was 352 by 288 pixels and was progressively scanned. The original and the decoded sequence were input to the model. Figure 3(a) shows a part of a non-optimized (MPEG) compressed frame. The corresponding function C(x,y,t) for this frame is shown in (b), and the resulting mask M(x,y,t) in (c). Clearly the visibility strongly differs over the frame. The optimization of f(M(x,y,t)) should result in a much more even distributedmask values.(a)(b)(c)Figure 3: Example of a non-optimized maskM(x,y,t).6. D ISCUSSIONIn this paper we have proposed a spatio-temporal quality measure. The measure distinguishes itself from most other quality measures in that it takes into account temporal contrast sensitivity and smooth pursuit eye movements. As the effects of temporal masking are of minor influence for compression (only after scene cuts) and are therefore not included in the model, we believe that the main effect of the proposed model in compression algorithms will be to avoid the introduction of artifacts with (SPEM-compensated) spatio-temporal components for which the visual sensitivity is high. We therefore come to the conclusion that rather the prevention of perceptually annoying temporal artifacts should be expected when using the proposed model than large bit rate reductions.R EFERENCES[1] Zetzsche C., and G. Hauske, “Multiple Channel Model forthe Prediction of Subjective Image Quality”, SPIE Conf. on Human Vision, Visual Proc., and Dig. Display, Vol. 1077, pp. 209-216, 1989.[2] Watson, A.B., Digital Images and Human Vision, MITPress, Cambridge, Massachusetts, 1993.[3] Westen, S.J.P., R.L. Lagendijk, J. Biemond, “Perceptualimage quality based on a multiple channel HVS model”, IEEE Int. Conf. on Acoustics, Speech and Signal Processing 1995, pp. 2351-2354, 1995.[4] Lukas, F.X.J., and Z.L. Budrikis, “Picture QualityPrediction Based on a Visual Model”, IEEE Transactions on Communications, vol. 30, no. 7, pp. 1679-1692, 1982.[5] Girod, B., Ein Modell der menschlichen visuellenWahrnehmung zur Irrelevanzreduktion von Fernsehlumi-nanzsignalen, VDI Verlag, Düsseldorf, 1988.[6] van den Branden Lambrecht, C., “Color Moving PicturesQuality Metric”, IEEE Int. Conf. on Image Processing 1996, pp. 877-880, Lausanne, Switzerland, 1996.[7] Girod, B., “Eye Movements and Coding of VideoSequences”, SPIE Visual Communications and Image Processing 1988, SPIE 1001.[8] Westen, S.J.P., R.L. Lagendijk and J. Biemond, “An EyeMovement Compensated Spatio-temporal Model for Predicting Distortion Visibility in Digital Image Sequences”, Proc. of the 18 Symposium on Information Theory in the Benelux, Veldhoven, 1997.[9] Simoncelli, E.P., W.T. Freeman, E.H. Adelson, and D.J.Heeger, “Shiftable Multiscale Transforms”, IEEE Transactions on Information Theory, vol. 38, no. 2, 1992.[10] Robson, J.G., “Spatial and Temporal Contrast SensitivityFunctions of the Visual System”, Journal of the Optical Society of America, vol. 56, pp. 1141-1142, 1966.[11] K. Ramchanadran, A. Ortega and M. Vetterli, “BitAllocation for Dependent Quantization with Applications to Multiresolution and MPEG Video Coders”, IEEE Trans.On Image Processing, vol. 3, no. 5, 1994, pp. 533-545.。