Video signals must contain synchronization information to allow the video
display device to properly locate lines and frames relative to each other in
space and time. Noise in the video signal, or degradation of the storage medium
on which the signal is stored (video tape) can cause the synchronization signals
to be corrupted. This
can cause the loss of `lock' in video digitizing and
playback apparatus. The loss of line synchronization pulses will prevent the
video manipulation device from locating the actual start and end of each
line thus yielding random line displacements (line
jitter) in the observed video images. In addition, the lack of
accurate frame timing would cause a vertical rolling effect in
the digitized video. In practice, the line
Whatever the source of the degradations, the line jitter effect is disturbing to the viewer. The jitter can range from +-1 pixel to more than +-5 pixels in severe cases. The pictures UTUBE, FACE and CAR illustrate typical distortion. It is possible to employ machines called `Time Base Correctors' which process the analogue video signal and recover the line synchronization pulses by trying to remove the noise on the non-picture parts of the video signal. This chapter considers an alternative solution to the problem which relies only on image data. The data can then be `de-jittered' even if the original analogue recording is unavailable.
Possible simple solutions to the problem involve either searching for the shift between lines that maximizes the correlation between them or averaging lines. Correlation matching tends to cause a bias towards vertical lines in the estimated shifts since it has a limited `memory'. That is to say, such a method involves only two lines in the image and good matches across these lines may not reflect the overall image structure. Averaging can remove much of the perceived jitter, provided the jitter is within +-1 pel, but this is at the expense of half of the vertical resolution. It is shown to be very unsuccessful in LENNA . What is needed therefore, is a method capable of retaining longer range vertical structure in the restored image without a loss of resolution. Such a method is developed here through a model based approach for estimating the jitter displacements. Although a variety of image models exist, it is thought that a spatial linear autoregressive structure is adequate for this problem, especially since it is able to capture the notion of local image smoothness. The method introduced in this chapter to provide an estimate for the relative shifts between lines is closely related to the 3DAR pel-recursive modelling framework discussed in Chapter 2. The method presented here uses a non-homogeneous 2-D AR model in a similar way that Efstratiadis et al use a homogeneous model for the considered pixel. The chapter begins with a model description and then explains the registration algorithm.
The pictures shown below are associated directly with particular sections in Chapter 5 of the book. The raw image data is contained in .PGM or .PPM files of the same name as indicated in the captions of each image.
See Figures 5.3 and 5.5 in the book. The Matlab file LENNA.M (Matlab 4.2c) will display the actual and estimated displacements used for this example.
The matlab file UTUBE.m (Matlab 4.2c) will display the estimated displacements for these images. See Figure 5.6 in the book. In addition, there are several sequences of 22 frames taken from this sequence. The file UTUBE64.SEQ is the original sequence showing jitter and shake. UTUB64DJ.SEQ shows the dejittered sequence without removing the low frequency trend in the estimated displacements. UTUB64RM.SEQ shows the resulting sequence after removing the trend. Finally, UTUB64RM.UNS shows the result of "unshaking" the sequence using a simple matching process to register the main circular feature through all the frames. The improvement in the final sequence is substantial as the main forms of image unsteadiness have been removed. There is still some small remaining image warp since only translational shake was compensated.
|
|
See Figures 5.8 and 5.9 in the book. Most of the full PAL resolution frame is shown here in colour although only the Y field was used for dejittering. The white borders in the original image show the region used for estimating the jitter.
 Original car image CAR
|
 Dejittered Car Trend removed UNJITCAR
|
See Figure 5.10 in the book. The full 512 x 512 image is shown here.
|
|
The line de-jittering algorithm has been demonstrated to be quite effective in removing the disturbing inter-line jitter that is often observed in captured frames. The success of the method lies primarily with the relatively good modelling capacity of the non-homogeneous 2DAR process and that the jitter between lines causes only relative horizontal shifts and no other distortion. Nevertheless, the problem of de-jittering an image based on spatial image information alone is difficult. The essence of the proposed algorithm is to shift lines so that the vertical image gradient (manipulated through the AR framework) is small over the whole image: thus removing jagged vertical edges and in so doing compensating for jitter. This is only effective when the corrupting jitter is primarily random, or some high frequency corruption. There is little that a low-level image processing algorithm can do to distinguish a smooth diagonal feature from a low frequency jitter corruption, which can occur (see CAR and FACE ). It may be possible to improve this algorithm by incorporating knowledge of a generating process for the line jitter, such as a low frequency component plus white noise. Then it may become feasible for a purely image based algorithm to distinguish between true smooth diagonal features and those that have been caused by the low frequency part of the jitter generating source. Although rare, severe jitter manifesting as non-linear line transformations e.g. FACE , provides a fruitful avenue for further work. The second stage of the process requires the user to identify stationary regions in the scene in order to `lock' lines in subsequent frames into position with reference to the first dejittered frame. The location of stationary areas may be done automatically and this is a subject for further work.
Jittered Lenna JITLENN3
Dejittered Lenna. no trend removal LENN3NT
Dejittered Lenna with trend removed LENN3
Original Lenna LENNA
Dejittered Lenna using averaging LENNAVG
Dejittered Lenna using median filter LENNMED
Lenna Pyramid level 1 Edges enhanced for processing LENNL1
Lenna Pyramid level 2 Edges enhanced for processing LENNL2