Frame 1 of FRANK
Both the entertainment and scientific community have some need for algorithms that automatically remove degradation from image sequences. Noise and blur are perhaps the most familiar problems (to the layman) with real world sequences. Examples of noisy sources include Scanning Electron/Tunnelling Microscopes, repeated playback from Video Tape, and of course film grain. Motion blur is perhaps the less familiar blurring artefact on television, film due to high speed events coupled with relatively long shutter times. It is difficult to observe this problem in a sequence played at 20-25 frames/sec, but is clearly observable on stills taken from the degraded sequence. Focus blur due to unfocused lenses etc. is a more familiar blurring artefact. Missing data problems such as dropouts, Dirt and Sparkle are very common affecting almost all archived stock of video and film. Line scratches which persist from frame to frame also fall into this category. Line jitter due to loss of horizontal synchronization from a noisy video source is a particular video artefact which does not occur often. It is however, a curious problem which gives rise to some interesting algorithms.
With the iminent widespread use of digital video broadcasting, archivists perceive that there will be a greater demand for their stock. Automatic image sequence restoration/reconstruction is seen to be a very important requirement for supplying this demand. Already digital manipulation tools such as Quantel's Harry are in regular use by television broadcasters for limited retouching work. A very good real time restoration tool is also available and in regular use, produced by Digital Vision. In an attempt to widen the range of automatic digital video reconstruction tools, the European sponsored project AURORA ( Automatic Restoration of Original Film and Video Archives) has recently kicked off as a conglomerate of companies and universities. The conglomerate includes the Signal Processing Group here and at Delft University ( P. B. M. Van Roosmalen, Jan Biemond, R. Lagendijk), the Institue National L'audiovisuel (France), SGT (France), the BBC, Snell and Wilcox (U.K.), RTP (Film and video archive in Portugal ) and the Digital Media Institute (Finland).
Missing data in archived film manifests as Dirt and Sparkle which is caused by the abrasion of the film as it passes through the projection mechanism (sparkle), or occlusion of the film by particles caught in the machine (dirt). The result is small patches of bright and dark areas which are uncorrelated with the surrounding image. This is also a problem in high speed photography which is employed for combustion research for instance. In digital video missing data occurs in the form of digital dropouts which may cause the loss of one or more video lines. In all cases the true image data has been obliterated and the task is to automatically detect then interpolate this missing data. The Signal Processing Lab here at Cambridge has been involved in this area for both audio and images for at least a decade. Many techniques have been deployed for the design of algorithms both heuristic and model based. Recent work has concentrated on stochastic approaches (Markov Chain Monte Carlo techniques) for the joint solution of all the parameters : model coefficients, motion and the missing data.
Frame 1 of FRANK
Frame 2 of FRANK with large blotches boxed
Frame 3 of FRANK
Detection on Frame 2
Interpolation using MAP estimate
without Motion correction.
Interpolation using MAP estimate
with Motion Correction.
Interpolation using Median
filter.
Sampled Interpolation