by Michaël Tits, Mohamed Boukhebouze and Christophe Ponsard (CETIC)
Recent advances in deep neural networks have enabled great improvements in image restoration, a long-standing problem in image processing. Our research centre has experimented with cutting edge algorithms to address denoising, moire removal, colourisation and super resolution, to restore key images representing the photographic heritage of some of Belgium’s top athletes.
The research field of image restoration is benefiting from recent major improvements in artificial intelligence algorithms, particularly deep learning, a family of black-box algorithms using deep neural networks allowing a machine to learn relevant relationships and abstractions, provided enough examples are available.
Such algorithms can improve poor quality images by learning how to “invent” detail from the initial image, for instance by retrieving colours of a monochrome image, or filling in missing pixels realistically. These techniques are still being developed, but they show great promise in different contexts that require improved image quality, such as video surveillance, medical and satellite imagery and culture (history and art), which was our experimentation field.
As part of the DigiMIR project [L1], CETIC (ICT R&D centre) and Numediart (multimedia multimedia research institute) have collaborated to select, test and compare recent techniques in this field, with the aim of developing a generic and innovative image restoration tool going beyond prototypes proposed in the scientific literature. Tools to date have generally been confined to a very specific algorithm (e.g. [1]) or prototype (e.g. [L2]) and are not integrated in a consistent processing chain. A more generic approach can have advantages, for example noise reduction can improve the texture of different surfaces, such as that of a lawn, which will also improve the results of a colourisation algorithm. The more comprehensible the image, the better the algorithm will work! Conversely, an image that is too noisy, or too smoothed by a poorly executed “denoising” algorithm, will prevent the lawn texture from being recognised, resulting in greyish colours.
Our work focused on the cascading of four types of techniques to address the following defects: unstructured noise (grain/ white noise), structured noise (stripes), monochromatic character and low resolution. For each aspect, a comparative analysis was carried out to identify the most relevant algorithms. We then developed a pipeline allowing an end-to-end restoration of the images and making it easy to cascade techniques, and to couple them with more conventional image processing. This active project is available as open source [L3].
Figure 1: Example of stripe removal (Paul Goffaux, Belgan boxing champion in 1942).
A concrete application case was carried out in collaboration with a Belgian cultural organisation and Jean Vandendries, the author of a book detailing the careers of athletes from the past century [2]. The book was facing the problem of printer rejection of about half of the images (i.e., 50) due to poor quality. The images contained a wide variety of defects, including pronounced grains, several scratches and streaks due to the aging of a physical medium, numerous monochrome images and very low resolution. As an example, Figure 1 shows different results of stripe removal from the original image on the left. This study was rewarding because we could successfully restore historical material. In addition, this challenging material was a strong driver for building our toolchain and considering many practical issues, such as avoiding the injection of new artefact and automatically assessing the quality of the restoration.
Links:
[L1] https://www.cetic.be/DigiMIR-2789
[L2] https://github.com/jantic/DeOldify
[L3] http://www.mickaeltits.be/open-image-restoration
References:
[1] J. Guan, R. Lai and A. Xiong: "Wavelet Deep Neural Network for Stripe Noise Removal," in IEEE Access, vol. 7, pp. 44544-44554, 2019.
[2] J. Vandendries: “Les Étoiles sportives nivelloises”, 2019.
Please contact:
Christophe Ponsard, CETIC, Belgium
+32 472 56 90 99
