by Alessandro Danielis, Daniela Giorgi and Sara Colantonio (ISTI-CNR)
We present a lip segmentation method based on simulated Lambertian shadings. The input consists of hyper-spectral images generated by a prototype for medical applications.
Lip contour segmentation is fundamental for many human-computer interaction applications, including lip reading, facial expression recognition, and fatigue syndrome diagnosis. Lip segmentation techniques suffer from similar drawbacks to those of face detection, including subject variability and adverse lighting conditions. Most techniques employ lip colour as the main feature, after applying a colour transform to enhance the contrast between the lips and the surrounding skin. However, different imaging systems may produce different colour values, even for the same subject and under the same lighting conditions.
We present a lip segmentation method for hyper-spectral images (HSI), namely blood concentration face maps generated from HSI data by using narrow spectral bands. The advantage is that in blood concentration maps, the lip spatial pattern is more enhanced than in standard colorimetric maps. To achieve this, we replace the usual colour space transformation with the simulation of Lambertian shading in different directions; Lambertian shaded images support more accurate and robust segmentations than raw images, and are less sensitive to acquisition conditions than colorimetric transforms. The Lambertian-shaded images then undergo morphological reconstruction to localise the lip region, and the lip boundary is reconstructed using Fourier descriptors on candidate segments. Figure 1 shows the flowchart of the method.
Figure 1: The pipeline of the lip segmentation method.
The acquisition system for generating the HSI data is part of a more complex multisensory platform for medical applications, namely, a smart mirror for the detection of facial signs of cardio-metabolic risk [1]. The platform, named Wize Mirror, has been developed within the European project SEMEOTICONS [L1]. While a subject sits in front of a mirror – either as a part of the daily morning routine at home, or in the gym or the pharmacy – the Wize Mirror detects signs of cardio-metabolic risk on the face, and advises how to reduce the risk through behavioural change. The signs include both physical and emotional traits. In this context, the lip segmentation algorithm we present was intended to support emotional analysis. However, Lambertian shading could be applied as a contrast enhancement step in other applications involving segmentation.
Hyper-spectral image generation: blood concentration maps
Colour is given by the scattering and absorption of light within the layers of skin. The blood supply in sub-cutaneous districts affects the optical properties of skin. Thus, these properties can be used to derive blood concentration maps. The acquisition system consists of five compact monochrome Flea3 3.2 MP USB 3.0 CMOS cameras (Point Grey) with band pass filters at selected wavelengths, and two computer controlled LED light sources (white light and ultraviolet). The blood concentration maps are calculated based on the average spectral intensity in four different wavelength bands, selected to maximise sensitivity to blood amount and saturation, while compensating for tissue scattering effects and melanin influence [2]. Figure 2(left) shows a face image along with its blood concentration map. The false-colour surface representations of the lip region in Figure 2(right) shows how the lip pattern is better discriminated in the blood concentration map than in standard colorimetric maps.
Figure 2: Left: a colour image of a face and the corresponding blood concentration map. Right: comparison of the blood concentration map with colorimetric maps (false colour representation).
Contrast enhancement through simulated Lambertian shading
The main idea is to enhance the lip spatial pattern in concentration maps by exploiting texture changes generated by different simulated Lambertian illuminations. Viewing the map I(x,y) as a topographic relief, the shading LS(x,y) can be obtained by simulating a Lambertian light source, i.e., considering the cosine of the angle between the relief normal at each point and the light direction. Different shadings can be computed by specifying different elevation and azimuth coordinates for the light source, and weights for controlling the results. The shaded image J(x,y) is simply the result of matrix multiplication: J(x,y) = I(x,y)·LS(x,y). We generate a set of shadings, which are then processed to segment the lip. On a Lambertian shaded concentration map, lips appear as compact and bright areas. Figure 3 illustrates Lambertian shading along different directions, together with the corresponding lip regions, with false colours.
Figure 3: Lambertian shadings corresponding to different coordinates of the light source (elevation and azimuth) and different weights (gradient scales).
Lip region detection and boundary extraction. To discriminate the lip region from the surrounding skin areas, we perform a series of morphological operations on the Lambertian shaded images, namely geodesic erosions followed by geometric dilations with a flat ball-shaped structuring element. The sequence is optimized by reducing the dimension of the structuring element for localising bright regions (peaks, in topographic terms), which satisfy geometric constraints (orientation, connectivity, solidity, eccentricity). Once the lip region is segmented, the lip candidate is selected by using shape descriptors on Fourier-based modelled lip boundaries.
Figure 4 shows some results of our automatic segmentation technique. Experiments on both a public hyper-spectral dataset and an in-house developed dataset of real data from volunteers showed robustness to noise and moustaches, and significant improvements in the performance with respect to competitors [3].
Figure 4: Lip contours segmented on concentration maps, and superimposed on the corresponding colour images.
Link:
[L1] www.semeoticons.eu
References:
[1] S. Colantonio, et al.: “A smart mirror to promote a healthy lifestyle”, Biosystem Engineering, 138 (2015), 33-43
[2] M. Everloff, et al.: “Estimating skin blood saturation by selecting a subset of hyperspectral imaging data”, Proc. SPIE, vol. 9328 (2015)
[3] A. Danielis, et al.: “Lip segmentation based on Lambertian shadings and morphological operators for hyper-spectral images”, Pattern Recognition, 63 (2017), 355-370
Please contact:
Daniela Giorgi
ISTI-CNR, Italy
