Artificial Intelligence in Rhinodiagnost

Problem definition: Segmentation (generation of content-related regions, here: determination of nasal cavities and paranasal sinuses from CT images) using a “Convolutional Neural Network” (CNN) with Google TensorFlow

In order to generate a 3D model from image data, the pixels that should represent the 3D model are to be selected. But CT pictures have image noise (random variation of brightness or colour information in images) since the radio source is operated with the least possible intensity for the benefit of the patient, and some structures in the CT images are so fine that they are not recorded correctly. Therefore, the images for the production of the 3D model have to be reworked. In the present experiment, the manual post processing process has been replaced by Machine Learning and Convolutional Neural Networks. As a training set, 160 images (axial sections) of a person were selected and fed into the algorithm.

The image series in Figure 1 shows an example of five CT images, which served as input data.

Figure 1 Test data (5 CT cross-sectional images unknown to CNN so far)


Through segmentation content-related regions are created within the image. The image series in Figure 2 shows the probabilities calculated by the CNN as to whether a pixel is part of segmentation. The pictures are shown with continuous colour values.

Figure 2 CNN predicted colour values


Subsequently, the CNN generated a binary segmentation, reduction to the colour values white and black. (“binary”: white = in the cavity, black = outside of a nasal cavity). Figure 3 depicts this segmentation.

Figure 3 CNN generated segmentation (binary)


Figure 4 shows the manually created segmentations, with only the white and black colour values used.

Figure 4 Manual segmentation (binary)


To simulate manual segmentation (the transition from the second to the third image row) in a good way the reduction to the two colour values white and black had to be improved and optimized. In another experiment, the parameters of the neural network (layer depth and learning rate) were changed, resulting in a significantly improved outcome (Figure 5).

Figure 5 CNN created segmentation with changed parameters


The essential part of this first attempt was to check whether the chosen method (CNN) leads to  useful results, and the aim was to detect automatically whether a point of the CT scan (DICOM image) is located inside or outside of a nasal (or paranasal) sinus. Despite that only a small data set of images of one person was used for the training set, the segmentation turned out to be similar or even improved compared to the manual editing (see and compare Figures 4 and 5).

As a result, the CNN is now further enhanced by including larger amounts of data (approx. 1500 people, approx. 250 images each) and optimizing the parameters. Finally, the development of the neural network should lead to a classification indicating the pathological condition of a nasal or paranasal sinus (location of the diseased tissue, etc.) and should finally allow for the automatic generation of  the corresponding 3D representation (STL file). This will build the basis for the further preparation and investigation of a 3D-modeled region (RoI, Region of Interest) with 3D printing, CFD simulation, virtual surgery, etc.

Training set: 160 images / axial section, test set: 10 images, 5 of which are shown in the pictures.
Test machine: Intel Core i7-7820X, 3.60 GHz; 64GB memory; 2 GPUs: GeForce GTX 1080 Ti.

Authors: AIT Angewandte Informationstechnik Forschungsgesellschaft, Matthias Lehner, Odo Benda