Evaluation of a computer-aided skin cancer diagnosis system for conventional digital photography with manual segmentation

We evaluate a computer-aided diagnosis (CADx) system developed for both melanocytic and non-melanocytic skin lesions by using conventional digital photographs with lesion boundaries manually marked by a dermatologist. Clinical images of skin lesions taken by conventional digital cameras can capture useful information such as shape, color, and texture for diagnosing skin cancer. However, shape/border features are difficult to analyze automatically because skin surface reflections may change skin color and make segmentation a challenging task. In this study, two non-medical users manually mark the boundaries of a dataset of 769 (174 malignant, 595 benign) conventional photographs of melanocytic and non-melanocytic skin lesions. A state-of-the-art software system for segmenting color images, JSEG, is also tested on the same dataset. Their results are compared to a dermatologist's markings, which are used as the gold standard in this study. The human users' markings are relatively close to the gold standard and achieve an overlapping rate of 70.4% (+/- 15.3%, std) and 74.5% (+/- 14.7%, std). Compared to human users, JSEG only succeeds in segmenting 636 (82.7%) out of 769 lesions and achieves an overlapping rate of 72.4% (+/-20.4%) for these 636 lesions. The estimated area under the receiver operating characteristic curve (AUC) of the CADx by using lesion boundary markings of users 1, 2, and JSEG are 0.915, 0.940, and 0.857 respectively. Our preliminary results indicate that manual segmentation can be repeated relatively consistent compared to automatic segmentation.