Original Article|Articles in Press

Clinicians’ perceptions of oral leukoplakia: a pitfall for image annotation in Supervised Learning

Published:March 06, 2023DOI:



      The present study aims to quantify clinicians’ perceptions of oral potentially malignant disorders (OPMD) when evaluating, classifying, and manually annotating clinical images, as well as to understand the source of interobserver variability when assessing these lesions. The hypothesis was that different interpretations can affect the quality of the annotations that are used to train a Supervised Learning (SL) model.

      Study design

      46 clinical images from 37 patients were reviewed, classified, and manually annotated at the pixel level by three labelers. We compared the inter-examiner assessment based on clinical criteria through the κ statistics (Fleiss's kappa). The segmentations were also compared using the mean pixel-wise intersection over union (IoU).


      The interobserver agreement for homogeneous/non-homogeneous criteria was substantial (κ=63, 95% confidence interval (CI): 0.47 to 0.80). For the subclassification of non-homogeneous lesions, the interobserver agreement was moderate (κ=43, 95% CI: 0.34 to 0.53) (p <0.001). The mean IoU of 0.53 (±0.22) was considered low.


      The subjective clinical assessment (based on human visual observation, variable criteria that has suffered adjustments over the years, different educational backgrounds, and personal experience) may explain the source of interobserver discordance for classification and annotation of OPMD. Therefore, there is a strong probability of transferring the subjectivity of human analysis to artificial intelligence (AI) models. The use of large datasets and segmentation based on the union of all labelers’ annotations holds the potential to overcome this limitation.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'