TY - JOUR
T1 - Sonographers' self-reported visualization of normal postmenopausal ovaries on transvaginal ultrasound is not reliable
T2 - results of expert review of archived images from UKCTOCS
AU - Stott, W.
AU - Campbell, S.
AU - Franchini, A.
AU - Blyuss, O.
AU - Zaikin, A.
AU - Ryan, A.
AU - Jones, C.
AU - Gentry-Maharaj, A.
AU - Fletcher, G.
AU - Kalsi, J.
AU - Skates, S.
AU - Parmar, M.
AU - Amso, N.
AU - Jacobs, I.
AU - Menon, U.
PY - 2018/3/1
Y1 - 2018/3/1
N2 - Objective: In the UK Collaborative Trial of Ovarian Cancer Screening (UKCTOCS), self-reported visualization rate (VR) of the ovaries by the sonographer on annual transvaginal sonographic (TVS) examinations was a key quality control (QC) metric. The objective of this study was to assess self-reported VR using expert review of a random sample of archived images of TVS examinations from UKCTOCS, and then to develop software for measuring VR automatically. Methods: A single expert reviewed images archived from 1000 TVS examinations selected randomly from 68 931 TVS scans performed in UKCTOCS between 2008 and 2011 with ovaries reported as ‘seen and normal’. Software was developed to identify the exact images used by the sonographer to measure the ovaries. This was achieved by measuring caliper dimensions in the image and matching them to those recorded by the sonographer. A logistic regression classifier to determine visualization was trained and validated using ovarian dimensions and visualization data reported by the expert. Results: The expert reviewer confirmed visualization of both ovaries (VR-Both) in 50.2% (502/1000) of the examinations. The software identified the measurement image in 534 exams, which were split 2:1:1 providing training, validation and test data. Classifier mean accuracy on validation data was 70.9% (95% CI, 70.0–71.8%). Analysis of test data (133 exams) provided a sensitivity of 90.5% (95% CI, 80.9–95.8%) and specificity of 47.5% (95% CI, 34.5–60.8%) in detecting expert confirmed visualization of both ovaries. Conclusions: Our results suggest that, in a significant proportion of TVS annual screens, the sonographers may have mistaken other structures for normal ovaries. It is uncertain whether or not this affected the sensitivity and stage at detection of ovarian cancer in the ultrasound arm of UKCTOCS, but we conclude that QC metrics based on self-reported visualization of normal ovaries are unreliable. The classifier shows some potential for addressing this problem, though further research is needed.
AB - Objective: In the UK Collaborative Trial of Ovarian Cancer Screening (UKCTOCS), self-reported visualization rate (VR) of the ovaries by the sonographer on annual transvaginal sonographic (TVS) examinations was a key quality control (QC) metric. The objective of this study was to assess self-reported VR using expert review of a random sample of archived images of TVS examinations from UKCTOCS, and then to develop software for measuring VR automatically. Methods: A single expert reviewed images archived from 1000 TVS examinations selected randomly from 68 931 TVS scans performed in UKCTOCS between 2008 and 2011 with ovaries reported as ‘seen and normal’. Software was developed to identify the exact images used by the sonographer to measure the ovaries. This was achieved by measuring caliper dimensions in the image and matching them to those recorded by the sonographer. A logistic regression classifier to determine visualization was trained and validated using ovarian dimensions and visualization data reported by the expert. Results: The expert reviewer confirmed visualization of both ovaries (VR-Both) in 50.2% (502/1000) of the examinations. The software identified the measurement image in 534 exams, which were split 2:1:1 providing training, validation and test data. Classifier mean accuracy on validation data was 70.9% (95% CI, 70.0–71.8%). Analysis of test data (133 exams) provided a sensitivity of 90.5% (95% CI, 80.9–95.8%) and specificity of 47.5% (95% CI, 34.5–60.8%) in detecting expert confirmed visualization of both ovaries. Conclusions: Our results suggest that, in a significant proportion of TVS annual screens, the sonographers may have mistaken other structures for normal ovaries. It is uncertain whether or not this affected the sensitivity and stage at detection of ovarian cancer in the ultrasound arm of UKCTOCS, but we conclude that QC metrics based on self-reported visualization of normal ovaries are unreliable. The classifier shows some potential for addressing this problem, though further research is needed.
KW - automated image analysis
KW - expert review
KW - ovarian cancer screening
KW - quality control
KW - transvaginal sonography
KW - UKCTOCS
KW - ultrasound
KW - visualization rate
UR - http://www.scopus.com/inward/record.url?scp=85043264033&partnerID=8YFLogxK
U2 - 10.1002/uog.18836
DO - 10.1002/uog.18836
M3 - Article
C2 - 28796383
AN - SCOPUS:85043264033
SN - 0960-7692
VL - 51
SP - 401
EP - 408
JO - Ultrasound in Obstetrics & Gynecology (UOG)
JF - Ultrasound in Obstetrics & Gynecology (UOG)
IS - 3
ER -