AI Technologies in Endoscopy

# AI Technologies in Endoscopy
## IRCAD Masterclass: Flexible Endoscopy
### Thomas Ward, MD<br/>Surgical AI and Innovation Laboratory<br/>Massachusetts General Hospital
### May 4, 2021

---

# Disclosures

Research support from the Olympus Corporation.

---

# Computer Vision (CV)

## Subfield of AI
---

# Computer Vision:<br/>Convolutional Neural Networks

---

# CV for Diagnosis in Endoscopy

## Lesion identification

---

# Upper GI Neoplasia Diagnosis

90% average accuracy across 19 studies for diagnosis of upper gastrointestinal neoplasias¹

.pull-left[.footnote[¹Arribas J, et al. Gut 2020;0:1–11. [doi:10.1136/gutjnl-2020-321922](https://doi.org/10.1136/gutjnl-2020-321922)]]

---

# Barrett's Esophagus Diagnosis

.pull-left[
- 1,704 images of Barrett's esophagus (BE) from 4 centers
- Trained to
    1. Classify lesions as dysplastic versus not
    2. Segment lesion, generating a heatmap of dysplatic areas and
       indiciator for best place to biopsy
- Evaluated on two external validation sets
]

.pull-left[.footnote[¹de Groof A, et al. Gastroenterology 2020;158:915-929. [doi:10.1053/j.gastro.2019.11.030](https://doi.org/10.1053/j.gastro.2019.11.030)]]

---

# Dysplastic BE Diagnosis: Performance

CV model versus 53 endoscopics on an external dataset with only *subtle* BE

.footnote[¹de Groof A, et al. Gastroenterology 2020;158:915-929. [doi:10.1053/j.gastro.2019.11.030](https://doi.org/10.1053/j.gastro.2019.11.030)]

---

# Lower GI Diagnostics:<br/>Real-time detection

Prospective trial of 1,058 patients randomized to colonoscopy with or without real-time
polyp detection system.

![real-time](imgs/wang.jpg)

.footnote[¹Wang P, et al. Gut 2019;68:1813–1819. [doi:10.1136/gutjnl-2018-317500](https://doi.org/10.1136/gutjnl-2018-317500)]

---

# Lower GI Diagnostics:<br/>Real-time detection

.footnote[¹Wang P, et al. Gut 2019;68:1813–1819. [doi:10.1136/gutjnl-2018-317500](https://doi.org/10.1136/gutjnl-2018-317500)]

## All polyps
- With system: 498
- Without: 269

## Adenoma-detection rate
- With system: 29.1%
- Without: 20.3%

## Advanced adenomas detected
- With system: 17
- Without: 16

---

# Lower GI Diagnostics:<br/>Real-time detection

.footnote[¹Wang P, et al. Gut 2019;68:1813–1819. [doi:10.1136/gutjnl-2018-317500](https://doi.org/10.1136/gutjnl-2018-317500)]

## False alarms
- 39 total
- **0.075** per colonoscopy

## Withdrawal time (excluding biopsy time)
- Without system: 6.07 minutes
- With system: 6.18 minutes

---

class:center, middle

# CV for Diagnosis in Endoscopy

## Blind spot prevention

---

# Blind spot prevention: Upper GI<br/>WISENSE

.footnote[¹Wu L, Zhang J, Zhou W et al. Gut 2019;68:2161-2169. [doi:10.1136/gutjnl-2018-317366](https://doi.org/10.1136/gutjnl-2018-317366)]

## Design
- Single-center randomized trial
- 324 patients undergoing routine esophagogastroduodenoscopy (EGD)
- Randomized to:
    1. Normal EGD
    2. EGD with WISENSE system

## WISENSE CV Model
- Blind spot prevention system
- Updates clinican, in real-time, to anatomical sites examined
- Need to examine 26 different anatomical areas to receive a perfect
  score

---

# WISENSE: Real-time feedback

.footnote[¹Wu L, Zhang J, Zhou W et al. Gut 2019;68:2161-2169. [doi:10.1136/gutjnl-2018-317366](https://doi.org/10.1136/gutjnl-2018-317366)]

---

# WISENSE: Results

.footnote[¹Wu L, Zhang J, Zhou W et al. Gut 2019;68:2161-2169. [doi:10.1136/gutjnl-2018-317366](https://doi.org/10.1136/gutjnl-2018-317366)]

## Blind spot rate
- Control group: 22.46%
- WISENSE group: 5.86%
- Mean difference: -15.39% (95% CI -11.54, -19.23)

## Procedure time
- Control group: 4.24 min
- WISENSE group: 5.03 min

---

# Blind spot prevention: Lower GI

## C2D2

Colonoscopy Coverage Deficiency via Depth

.pull-left[.footnote[Freeman D et al. IEEE Transactions on Medical Imaging 2020;39;11:3451-3462. [doi:10.1109/TMI.2020.2994221](https://doi.org/10.1109/TMI.2020.2994221)]]

---

# C2D2: Coverage

Coverage = Orange/Green

.footnote[Freeman D et al. IEEE Transactions on Medical Imaging 2020;39;11:3451-3462. [doi:10.1109/TMI.2020.2994221](https://doi.org/10.1109/TMI.2020.2994221)]

---

# C2D2: Synthetic Data

.footnote[Freeman D et al. IEEE Transactions on Medical Imaging 2020;39;11:3451-3462. [doi:10.1109/TMI.2020.2994221](https://doi.org/10.1109/TMI.2020.2994221)]

---

# C2D2: Why synthetic data?

.footnote[Freeman D et al. IEEE Transactions on Medical Imaging 2020;39;11:3451-3462. [doi:10.1109/TMI.2020.2994221](https://doi.org/10.1109/TMI.2020.2994221)]

--
1. Know ground truth with 100% certainty
--

2. Physicians are not great at judging coverage:

---

# C2D2: Performance

.footnote[Freeman D et al. IEEE Transactions on Medical Imaging 2020;39;11:3451-3462. [doi:10.1109/TMI.2020.2994221](https://doi.org/10.1109/TMI.2020.2994221)]

--
Physicians agreed with C2D2's prediction on actual patient video clips 93% of the time.

---

class:center, middle

# Procedure

## POEM

---

# POEM

.pull-left[
- Trained CV model to recognize surgical phases
- 50 full-length POEM videos from two hospitals. One in the US and one in Japan
- 87.6% accuracy
]

.pull-left[.footnote[Ward T, et al. Surg Endosc 2020. [doi:10.1007/s00464-020-07833-9](https://doi.org/10.1007/s00464-020-07833-9)]]

---

# Surgical Fingerprints

.pull-left[.footnote[Ward T, et al. Surg Endosc 2020. [doi:10.1007/s00464-020-07833-9](https://doi.org/10.1007/s00464-020-07833-9)]]

---

# Conclusions

## AI/CV in Diagnostics

- Accurate classification of neoplastic lesions

- Blind spot warning systems to ensure complete inspection

## AI/CV in Procedures

- Accurate identification of procedure phase in POEM

## Moving forward

- Larger, more diverse, datasets from multiple institutions and countries
- Continue to transition from theoretical papers to actual patient applications

---

# Questions?