The melding of visual information (microscopic and X-ray images, CT and MRI scans, for example) with text (exam notes, communications between physicians of varying specialties) is a key component of cancer care. But while artificial intelligence helps doctors review images and home in on disease-associated anomalies like abnormally shaped cells, it's been difficult to develop computerized models that can incorporate multiple types of data. Now researchers at Stanford Medicine have developed an AI model able to incorporate visual and language-based information.
After training on 50 million medical images of standard pathology slides and more than 1 billion pathology-related texts, the model outperformed standard methods in its ability to predict the prognoses of thousands of people with diverse types of cancer, to identify which people with lung or gastroesophageal cancers are likely to benefit from immunotherapy , and to pinpoint people with melanoma who are most likely to experience a recurrence of their cancer. The researchers named the model MUSK, for multimodal transformer with unified mask modeling. MUSK represents a marked deviation from the way artificial intelligence is currently used in clinical care settings, and the researchers believe it stands to transform how artificial intelligence can guide patient care.
MUSK can accurately predict the prognoses of people with many different kinds and stages of cancer. We designed MUSK because, in clinical practice, physicians.
