Advertisement
Journal of Investigative Dermatology Home

Deep learning and pathomics analyses reveal cell nuclei as important features for mutation prediction of BRAF-mutated melanomas

  • Author Footnotes
    † These authors contributed equally to this work.
    Randie H. Kim
    Footnotes
    † These authors contributed equally to this work.
    Affiliations
    The Ronald O. Perelman Department of Dermatology, NYU Grossman School of Medicine, New York, New York

    Interdisciplinary Melanoma Cooperative Group, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Author Footnotes
    † These authors contributed equally to this work.
    Sofia Nomikou
    Footnotes
    † These authors contributed equally to this work.
    Affiliations
    Department of Pathology, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Nicolas Coudray
    Affiliations
    Applied Bioinformatics Laboratories, NYU Grossman School of Medicine, New York, New York

    Skirball Institute Department of Cell Biology, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • George Jour
    Affiliations
    The Ronald O. Perelman Department of Dermatology, NYU Grossman School of Medicine, New York, New York

    Interdisciplinary Melanoma Cooperative Group, NYU Grossman School of Medicine, New York, New York

    Department of Pathology, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Zarmeena Dawood
    Affiliations
    Interdisciplinary Melanoma Cooperative Group, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Runyu Hong
    Affiliations
    Institute for Systems Genetics, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Eduardo Esteva
    Affiliations
    New York University Tandon School of Engineering, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Theodore Sakellaropoulos
    Affiliations
    Department of Pathology, NYU Grossman School of Medicine, New York, New York

    Laura and Isaac Perlmutter Cancer Center, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Douglas Donnelly
    Affiliations
    The Ronald O. Perelman Department of Dermatology, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Una Moran
    Affiliations
    Interdisciplinary Melanoma Cooperative Group, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Aristides Hatzimemos
    Affiliations
    The Ronald O. Perelman Department of Dermatology, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Jeffrey S. Weber
    Affiliations
    Interdisciplinary Melanoma Cooperative Group, NYU Grossman School of Medicine, New York, New York

    Laura and Isaac Perlmutter Cancer Center, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Narges Razavian
    Affiliations
    Deparmtent of Radiology, NYU Grossman School of Medicine, New York, New York

    Department of Population Health, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Iannis Aifantis
    Affiliations
    Department of Pathology, NYU Grossman School of Medicine, New York, New York

    Laura and Isaac Perlmutter Cancer Center, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • David Fenyo
    Affiliations
    Institute for Systems Genetics, NYU Grossman School of Medicine, New York, New York

    Department of Biochemistry and Molecular Pharmacology, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Matija Snuderl
    Affiliations
    Department of Pathology, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Richard Shapiro
    Affiliations
    Interdisciplinary Melanoma Cooperative Group, NYU Grossman School of Medicine, New York, New York

    Department of Surgery, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Russell S. Berman
    Affiliations
    Interdisciplinary Melanoma Cooperative Group, NYU Grossman School of Medicine, New York, New York

    Department of Surgery, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Iman Osman
    Correspondence
    Corresponding Author: Aristotelis Tsirigos, PhD, Professor of Pathology, Director, Applied Bioinformatics Laboratories, NYU Grossman School of Medicine, 435 East 30th Street, New York, New York Phone: 646-501-2693,
    Affiliations
    The Ronald O. Perelman Department of Dermatology, NYU Grossman School of Medicine, New York, New York

    Interdisciplinary Melanoma Cooperative Group, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Aristotelis Tsirigos
    Correspondence
    Corresponding Author: Aristotelis Tsirigos, PhD, Professor of Pathology, Director, Applied Bioinformatics Laboratories, NYU Grossman School of Medicine, 435 East 30th Street, New York, New York Phone: 646-501-2693,
    Affiliations
    Department of Pathology, NYU Grossman School of Medicine, New York, New York

    Applied Bioinformatics Laboratories, NYU Grossman School of Medicine, New York, New York
    Search for articles by this author
  • Author Footnotes
    † These authors contributed equally to this work.
Published:October 29, 2021DOI:https://doi.org/10.1016/j.jid.2021.09.034

      Abstract

      Image-based analysis as a method for mutation detection can be advantageous in settings when tumor tissue is limited or unavailable for direct testing. Here, we utilize two distinct and complementary machine learning methods of analyzing whole slide images (WSI) for predicting mutated BRAF. In the first method, WSI of melanomas from 256 patients were used to train a deep convolutional neural network (CNN) in order to develop a fully automated model that first selects for tumor-rich areas (Area Under the Curve AUC=0.96) then predicts for mutated BRAF (AUC=0.71). Saliency mapping was performed and revealed that pixels corresponding to nuclei were the most relevant to network learning. In the second method, WSI were analyzed using a pathomics pipeline that first annotates nuclei and then quantifies nuclear features, demonstrating that mutated BRAF nuclei were significantly larger and rounder nuclei compared to BRAF WT nuclei. Lastly, we developed a model that combines clinical information, deep learning, and pathomics that improves the predictive performance for mutated BRAF to AUC=0.89. Not only does this provide additional insights on how BRAF mutations affect tumor structural characteristics, machine learning-based analysis of WSI has the potential to be integrated into higher order models for understanding tumor biology.

      Abbreviations:

      AUC (Area Under the Curve), BRAF (B-Raf proto-oncogene), CI (Confidence interval), CNN (Convolutional Neural Network), H&E (Hematoxylin and eosin), FFPE (Formalin-fixed paraffin-embedded), NYU (New York University), ROC (Receiver Operating Curve), ROI (Regions of Interest), TCGA (The Cancer Genome Atlas), WSI (Whole Slide Images), WT (Wild-type)
      To read this article in full you will need to make a payment
      Purchase one-time access
      Society Members (SID/ESDR), remember to log in for access.
      Subscribe to Journal of Investigative Dermatology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References:

        • Ascierto P.A.
        • Kirkwood J.M.
        • Grob J.J.
        • Simeone E.
        • Grimaldi A.M.
        • Maio M.
        • et al.
        The role of BRAF V600 mutation in melanoma.
        J Transl Med. 2012; 10: 85
        • Bauer J.
        • Buttner P.
        • Murali R.
        • Okamoto I.
        • Kolaitis N.A.
        • Landi M.T.
        • et al.
        BRAF mutations in cutaneous melanoma are independently associated with age, anatomic site of the primary tumor, and the degree of solar elastosis at the primary tumor site.
        Pigment Cell Melanoma Res. 2011; 24: 345-351
        • Broekaert S.M.
        • Roy R.
        • Okamoto I.
        • van den Oord J.
        • Bauer J.
        • Garbe C.
        • et al.
        Genetic and morphologic features for melanoma classification.
        Pigment Cell Melanoma Res. 2010; 23: 763-770
        • Campanella G.
        • Hanna M.G.
        • Geneslaw L.
        • Miraflor A.
        • Werneck Krauss Silva V.
        • Busam K.J.
        • et al.
        Clinical-grade computational pathology using weakly supervised deep learning on whole slide images.
        Nat Med. 2019; 25: 1301-1309
        • Cancer Genome Atlas N.
        Genomic Classification of Cutaneous Melanoma.
        Cell. 2015; 161: 1681-1696
        • Carpenter A.E.
        • Jones T.R.
        • Lamprecht M.R.
        • Clarke C.
        • Kang I.H.
        • Friman O.
        • et al.
        CellProfiler: image analysis software for identifying and quantifying cell phenotypes.
        Genome Biol. 2006; 7: R100
        • Coudray N.
        • Ocampo P.S.
        • Sakellaropoulos T.
        • Narula N.
        • Snuderl M.
        • Fenyo D.
        • et al.
        Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning.
        Nat Med. 2018; 24: 1559-1567
        • Fu Y.
        • Jung A.W.
        • Torne R.V.
        • Gonzalez S.
        • Vöhringer H.
        • Shmatko A.
        • et al.
        Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis.
        Nature Cancer. 2020;
        • Graham S.
        • Vu Q.D.
        • Raza S.E.A.
        • Azam A.
        • Tsang Y.W.
        • Kwak J.T.
        • et al.
        Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images.
        Med Image Anal. 2019; 58: 101563
        • Johannet P.
        • Coudray N.
        • Donnelly D.M.
        • Jour G.
        • Illa-Bochaca I.
        • Xia Y.
        • et al.
        Using Machine Learning Algorithms to Predict Immunotherapy Response in Patients with Advanced Melanoma.
        Clin Cancer Res. 2020;
        • Kather J.N.
        • Heij L.R.
        • Grabsch H.I.
        • Loeffler C.
        • Echle A.
        • Muti H.S.
        • et al.
        Pan-cancer image-based detection of clinically actionable genetic alterations.
        Nature Cancer. 2020;
        • Kwon M.R.
        • Shin J.H.
        • Park H.
        • Cho H.
        • Hahn S.Y.
        • Park K.W.
        Radiomics Study of Thyroid Ultrasound for Predicting BRAF Mutation in Papillary Thyroid Carcinoma: Preliminary Results.
        AJNR Am J Neuroradiol. 2020; 41: 700-705
        • Liu W.
        • Kelly J.W.
        • Trivett M.
        • Murray W.K.
        • Dowling J.P.
        • Wolfe R.
        • et al.
        Distinct clinical and pathological features are associated with the BRAF(T1799A(V600E)) mutation in primary melanoma.
        J Invest Dermatol. 2007; 127: 900-905
        • Menzies A.M.
        • Haydu L.E.
        • Visintin L.
        • Carlino M.S.
        • Howle J.R.
        • Thompson J.F.
        • et al.
        Distinguishing clinicopathologic features of patients with V600E and V600K BRAF-mutant metastatic melanoma.
        Clin Cancer Res. 2012; 18: 3242-3249
        • Mobadersany P.
        • Yousefi S.
        • Amgad M.
        • Gutman D.A.
        • Barnholtz-Sloan J.S.
        • Velazquez Vega J.E.
        • et al.
        Predicting cancer outcomes from histology and genomics using convolutional networks.
        Proc Natl Acad Sci U S A. 2018; 115: E2970-E2979
        • Ninatti G.
        • Kirienko M.
        • Neri E.
        • Sollini M.
        • Chiti A.
        Imaging-Based Prediction of Molecular Therapy Targets in NSCLC by Radiogenomics and AI Approaches: A Systematic Review.
        Diagnostics (Basel). 2020; 10
        • Reinhard E.
        • Ashikhmin M.
        • Gooch B.
        • Shirley P.
        Color Transfer between Images.
        IEEE Comput Graph Appl. 2001; 21: 34-41
        • Shofty B.
        • Artzi M.
        • Shtrozberg S.
        • Fanizzi C.
        • DiMeco F.
        • Haim O.
        • et al.
        Virtual biopsy using MRI radiomics for prediction of BRAF status in melanoma brain metastasis.
        Sci Rep. 2020; 10: 6623
      1. Smilkov D, Thorat N, Kim B, ViÈgas F, Wattenberg M. SmoothGrad: removing noise by adding noise. ArXiv 2017;abs/1706.03825.

        • Thomas N.E.
        • Edmiston S.N.
        • Alexander A.
        • Groben P.A.
        • Parrish E.
        • Kricker A.
        • et al.
        Association Between NRAS and BRAF Mutational Status and Melanoma-Specific Survival Among Patients With Higher-Risk Primary Melanoma.
        JAMA Oncol. 2015; 1: 359-368
        • van der Laak J.
        • Litjens G.
        • Ciompi F.
        Deep learning in histopathology: the path to the clinic.
        Nat Med. 2021; 27: 775-784
        • Viros A.
        • Fridlyand J.
        • Bauer J.
        • Lasithiotakis K.
        • Garbe C.
        • Pinkel D.
        • et al.
        Improving melanoma classification by integrating genetic and morphologic features.
        PLoS Med. 2008; 5e120
        • Vu Q.D.
        • Graham S.
        • Kurc T.
        • To M.N.N.
        • Shaban M.
        • Qaiser T.
        • et al.
        Methods for Segmentation and Classification of Digital Microscopy Tissue Images.
        Front Bioeng Biotechnol. 2019; 7: 53
        • Wang Z.
        • Jensen M.A.
        • Zenklusen J.C.
        A Practical Guide to The Cancer Genome Atlas (TCGA).
        Methods Mol Biol. 2016; 1418: 111-141
        • Wich L.G.
        • Hamilton H.K.
        • Shapiro R.L.
        • Pavlick A.
        • Berman R.S.
        • Polsky D.
        • et al.
        Developing a multidisciplinary prospective melanoma biospecimen repository to advance translational research.
        Am J Transl Res. 2009; 1: 35-43
        • Yu K.H.
        • Zhang C.
        • Berry G.J.
        • Altman R.B.
        • Re C.
        • Rubin D.L.
        • et al.
        Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features.
        Nat Commun. 2016; 7: 12474