Interpreting and summarising the insights gained from medical images such as radiology output is a time-consuming task that involves highly trained experts and often represents a bottleneck in clinical diagnosis pipelines.
Consequently, there is a considerable need for automatic methods that can approximate this mapping from visual information to condensed textual descriptions. The more image characteristics are known, the more structured are the radiology scans and hence, the more efficient are the radiologists regarding interpretation.
We work on the basis of a large-scale collection of figures from open access biomedical journal articles (PubMed Central). All images in the training data are accompanied by UMLS concepts extracted from the original image caption.
The first step to automatic image captioning and scene understanding is identifying the presence and location of relevant concepts in a large corpus of medical images. Based on the visual image content, this projects provides the building blocks for the scene understanding step by identifying the individual components from which captions are composed. The concepts can be further applied for context-based image and information retrieval purposes.
ImageCLEFcaption was added as a task of the ImageCLEF evaluation campaign in 2017. ImageCLEF is part of the Cross Language Evaluation Forum (CLEF). The goal is to create databases to interpret medical images.
The organisers of the ImageCLEFcaption challenge distribute the “collection” consisting of images and annotations for image concept extraction. Participants then apply their tools and techniques which are then evaluated in a blind collection.
Partners
This project is run in collaboration with the University of Applied Sciences and Arts Dortmund, Germany, and the University of Applied Sciences Western Switzerland, Sierre, Switzerland.
Publications
Müller, H., Kalpathy-Cramer, J. and Garcia Seco De Herrera, A., (2019). Experiences from the ImageCLEF Medical Retrieval and Annotation Tasks. In: Information Retrieval Evaluation in a Changing World Lessons Learned from 20 Years of CLEF. Editors: Ferro, N. and Peters, C., . Springer. 231- 250. 978-3-030-22947-4.
Pelka, O., Friedrich, CM., García Seco de Herrera, A. and Müller, H., Overview of the ImageCLEFmed 2020 Concept Prediction Task: Medical Image Understanding.
García Seco de Herrera, A., Parrilla Andrade, F., Bentley, L. and Aceves Compean, A., Essex at ImageCLEFcaption 2020 task.
Ionescu, B., Müller, H., Péteri, R., Dang-Nguyen, D-T., Zhou, L., Piras, L., Riegler, M., Halvorsen, P., Tran, M-T., Lux, M., Gurrin, C., Chamberlain, J., Clark, A., Campello, A., Garcia Seco De Herrera, A., Ben Abacha, A., Datla, V., A. Hasan, S., Liu, J., Demner-Fushman, D., Obioma, P., Friedrich, CM., Dicente Cid, Y., Kozlovski, S., Liauchuk, V., Kovalev, V., Berari, R., Brie, P., Fichou, D., Dogariu, M., Daniel Stefan, L. and Constantin, MG., (2020). ImageCLEF 2020: Multimedia Retrieval in Lifelogging, Medical, Nature, and Security Applications.
Ionescu, B., Müller, H., Péteri, R., Abacha, AB., Datla, V., Hasan, SA., Demner-Fushman, D., Kozlovski, S., Liauchuk, V., Cid, YD., Kovalev, V., Pelka, O., Friedrich, CM., Garcia Seco De Herrera, A., Ninh, V-T., Le, T-K., Zhou, L., Piras, L., Riegler, M., Halvorsen, PL., Tran, M-T., Lux, M., Gurrin, C., Dang-Nguyen, D-T., Chamberlain, J., Clark, A., Campello, A., Fichou, D., Berari, R., Brie, P., Dogariu, M., Stefan, LD. and Constantin, MG., (2020). ImageCLEF 2020: Multimedia Retrieval in Lifelogging, Medical, Nature, and Internet Applications.
Pelka, O., Friedrich, CM., Seco De Herrera, AG. and Müller, H., (2019). Overview of the ImageCLEFmed 2019 concept detection task.
Ionescu, B., Müller, H., Péteri, R., Cid, YD., Liauchuk, V., Kovalev, V., Klimuk, D., Tarasau, A., Abacha, AB., Hasan, SA., Datla, V., Liu, J., Demner-Fushman, D., Dang-Nguyen, D-T., Piras, L., Riegler, M., Tran, M-T., Lux, M., Gurrin, C., Pelka, O., Friedrich, CM., Garcia Seco De Herrera, A., Garcia, N., Kavallieratou, E., del Blanco, CR., Cuevas, C., Vasillopoulos, N., Karampidis, K., Chamberlain, J., Clark, A. and Campello, A., (2019). ImageCLEF 2019: Multimedia Retrieval in Medicine, Lifelogging, Security and Nature.
Ionescu, B., Müller, H., Péteri, R., Dang-Nguyen, D-T., Piras, L., Riegler, M., Tran, M-T., Lux, M., Gurrin, C., Cid, YD., Liauchuk, V., Kovalev, V., Ben Abacha, A., Hasan, SA., Datla, V., Liu, J., Demner-Fushman, D., Pelka, O., Friedrich, CM., Chamberlain, J., Clark, A., Garcia Seco De Herrera, A., Garcia, N., Kavallieratou, E., del Blanco, CR., Rodríguez, CC., Vasillopoulos, N. and Karampidis, K., (2019). ImageCLEF 2019: Multimedia Retrieval in Lifelogging, Medical, Nature, and Security Applications.
Garcia Seco De Herrera, A., Eickhof, C., Andrearczyk, V. and Müller, H., (2018). Overview of the ImageCLEF 2018 Caption Prediction Tasks.
Ionescu, B., Müller, H., Villegas, M., Garcia Seco De Herrera, A., Eickhoff, C., Andrearczyk, V., Dicente Cid, Y., Liauchuk, V., Kovalev, V., Hasan, SA., Ling, Y., Farri, O., Liu, J., Lungren, M., Dang-Nguyen, D-T., Piras, L., Riegler, M., Zhou, L., Lux, M. and Gurrin, C., (2018). Overview of ImageCLEF 2018: Challenges, Datasets and Evaluation.
Eickhoff, C. ,and Schwall, I., and Garcia Seco De Herrera, A., and Müller, H., (2017) "Overview of ImageCLEFcaption 2017 – Image Caption Prediction and Concept Detection for Biomedical Images". In: CLEF Conference and Labs of the Evaluation Forum, CLEF 2017.
Ionescu, B., Müller, H., Villegas, M., Arenas, H., Boato, G., Dang-Nguyen, D-T., Dicente Cid, Y., Eickhoff, C., Garcia Seco De Herrera, A., Gurrin, C., Islam, B., Kovalev, V., Liauchuk, V., Mothe, J., Piras, L., Riegler, M. and Schwall, I., (2017). Overview of ImageCLEF 2017: Information extraction from images.
Abacha, AB., Seco De Herrera, AG., Gayen, S., Demner-Fushman, D. and Antani, S., (2017). NLM at ImageCLEF 2017 caption task.
