Seminar “Arabic Multimodal Language Modeling”
May 12 @ 11:00 AM - 1:00 PM
On May 12, the Faedo Lecture Hall at ISTI-CNR will host the seminar “Arabic Multimodal Language Modeling”, presented by Hawau Olamide Toyin, a PhD student in Natural Language Processing at the Mohammed bin Zayed University of Artificial Intelligence (MBZUAI).
Hawau Olamide Toyin is a visiting scholar at ISTI-CNR thanks to a TNA (Transnational Access) grant, awarded within the framework of ITSERR – Italian Strengthening of ESFRI RI Resilience, a project funded by the Italian National Recovery and Resilience Plan (PNRR). Her research stay is part of the international mobility initiatives aimed at promoting access to cutting-edge research infrastructures in the field of humanities and social sciences.
During the seminar, she will present ArTST, a pre-trained transformer model for Arabic speech and text, which extends the unified SpeechT5 architecture originally developed for English. ArTST enables joint learning of speech and text representations and can be fine-tuned for a variety of tasks, including speech recognition, speech synthesis, speech enhancement, speaker identification, and dialect classification.
The seminar also addresses a key research question: Can a single model be trained to handle multiple cross-modal speech-text tasks? The results obtained so far—both in high-resource (English) and low-resource (Arabic) settings—are promising.
This seminar is part of the ITSERR research activities in artificial intelligence, language technologies, and digital infrastructures for the humanities.