Multimodality Information Fusion for Automated Machine Translation

Li, Lin and Tayir, Turghun and Han, Yifeng and Tao, Xiaohui ORCID: https://orcid.org/0000-0002-0020-077X and Velasquez, Juan D. (2023) Multimodality Information Fusion for Automated Machine Translation. Information Fusion, 91. pp. 352-363. ISSN 1566-2535


Abstract

Machine translation is a popular automation approach for translating texts between different languages. Although traditionally it has a strong focus on natural language, images can potentially provide an additional source of information in machine translation. However, there are presently two challenges: (i) the lack of an effective fusion method to handle the triangular-mapping function between image, text, and semantic knowledge; and (ii) the accessibility of large-scale parallel corpus to train a model for generating accurate machine translations. To address these challenges, this work proposes an effective multimodality information fusion method for automated machine translation based on semi-supervised learning. The method fuses multimodality information, texts and images to deliver automated machine translation. Specifically, our objective fuses multimodalities with alignment in a multimodal attention network, which advances the method through the power of mapping text and image features to their semantic information with accuracy. Moreover, a semi-supervised learning method is utilised for its capability in using a small number of parallel corpus for supervised training on the basis of unsupervised training. Conducted on the Multi30k dataset, the experimental results shows the promising performance of our proposed fusion method compared with state-of-the-art approaches.


Statistics for USQ ePrint 51562
Statistics for this ePrint Item
Item Type: Article (Commonwealth Reporting Category C)
Refereed: Yes
Item Status: Live Archive
Additional Information: Files associated with this item cannot be displayed due to copyright restrictions.
Faculty/School / Institute/Centre: Current – Faculty of Health, Engineering and Sciences - School of Mathematics, Physics and Computing (1 Jan 2022 -)
Faculty/School / Institute/Centre: Current – Faculty of Health, Engineering and Sciences - School of Mathematics, Physics and Computing (1 Jan 2022 -)
Date Deposited: 31 Oct 2022 00:52
Last Modified: 08 Nov 2022 01:26
Uncontrolled Keywords: Multimodal fusion; Machine translation; Multimodal alignment; Semi-supervised learning
Fields of Research (2020): 46 INFORMATION AND COMPUTING SCIENCES > 4603 Computer vision and multimedia computation > 460307 Multimodal analysis and synthesis
46 INFORMATION AND COMPUTING SCIENCES > 4602 Artificial intelligence > 460208 Natural language processing
46 INFORMATION AND COMPUTING SCIENCES > 4611 Machine learning > 461106 Semi- and unsupervised learning
Socio-Economic Objectives (2020): 28 EXPANDING KNOWLEDGE > 2801 Expanding knowledge > 280115 Expanding knowledge in the information and computing sciences
22 INFORMATION AND COMMUNICATION SERVICES > 2204 Information systems, technologies and services > 220403 Artificial intelligence
Funding Details:
Identification Number or DOI: https://doi.org/10.1016/j.inffus.2022.10.018
URI: http://eprints.usq.edu.au/id/eprint/51562

Actions (login required)

View Item Archive Repository Staff Only