Enhancing use of BERT information in neural machine translation with masking-BERT attention

Xi Chen; Lin Wu; Yuanhao Zhang

doi:10.1117/12.2684653

21 July 2023 Enhancing use of BERT information in neural machine translation with masking-BERT attention

Xi Chen, Lin Wu, Yuanhao Zhang

Proceedings Volume 12717, 3rd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2023); 1271733 (2023) https://doi.org/10.1117/12.2684653
Event: 3rd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2023), 2023, Wuhan, China

Abstract

BERT has shown remarkable performance in several natural language processing tasks, but it fails to exhibit the same high performance in cross-lingual tasks, particularly machine translation. To address this issue, we propose a BERT-enhanced neural machine translation (BE-NMT) model that optimizes the use of the information contained in BERT by NMT. Our proposed model comprises three components: (1) A MASKING strategy to mitigate knowledge forgetting caused by fine-tuning of BERT on the NMT task; (2) Serial and parallel processing of multi-attention models for incorporating BERT into the NMT system; (3) Fusing multiple hidden layer outputs of BERT to supplement the missing linguistic information of its final hidden layer output. We conducted experiments on several translation tasks, and our proposed model notably outperforms the strong baseline by improving 1.93 BLEU points on the United Nations Parallel Corpus English→Chinese task. Additionally, our model also achieves remarkable performance on other translation tasks.

Citation Download Citation

Xi Chen, Lin Wu, and Yuanhao Zhang "Enhancing use of BERT information in neural machine translation with masking-BERT attention", Proc. SPIE 12717, 3rd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2023), 1271733 (21 July 2023); https://doi.org/10.1117/12.2684653

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available