FNCSE: contrastive learning for unsupervised sentence embedding with false negative samples

Guiqiang Wu

doi:10.1117/12.2685528

15 August 2023 FNCSE: contrastive learning for unsupervised sentence embedding with false negative samples

Guiqiang Wu

Proceedings Volume 12719, Second International Conference on Electronic Information Technology (EIT 2023); 127194R (2023) https://doi.org/10.1117/12.2685528
Event: Second International Conference on Electronic Information Technology (EIT 2023), 2023, Wuhan, China

Abstract

In recent years, contrastive learning has gradually been applied to the field of natural language processing from the explosion of the computer vision community. Generally speaking, for a given sentence, the original sentence is used as the anchor point, the current model uses a special data enhancement method to generate positive samples, and the remaining sentences in the batch are used as negative samples for the sentence. Although this approach effectively improves sentence embedding, we believe that in the process of constructing negative samples, negative samples will also have information similar to the current anchor point, and should not be mistakenly attributed to negative samples, which will confuse the learning ability of the model, and The increase in the number of negative pairs during training is beneficial to the training of the model. In this paper, we propose methods to identify false negatives in the text domain, and two strategies to mitigate the impact of false negatives: masking false negatives and adding to the set of positives. In addition, we propose an optimized version of momentum contrast to expand the number of negative pairs. Our method is based on the improvement made on SimCSE called FNCSE. We evaluate FNCSE on multiple benchmark datasets in the semantic similarity (STS) task. The experimental results show that in the Bert base, FNCSE averages 1.29% Spearman Correlation is better than SimCSE.

Citation Download Citation

Guiqiang Wu "FNCSE: contrastive learning for unsupervised sentence embedding with false negative samples", Proc. SPIE 12719, Second International Conference on Electronic Information Technology (EIT 2023), 127194R (15 August 2023); https://doi.org/10.1117/12.2685528

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available