|
The question generation task is a type of task in the NLP field that uses the answer of the question as the input for a model, allowing the model to trace back to the corresponding original question. Prompts can be designed to guide the large language model to generate questions that are closer to its corresponding answers. Previous work involves adding textual content to the prompt to clarify specific and help the model understand, however, there is still room for improvement in the quality of the questions generated by these prompts. On the basis of previous work, this paper adopts the method of automatic question generation and includes few-shot examples as part of the model input. We automatically generated 20 basic prompts and proposed three methods for constructing demonstrations:(1) Most helpful shots (2) Maximum entropy shots (3) Mixed method. We used GPT-3.5-turbo as the experimental model and selected two suitable datasets TruthfulQA and Alpaca-cleaned for this task. In the stage of evaluating the quality of generated questions, we used text cosine similarity and GPT-4 scoring to evaluate the impact of designing demonstrations on the quality of generated questions before and after. We compared the generated questions of the model with the preset questions corresponding to the answers in the dataset. Quantify the difference using cosine similarity: The improved result is 3.09% higher than the original prompt's generation question result ; we used the GPT-4 large language model for intelligent scoring, which also showed an improvement in scores on both datasets. The study also explored the impact of different numbers of shots under a specific method of the task. When the number of shots is 6, the combined result of text similarity on two datasets was 0.828, achieving the best result. When the number of shots is 5, the best GPT-4 rating result is 4.435.
|
