This paper investigates the influence of prompt politeness on the performance of Large Language Models (LLMs) across English, Chinese, and Japanese, testing the hypothesis that LLMs reflect human cultural norms regarding social etiquette. The authors evaluate whether impolite prompts degrade performance and whether there is an optimal politeness level that varies across different languages.

Core Methodology

The researchers designed eight distinct prompt templates per language, ranging from highly polite to highly impolite, and validated these levels using questionnaires administered to native speakers. The experimental framework utilizes three primary categories of tasks:

1. Summarization: Evaluated using CNN/Dailymail (English) and XL-Sum (Chinese/Japanese) datasets, with metrics including BERTScore, ROUGE-L, and output length.
2. Language Understanding: The authors constructed a new benchmark, the Japanese Massive Multitask Language Understanding (JMMLU), by translating and culturally adapting the MMLU dataset. Performance was measured on MMLU (English), C-Eval (Chinese), and JMMLU (Japanese). Answers were deemed correct if the choice index appeared anywhere in the generated output. Statistical significance was verified using a t-test heatmap based on the logarithmic p-value of performance differences.
3. Stereotypical Bias Detection: Due to the limitations of existing bias detection methods for API-based models, the authors proposed a "Bias Index" (BI). This metric measures the frequency of inconsistent evaluations across pairs of sentences where only the bias-triggering term (e.g., age or gender) is swapped:

$$BI = \frac{\text{Number of Different Pairs}}{\text{Total Number of Pairs}} \times 100$$

Key Findings

• Performance Impact: Impolite prompts consistently result in performance deterioration across most models. However, excessive politeness does not guarantee superior performance; the "best" politeness level is highly language-dependent.
• Cultural Mirroring: The models appear to encode cultural tendencies. For example, in Japanese—a language with a complex "Keigo" (honorific) system—the models exhibit specific fluctuations in generation length and accuracy that align with established social hierarchical dynamics.
• Advanced vs. Specialized Models: While smaller or specialized models (e.g., Llama2-70B, ChatGLM3) demonstrate high sensitivity to prompt politeness, stronger models like GPT-4 show greater stability, often effectively ignoring minor politeness variations to prioritize task completion.
• Role of Training: Comparison between the Llama2-70B base model and its RLHF-tuned counterpart reveals that while base models are susceptible to prompt tone, RLHF serves as a mechanism that aligns the model’s performance volatility more closely with human social expectations.
• Bias Sensitivity: The study finds that stereotypical bias in LLMs is not static but fluctuates according to the politeness of the prompt, suggesting that human-like "moral relaxation" in polite environments—or defensive posturing in impolite ones—is implicitly mirrored in the models' outputs.

The paper concludes that prompt engineering must account for cross-cultural nuances in politeness to maximize LLM efficacy and reduce bias in globalized applications.