Talk Less, Call RightEnhancing Role-Play LLM Agents with Automatic Prompt Optimization and Role Prompting

Abstract

This report investigates approaches for prompting a tool-augmented large language model (LLM) to act as a role-playing dialogue agent in the API track of the Commonsense Persona-grounded Dialogue Challenge (CPDC) 2025. In this setting, dialogue agents often produce overly long in-character responses (over-speaking) while failing to use tools effectively according to the persona (under-acting), such as generating function calls that do not exist or making unnecessary tool calls before answering.

We explore four prompting approaches to address these issues: basic role prompting, improved role prompting, automatic prompt optimization (APO), and rule-based role prompting. The rule-based role prompting (RRP) approach achieved the best performance through two techniques—character-card/scene-contract design and strict enforcement of function calling—leading to an overall score of 0.571, compared with the zero-shot baseline’s 0.519.

These findings show how explicit rules can improve the effectiveness and reliability of role-playing dialogue agents in this setting. The best-performing prompts and APO tool are available in the project repository.

Adapted from the abstract of arXiv v2.

Conversation and action in the same game world

A game’s non-player character needs to stay in character while selecting the right tools and arguments. CPDC 2025 tests both capabilities. Team HarryPeter participated in the API track using GPT-4o-mini, improving prompts for a shared underlying model.

Setting: paper §3 and the official challenge.

From role descriptions to explicit rules

The baseline uses separate function-planning and dialogue prompts. Four approaches build on that starting point:

1. Basic role prompting. Add persona and scene information, and express tool outputs as narrative summaries.
2. Improved role prompting. Refine instructions manually after examining invented functions, repeated calls, and tool misuse.
3. Automatic prompt optimization. Try a zero-shot rewrite with Claude Sonnet 4 and an iterative ProTeGi feedback loop.
4. Rule-based role prompting. Combine a Character-Card/Scene Contract (CSC) with Hard-Enforced Function Calling (HEF) instructions.

Rules that address recurring failures

Action first: make the relevant function call before producing the character’s response. Fewer calls: consolidate information needs with a composite tool where available. Exact schemas: use valid function names and parameter keys, and ask for clarification when the target is ambiguous.

The CSC listing specifies at most one call per turn. The HEF listing allows a hard cap of four while favoring one. These are distinct prompt constraints in the source; the page does not assume a single universal call limit.

The largest gain is in task execution

Rule-based role prompting achieves the highest overall and Task 1 scores among the reported variants. Improved role prompting retains the highest Task 2 score.

Task 1 rises from 0.442 to 0.531: a gain of 0.089, or about 20.1% relative to the baseline. Task 2 rises from 0.597 to 0.611, but remains below the improved role prompt’s 0.617.

Partial correctness still exceeds exact accuracy

A separate local evaluation uses the starter kit’s training data as a validation set. These call-level metrics describe the best configuration; they are not the private-test leaderboard scores above.

The gap between partial credit and exact matches leaves substantial room to improve argument grounding. In particular, 71.4% partial function-name accuracy does not mean 71.4% fully correct calls.

Results and evaluation definitions: paper §5, Tables 1–2.

What changes in a conversation?

Two examples from the paper illustrate changes in tool selection. The traces below are reported examples, not a live agent demo.

check_basic_info("Hunter’s Bow")
check_attack("Hunter’s Bow")

check_basic_info("Hunter’s Bow")

check_basic_info("Commercial Escort quest")

Adapted from paper §5.3 and slide 11 of the supplied presentation.

Limitations

The study concerns the CPDC API-track setting with GPT-4o-mini. Its results do not establish that these constraints work equally well across models or open-ended environments. A strict one-call policy can underserve requests that need multiple tools, and exact argument accuracy remains low. Including the full dialogue history also adds latency; history-truncation strategies were not studied.

@misc{ruangtanusak2025talklesscallright,
  title = {Talk Less, Call Right: Enhancing Role-Play LLM Agents with Automatic Prompt Optimization and Role Prompting},
  author = {Saksorn Ruangtanusak and Pittawat Taveekitworachai and Kunat Pipatanakul},
  year = {2025},
  eprint = {2509.00482},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  doi = {10.48550/arXiv.2509.00482},
  url = {https://arxiv.org/abs/2509.00482}
}