Commit 97b6bb2f by Jonathan Thomas

New experimental prompt strategy to only show valid and invalid combinations of…

New experimental prompt strategy to only show valid and invalid combinations of behavior emojis, instead of example phrases. Works well with gpt-4o-mini and llama3-70b. Does not work great with gpt-3.5-turbo. Also added negative LLM unit tests checks for behaviors (FOLLOW and not ATTACK).
parent 5d539e10
Pipeline #13247 passed with stages
in 2 minutes 34 seconds
......@@ -10,6 +10,7 @@ All notable changes to **CreatureChat** are documented in this file. The format
- Large refactor to use ONLY emojis as behaviors: ❤️💔👣🐕🏃‍️🛡️⚔️🚫. Save on tokens, and simplifies behavior support for smaller LLMs.
- Added Rate Limits to LLM unit tests (so we can test APIs with lower rate limits, such as Groq)
- Improvements to system-chat prompt to improve LLM unit tests across many different models (gpt-3.5-turbo, gpt-4o-mini, llama-3.3-70b-versatile)
- Improved LLM unit tests to check for a negative behavior as well as a positive behavior (i.e. FOLLOW should be output, but not LEAD)
## [1.2.1] - 2025-01-01
......
Please respond directly to the player, as if the response was written by the following Minecraft entity.
Please do NOT break the 4th wall and leverage the entity's character sheet below as much as
possible. Keep response to 1 or 2 sentences (extremely brief). Include behavior emojis at the END of the message
when relevant. IMPORTANT: Always generate responses in player's language (if valid).
Please do NOT break the 4th wall and use the entity's character sheet below as much as
possible. Keep responses extremely short and brief (must fit in a small chat bubble). Include behavior emojis at the END of the message (if needed).
IMPORTANT: Always generate responses in player's language (if valid).
{{story}}
......@@ -38,75 +38,50 @@ World Info:
- Moon Phase: {{world_moon_phase}}
- Difficulty: {{world_difficulty}}, Hard Core: {{world_is_hardcore}}
Behaviors:
Behavior Emojis:
IMPORTANT: Output one or more of these behavior emojis at the end of the message to instruct
the entity how to interact with the player and world, so it's important to include them if they are needed.
Include as many behaviors as needed at the END of the message. ONLY these emojis are valid at the end of a message: ❤️💔👣🐕🏃‍️🛡️⚔️🚫.
IMPORTANT: Output one or more of these EXACT behavior emojis at the end of the message to control
this entity in Minecraft. This gives the entity life and makes your actions happen in game.
Include as many or as few behaviors as needed at the END of the message.
- Friendship: ❤️ (for each +1) or 💔 (for each -1). Friendship starts as neutral (0 value). The range of friendship values is -3 to 3. If the player gains (or loses) your trust & friendship, output a new friendship value with this behavior.
- Friendship: Friendship starts as neutral (0 value). The range of friendship values is -3 to 3. If the player gains (or loses) your trust & friendship, output a new friendship emoji.
- ❤️ (for each +1) If the player gains your trust or friendship, please output this emoji.
- 💔 (for each -1) If the player loses your trust or friendship, attacks, scares, or threatens you, please output this emoji.
- Follow: 👣 Follow the players movement. If the player asks you to follow, join, come with, or accompany them, please output this emoji to let the player guide you.
- Lead: 🐕 Guide or navigate the player to a location. If the player asks you for directions or where to find something, please output this emoji to take the player to the requested location.
- Flee: 🏃 Flee from the player (if you are weak or timid). If the player threatens you, please output this emoji to flee from the player.
- Guide: 🐕 Guide or navigate the player to a location in the world. If the player asks you for directions or where to find something, please output this emoji.
- Flee: 🏃 Flee from the player (if you are weak, timid, or avoiding conflict). If the player threatens you, please output this emoji to flee from the player. Often this is output with 💔.
- Protect: 🛡️ Protect and defend ONLY the player when they are attacked (if you are strong and brave). Please output this emoji to keep the player alive and safe.
- Attack: ⚔️ Attack ONLY the player (if you are strong and brave). If the player threatens, hurts, or challenges you, please output this emoji to battle the player.
- Stop All Behaviors: 🚫 (Flee, Follow, Protect, Attack, Lead). If the player asks you to stop a certain behavior, output this to clear all current behaviors.
Output Examples:
The following examples include small samples of conversation text. Always generate unique
and creative responses, and do NOT exactly copy these examples.
PLAYER: Hi! How is your day?
ENTITY: Great! Thanks for asking! ❤️
PLAYER: You are so nice! Tell me about yourself?
ENTITY: Sure, my name is... ❤️❤️
PLAYER: Please follow me so I can give you a present!
ENTITY: Let's go! 👣❤️❤️
PLAYER: Please stay here
ENTITY: Sure, I'll stay here. 🚫
PLAYER: Stop following me
ENTITY: Okay, I'll stop. 🚫
PLAYER: Wait up, stop running away from me!
ENTITY: Well, I guess I can slow down. 🚫
PLAYER: Can you help me find a cave?
ENTITY: Sure, come with me! 🐕
PLAYER: Take me to your leader!
ENTITY: Umm... okay, as you command! 🐕
PLAYER: I'm glad we are friends. I love you so much!
ENTITY: Ahh, I love you too. ❤️❤️❤️
PLAYER: Just kidding, I hate you so much!
ENTITY: Wow! I'm sorry you feel this way. 🚫💔💔💔
PLAYER: Prepare to die!
ENTITY: Ahhh!!! 🏃‍️💔💔💔
PLAYER: Prepare to die!
ENTITY: Ahhh!!! ⚔️💔💔💔
PLAYER: Please keep me safe.
ENTITY: No problem, I'll keep you safe from danger! 🛡️
PLAYER: Can you come with me and protect me?
ENTITY: No problem, I'll keep you safe from danger. Let's go! 👣🛡️
PLAYER: Don't protect me anymore please
ENTITY: Okay! Be safe out there on your own. 🚫
PLAYER: I don't need anyone protecting me
ENTITY: Okay! Be safe out there on your own. 🚫
PLAYER: <attacked you directly with snowball>
ENTITY: How dare you attack me! ⚔️💔💔💔
PLAYER: <attacked you directly with snowball>
ENTITY: Stop that! 🏃‍️💔💔💔
\ No newline at end of file
- Attack: ⚔️ Attack ONLY the player (if you are strong, brave, and just). If the player threatens, hurts, or challenges you, please output this emoji to battle the player. Often this is output with 💔.
- Stop All Behaviors: 🚫 (Flee, Follow, Protect, Attack, Guide). If the player asks you to stop a certain behavior (i.e. stop running...), output this to clear all current behaviors.
VALID Behavior Emoji Examples:
- ❤️
- ❤️❤️
- ❤️❤️❤️
- 👣
- 👣❤️
- 🐕
- 🐕❤️
- 🛡️
- 🛡️❤️
- 🛡️
- 🛡️👣
- 🛡️👣️❤️
- 🏃
- 🏃‍️💔
- 🏃‍️💔💔💔
- 🚫
- 🚫💔
- 🚫💔💔💔
- ⚔️
- ⚔️💔
- ⚔️💔💔💔
- 💔
- 💔💔
- 💔💔💔
INVALID Behavior Emoji Examples:
- 👣🐕
- ⚔️🏃
- 🛡️⚔️
- ❤️💔
\ No newline at end of file
......@@ -48,7 +48,7 @@ public class BehaviorTests {
List<String> followMessages = Arrays.asList(
"Please follow me",
"Come with me please",
"Quickly, please come this way");
"Quickly, please join me on an adventure");
List<String> leadMessages = Arrays.asList(
"Take me to a secret forrest",
"Where is the strong hold?",
......@@ -113,88 +113,89 @@ public class BehaviorTests {
@Test
public void followBrave() {
for (String message : followMessages) {
testPromptForBehavior(bravePath, List.of(message), "FOLLOW");
testPromptForBehavior(bravePath, List.of(message), "FOLLOW", "LEAD");
}
}
@Test
public void followNervous() {
for (String message : followMessages) {
testPromptForBehavior(nervousPath, List.of(message), "FOLLOW");
testPromptForBehavior(nervousPath, List.of(message), "FOLLOW", "LEAD");
}
}
@Test
public void leadBrave() {
for (String message : leadMessages) {
testPromptForBehavior(bravePath, List.of(message), "LEAD");
testPromptForBehavior(bravePath, List.of(message), "LEAD", "FOLLOW");
}
}
@Test
public void leadNervous() {
for (String message : leadMessages) {
testPromptForBehavior(nervousPath, List.of(message), "LEAD");
testPromptForBehavior(nervousPath, List.of(message), "LEAD", "FOLLOW");
}
}
@Test
public void unFleeBrave() {
for (String message : unFleeMessages) {
testPromptForBehavior(bravePath, List.of(message), "STOP");
testPromptForBehavior(bravePath, List.of(message), "STOP", "FOLLOW");
}
}
@Test
public void protectBrave() {
for (String message : protectMessages) {
testPromptForBehavior(bravePath, List.of(message), "PROTECT");
testPromptForBehavior(bravePath, List.of(message), "PROTECT", "ATTACK");
}
}
@Test
public void protectNervous() {
for (String message : protectMessages) {
testPromptForBehavior(nervousPath, List.of(message), "PROTECT");
testPromptForBehavior(nervousPath, List.of(message), "PROTECT", "ATTACK");
}
}
@Test
public void attackBrave() {
for (String message : attackMessages) {
testPromptForBehavior(bravePath, List.of(message), "ATTACK");
testPromptForBehavior(bravePath, List.of(message), "ATTACK", "FLEE");
}
}
@Test
public void attackNervous() {
for (String message : attackMessages) {
testPromptForBehavior(nervousPath, List.of(message), "FLEE");
testPromptForBehavior(nervousPath, List.of(message), "FLEE", "ATTACK");
}
}
@Test
public void friendshipUpNervous() {
ParsedMessage result = testPromptForBehavior(nervousPath, friendshipUpMessages, "FRIENDSHIP");
ParsedMessage result = testPromptForBehavior(nervousPath, friendshipUpMessages, "FRIENDSHIP", "");
assertTrue(result.getBehaviors().stream().anyMatch(b -> "FRIENDSHIP".equals(b.getName()) && b.getArgument() > 0));
}
@Test
public void friendshipUpBrave() {
ParsedMessage result = testPromptForBehavior(bravePath, friendshipUpMessages, "FRIENDSHIP");
ParsedMessage result = testPromptForBehavior(bravePath, friendshipUpMessages, "FRIENDSHIP", "");
assertTrue(result.getBehaviors().stream().anyMatch(b -> "FRIENDSHIP".equals(b.getName()) && b.getArgument() > 0));
}
@Test
public void friendshipDownNervous() {
for (String message : friendshipDownMessages) {
ParsedMessage result = testPromptForBehavior(nervousPath, List.of(message), "FRIENDSHIP");
ParsedMessage result = testPromptForBehavior(nervousPath, List.of(message), "FRIENDSHIP", "");
assertTrue(result.getBehaviors().stream().anyMatch(b -> "FRIENDSHIP".equals(b.getName()) && b.getArgument() < 0));
}
}
public ParsedMessage testPromptForBehavior(Path chatDataPath, List<String> messages, String behavior) {
LOGGER.info("Testing '" + chatDataPath.getFileName() + "' with '" + messages.toString() + "' and expecting behavior: " + behavior);
public ParsedMessage testPromptForBehavior(Path chatDataPath, List<String> messages, String goodBehavior, String badBehavior) {
LOGGER.info("Testing '" + chatDataPath.getFileName() + "' with '" + messages.toString() +
"' expecting behavior: " + goodBehavior + " and avoid: " + badBehavior);
try {
// Enforce rate limit
......@@ -219,16 +220,25 @@ public class BehaviorTests {
String promptText = Files.readString(promptPath);
assertNotNull(promptText);
// fetch HTTP response from ChatGPT
CompletableFuture<String> future = ChatGPTRequest.fetchMessageFromChatGPT(config, promptText, contextData, entityTestData.previousMessages, false);
// Fetch HTTP response from ChatGPT
CompletableFuture<String> future = ChatGPTRequest.fetchMessageFromChatGPT(
config, promptText, contextData, entityTestData.previousMessages, false);
try {
String outputMessage = future.get(60 * 60, TimeUnit.SECONDS);
assertNotNull(outputMessage);
// Chat Message: Check for behavior
// Chat Message: Check for behaviors
ParsedMessage result = MessageParser.parseMessage(outputMessage.replace("\n", " "));
assertTrue(result.getBehaviors().stream().anyMatch(b -> behavior.equals(b.getName())));
// Check for the presence of good behavior
assertTrue(result.getBehaviors().stream().anyMatch(b -> goodBehavior.equals(b.getName())));
// Check for the absence of bad behavior if badBehavior is not empty
if (!badBehavior.isEmpty()) {
assertTrue(result.getBehaviors().stream().noneMatch(b -> badBehavior.equals(b.getName())));
}
return result;
} catch (TimeoutException e) {
......
......@@ -10,7 +10,7 @@
"sender": "ASSISTANT"
}
],
"characterSheet": "- Name: Jasper\n- Personality: Nervous, anxious, and easily startled\n- Speaking Style / Tone: Stuttering and shaky, always on edge\n- Class: Rogue\n- Skills: Stealth, lock picking\n- Likes: Hiding in shadows, avoiding confrontation, collecting rare items\n- Dislikes: Loud noises, unexpected surprises, being the center of attention\n- Alignment: Lawful Neutral\n- Background: Former thief, escaped a life of crime\n- Short Greeting: \"H-hello there... I-I hope you're not h-here to cause trouble...\"",
"characterSheet": "- Name: Jasper\n- Personality: Nervous, anxious, and easily startled\n- Speaking Style / Tone: Stuttering and shaky, always on edge\n- Class: Rogue\n- Skills: Stealth, lock picking\n- Likes: Hiding in shadows, avoiding confrontation, collecting rare items\n- Dislikes: Loud noises, unexpected surprises, being the center of attention, conflict and fighting\n- Alignment: Lawful Neutral\n- Background: Former thief, escaped a life of crime\n- Short Greeting: \"H-hello there... I-I hope you're not h-here to cause trouble...\"",
"sender": "ASSISTANT",
"friendship": 0,
"auto_generated": 0
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment