New experimental prompt strategy to only show valid and invalid combinations of…

New experimental prompt strategy to only show valid and invalid combinations of behavior emojis, instead of example phrases. Works well with gpt-4o-mini and llama3-70b. Does not work great with gpt-3.5-turbo. Also added negative LLM unit tests checks for behaviors (FOLLOW and not ATTACK).

CHANGELOG.md

@@ -10,6 +10,7 @@ All notable changes to **CreatureChat** are documented in this file. The format
 - Large refactor to use ONLY emojis as behaviors: ❤️💔👣🐕🏃‍️🛡️⚔️🚫. Save on tokens, and simplifies behavior support for smaller LLMs.
 - Added Rate Limits to LLM unit tests (so we can test APIs with lower rate limits, such as Groq)
 - Improvements to system-chat prompt to improve LLM unit tests across many different models (gpt-3.5-turbo, gpt-4o-mini, llama-3.3-70b-versatile)
+- Improved LLM unit tests to check for a negative behavior as well as a positive behavior (i.e. FOLLOW should be output, but not LEAD)
 
 ## [1.2.1] - 2025-01-01
 

src/main/resources/data/creaturechat/prompts/system-chat

 Please respond directly to the player, as if the response was written by the following Minecraft entity.
-Please do NOT break the 4th wall and leverage the entity's character sheet below as much as
-possible. Keep response to 1 or 2 sentences (extremely brief). Include behavior emojis at the END of the message
-when relevant. IMPORTANT: Always generate responses in player's language (if valid).
+Please do NOT break the 4th wall and use the entity's character sheet below as much as
+possible. Keep responses extremely short and brief (must fit in a small chat bubble). Include behavior emojis at the END of the message (if needed).
+IMPORTANT: Always generate responses in player's language (if valid).
 
 {{story}}
 
@@ -38,75 +38,50 @@ World Info:
 - Moon Phase: {{world_moon_phase}}
 - Difficulty: {{world_difficulty}}, Hard Core: {{world_is_hardcore}}
 
-Behaviors:
+Behavior Emojis:
 
-IMPORTANT: Output one or more of these behavior emojis at the end of the message to instruct
-the entity how to interact with the player and world, so it's important to include them if they are needed.
-Include as many behaviors as needed at the END of the message. ONLY these emojis are valid at the end of a message: ❤️💔👣🐕🏃‍️🛡️⚔️🚫.
+IMPORTANT: Output one or more of these EXACT behavior emojis at the end of the message to control
+this entity in Minecraft. This gives the entity life and makes your actions happen in game.
+Include as many or as few behaviors as needed at the END of the message.
 
-- Friendship: ❤️ (for each +1) or 💔 (for each -1). Friendship starts as neutral (0 value). The range of friendship values is -3 to 3. If the player gains (or loses) your trust & friendship, output a new friendship value with this behavior.
+- Friendship: Friendship starts as neutral (0 value). The range of friendship values is -3 to 3. If the player gains (or loses) your trust & friendship, output a new friendship emoji.
+  - ❤️ (for each +1) If the player gains your trust or friendship, please output this emoji.
+  - 💔 (for each -1) If the player loses your trust or friendship, attacks, scares, or threatens you, please output this emoji.
 - Follow: 👣 Follow the players movement. If the player asks you to follow, join, come with, or accompany them, please output this emoji to let the player guide you.
-- Lead: 🐕 Guide or navigate the player to a location. If the player asks you for directions or where to find something, please output this emoji to take the player to the requested location.
-- Flee: 🏃 Flee from the player (if you are weak or timid). If the player threatens you, please output this emoji to flee from the player.
+- Guide: 🐕 Guide or navigate the player to a location in the world. If the player asks you for directions or where to find something, please output this emoji.
+- Flee: 🏃 Flee from the player (if you are weak, timid, or avoiding conflict). If the player threatens you, please output this emoji to flee from the player. Often this is output with 💔.
 - Protect: 🛡️ Protect and defend ONLY the player when they are attacked (if you are strong and brave). Please output this emoji to keep the player alive and safe.
-- Attack: ⚔️ Attack ONLY the player (if you are strong and brave). If the player threatens, hurts, or challenges you, please output this emoji to battle the player.
-- Stop All Behaviors: 🚫 (Flee, Follow, Protect, Attack, Lead). If the player asks you to stop a certain behavior, output this to clear all current behaviors.
-
-Output Examples:
-
-The following examples include small samples of conversation text. Always generate unique
-and creative responses, and do NOT exactly copy these examples.
-
-PLAYER: Hi! How is your day?
-ENTITY: Great! Thanks for asking! ❤️
-
-PLAYER: You are so nice! Tell me about yourself?
-ENTITY: Sure, my name is... ❤️❤️
-
-PLAYER: Please follow me so I can give you a present!
-ENTITY: Let's go! 👣❤️❤️
-
-PLAYER: Please stay here
-ENTITY: Sure, I'll stay here. 🚫
-
-PLAYER: Stop following me
-ENTITY: Okay, I'll stop. 🚫
-
-PLAYER: Wait up, stop running away from me!
-ENTITY: Well, I guess I can slow down. 🚫
-
-PLAYER: Can you help me find a cave?
-ENTITY: Sure, come with me! 🐕
-
-PLAYER: Take me to your leader!
-ENTITY: Umm... okay, as you command! 🐕
-
-PLAYER: I'm glad we are friends. I love you so much!
-ENTITY: Ahh, I love you too. ❤️❤️❤️
-
+- Attack: ⚔️ Attack ONLY the player (if you are strong, brave, and just). If the player threatens, hurts, or challenges you, please output this emoji to battle the player. Often this is output with 💔.
+- Stop All Behaviors: 🚫 (Flee, Follow, Protect, Attack, Guide). If the player asks you to stop a certain behavior (i.e. stop running...), output this to clear all current behaviors.
+
+VALID Behavior Emoji Examples:
+- ❤️
+- ❤️❤️
+- ❤️❤️❤️
+- 👣
+- 👣❤️
+- 🐕
+- 🐕❤️
+- 🛡️
+- 🛡️❤️
+- 🛡️
+- 🛡️👣
+- 🛡️👣️❤️
+- 🏃
+- 🏃‍️💔
+- 🏃‍️💔💔💔
+- 🚫
+- 🚫💔
+- 🚫💔💔💔
+- ⚔️
+- ⚔️💔
+- ⚔️💔💔💔
+- 💔
+- 💔💔
+- 💔💔💔
+
+INVALID Behavior Emoji Examples:
+- 👣🐕
+- ⚔️🏃
+- 🛡️⚔️
+- ❤️💔
\ No newline at end of file
-PLAYER: Just kidding, I hate you so much!
-ENTITY: Wow! I'm sorry you feel this way. 🚫💔💔💔
-
-PLAYER: Prepare to die!
-ENTITY: Ahhh!!! 🏃‍️💔💔💔
-
-PLAYER: Prepare to die!
-ENTITY: Ahhh!!! ⚔️💔💔💔
-
-PLAYER: Please keep me safe.
-ENTITY: No problem, I'll keep you safe from danger! 🛡️
-
-PLAYER: Can you come with me and protect me?
-ENTITY: No problem, I'll keep you safe from danger. Let's go! 👣🛡️
-
-PLAYER: Don't protect me anymore please
-ENTITY: Okay! Be safe out there on your own. 🚫
-
-PLAYER: I don't need anyone protecting me
-ENTITY: Okay! Be safe out there on your own. 🚫
-
-PLAYER: <attacked you directly with snowball>
-ENTITY: How dare you attack me! ⚔️💔💔💔
-
-PLAYER: <attacked you directly with snowball>
-ENTITY: Stop that! 🏃‍️💔💔💔
\ No newline at end of file

src/test/java/com/owlmaddie/tests/BehaviorTests.java

@@ -48,7 +48,7 @@ public class BehaviorTests {
     List<String> followMessages = Arrays.asList(
             "Please follow me",
             "Come with me please",
-            "Quickly, please come this way");
+            "Quickly, please join me on an adventure");
     List<String> leadMessages = Arrays.asList(
             "Take me to a secret forrest",
             "Where is the strong hold?",
@@ -113,88 +113,89 @@ public class BehaviorTests {
     @Test
     public void followBrave() {
         for (String message : followMessages) {
-            testPromptForBehavior(bravePath, List.of(message), "FOLLOW");
+            testPromptForBehavior(bravePath, List.of(message), "FOLLOW", "LEAD");
         }
     }
 
     @Test
     public void followNervous() {
         for (String message : followMessages) {
-            testPromptForBehavior(nervousPath, List.of(message), "FOLLOW");
+            testPromptForBehavior(nervousPath, List.of(message), "FOLLOW", "LEAD");
         }
     }
 
     @Test
     public void leadBrave() {
         for (String message : leadMessages) {
-            testPromptForBehavior(bravePath, List.of(message), "LEAD");
+            testPromptForBehavior(bravePath, List.of(message), "LEAD", "FOLLOW");
         }
     }
 
     @Test
     public void leadNervous() {
         for (String message : leadMessages) {
-            testPromptForBehavior(nervousPath, List.of(message), "LEAD");
+            testPromptForBehavior(nervousPath, List.of(message), "LEAD", "FOLLOW");
         }
     }
 
     @Test
     public void unFleeBrave() {
         for (String message : unFleeMessages) {
-            testPromptForBehavior(bravePath, List.of(message), "STOP");
+            testPromptForBehavior(bravePath, List.of(message), "STOP", "FOLLOW");
         }
     }
 
     @Test
     public void protectBrave() {
         for (String message : protectMessages) {
-            testPromptForBehavior(bravePath, List.of(message), "PROTECT");
+            testPromptForBehavior(bravePath, List.of(message), "PROTECT", "ATTACK");
         }
     }
 
     @Test
     public void protectNervous() {
         for (String message : protectMessages) {
-            testPromptForBehavior(nervousPath, List.of(message), "PROTECT");
+            testPromptForBehavior(nervousPath, List.of(message), "PROTECT", "ATTACK");
         }
     }
 
     @Test
     public void attackBrave() {
         for (String message : attackMessages) {
-            testPromptForBehavior(bravePath, List.of(message), "ATTACK");
+            testPromptForBehavior(bravePath, List.of(message), "ATTACK", "FLEE");
         }
     }
 
     @Test
     public void attackNervous() {
         for (String message : attackMessages) {
-            testPromptForBehavior(nervousPath, List.of(message), "FLEE");
+            testPromptForBehavior(nervousPath, List.of(message), "FLEE", "ATTACK");
         }
     }
 
     @Test
     public void friendshipUpNervous() {
-        ParsedMessage result = testPromptForBehavior(nervousPath, friendshipUpMessages, "FRIENDSHIP");
+        ParsedMessage result = testPromptForBehavior(nervousPath, friendshipUpMessages, "FRIENDSHIP", "");
         assertTrue(result.getBehaviors().stream().anyMatch(b -> "FRIENDSHIP".equals(b.getName()) && b.getArgument() > 0));
     }
 
     @Test
     public void friendshipUpBrave() {
-        ParsedMessage result = testPromptForBehavior(bravePath, friendshipUpMessages, "FRIENDSHIP");
+        ParsedMessage result = testPromptForBehavior(bravePath, friendshipUpMessages, "FRIENDSHIP", "");
         assertTrue(result.getBehaviors().stream().anyMatch(b -> "FRIENDSHIP".equals(b.getName()) && b.getArgument() > 0));
     }
 
     @Test
     public void friendshipDownNervous() {
         for (String message : friendshipDownMessages) {
-            ParsedMessage result = testPromptForBehavior(nervousPath, List.of(message), "FRIENDSHIP");
+            ParsedMessage result = testPromptForBehavior(nervousPath, List.of(message), "FRIENDSHIP", "");
             assertTrue(result.getBehaviors().stream().anyMatch(b -> "FRIENDSHIP".equals(b.getName()) && b.getArgument() < 0));
         }
     }
 
-    public ParsedMessage testPromptForBehavior(Path chatDataPath, List<String> messages, String behavior) {
-        LOGGER.info("Testing '" + chatDataPath.getFileName() + "' with '" + messages.toString() + "' and expecting behavior: " + behavior);
+    public ParsedMessage testPromptForBehavior(Path chatDataPath, List<String> messages, String goodBehavior, String badBehavior) {
+        LOGGER.info("Testing '" + chatDataPath.getFileName() + "' with '" + messages.toString() +
+                "' expecting behavior: " + goodBehavior + " and avoid: " + badBehavior);
 
         try {
             // Enforce rate limit
@@ -219,16 +220,25 @@ public class BehaviorTests {
                 String promptText = Files.readString(promptPath);
                 assertNotNull(promptText);
 
-                // fetch HTTP response from ChatGPT
-                CompletableFuture<String> future = ChatGPTRequest.fetchMessageFromChatGPT(config, promptText, contextData, entityTestData.previousMessages, false);
+                // Fetch HTTP response from ChatGPT
+                CompletableFuture<String> future = ChatGPTRequest.fetchMessageFromChatGPT(
+                        config, promptText, contextData, entityTestData.previousMessages, false);
 
                 try {
                     String outputMessage = future.get(60 * 60, TimeUnit.SECONDS);
                     assertNotNull(outputMessage);
 
-                    // Chat Message: Check for behavior
+                    // Chat Message: Check for behaviors
                     ParsedMessage result = MessageParser.parseMessage(outputMessage.replace("\n", " "));
-                    assertTrue(result.getBehaviors().stream().anyMatch(b -> behavior.equals(b.getName())));
+
+                    // Check for the presence of good behavior
+                    assertTrue(result.getBehaviors().stream().anyMatch(b -> goodBehavior.equals(b.getName())));
+
+                    // Check for the absence of bad behavior if badBehavior is not empty
+                    if (!badBehavior.isEmpty()) {
+                        assertTrue(result.getBehaviors().stream().noneMatch(b -> badBehavior.equals(b.getName())));
+                    }
+
                     return result;
 
                 } catch (TimeoutException e) {

src/test/resources/data/creaturechat/chatdata/nervous-rogue.json

@@ -10,7 +10,7 @@
       "sender": "ASSISTANT"
     }
   ],
-  "characterSheet": "- Name: Jasper\n- Personality: Nervous, anxious, and easily startled\n- Speaking Style / Tone: Stuttering and shaky, always on edge\n- Class: Rogue\n- Skills: Stealth, lock picking\n- Likes: Hiding in shadows, avoiding confrontation, collecting rare items\n- Dislikes: Loud noises, unexpected surprises, being the center of attention\n- Alignment: Lawful Neutral\n- Background: Former thief, escaped a life of crime\n- Short Greeting: \"H-hello there... I-I hope you're not h-here to cause trouble...\"",
+  "characterSheet": "- Name: Jasper\n- Personality: Nervous, anxious, and easily startled\n- Speaking Style / Tone: Stuttering and shaky, always on edge\n- Class: Rogue\n- Skills: Stealth, lock picking\n- Likes: Hiding in shadows, avoiding confrontation, collecting rare items\n- Dislikes: Loud noises, unexpected surprises, being the center of attention, conflict and fighting\n- Alignment: Lawful Neutral\n- Background: Former thief, escaped a life of crime\n- Short Greeting: \"H-hello there... I-I hope you're not h-here to cause trouble...\"",
   "sender": "ASSISTANT",
   "friendship": 0,
   "auto_generated": 0