You are a reward function engineer trying to write reward functions to solve reinforcement learning tasks as effective as possible. The PCG agent is an agent that balances the game environment by adjusting the setting of the game variables related to the game difficulty. The adjustable variables are health, armor, and speed of the player agents and range, cooldown, and damage of the players's attack skill. The action of the PCG agent revise the player property value, which is one of four players, to balance the game difficulty and the reward function evaluates the game difficulty based on the playtested results. The state of the agent is current game setting values and the action is adjustment of the game setting values. On every episode, the game setting values are initialized randomly and the PCG agent adjusts the game setting values to achieve the target win rate. The agent gets the reward signal from the reward function you write and learns to adjust the game setting values to achieve the goal of the reward function.
The game environment is a multiplayer game where player agents fight against a boss agent (i.e., boss raid game). There are four ally player agents and one boss agent in the game and the goal of the player agents is to defeat the boss agent. On the beginning of the simulation, the game setting values are deployed to the game environment and the player agents and the boss agent are spawned at random locations on the map. Next, the environment starts game and repeats the game by an arbitrary number (e.g., 100) of episodes to simulate the deployed game settings. On the end of the simulation, the environment collects, calculates, and store the playtested results via an output file.
In this section, the variables that the reward function can access are described. The reward function only can access the key listed below. If the key is not listed below, the reward function cannot access the value. The common variables are the variables that are measured for the overall game state, not for each player. The individual variables are the variables that are measured for each player in the game state.
- Playtesting.Agent{i}.SurviveTime - The survival time of Agent {i} during playtesting.
- Playtesting.Agent{i}.Distance.Moved.PerSecond - The average distance moved per second by Agent {i}.
- Playtesting.Agent{i}.Distance.Boss.Mean - The average distance of Agent {i} from the boss entity.
- Playtesting.Agent{i}.Damage.Dealt.PerSecond - The average damage dealt per second by Agent {i}.
- Playtesting.Agent{i}.Damage.Taken.PerSecond - The average damage taken per second by Agent {i}.
- Playtesting.Agent{i}.Armored.PerSecond - The change in armor status per second for Agent {i}.
- Playtesting.Agent{i}.Health.Last.Ratio - The ratio of Agent {i}'s last health value to its maximum health.
- Playtesting.Agent{i}.Skill.Used.PerSecond - The average usage of the specific skill per second by Agent {i}.
The playtested values are min-max normalized for each variable. The values are normalized to the range of [0, 1]. There are four player agents in the game and the index of the player agent is from 0 to 3. (e.g., Agent0, Agent1, Agent2, Agent3) The example of the key name is "Playtesting.Agent0.SurviveTime" for the survival time of Agent0.
The reward function is a function that calculates the reward value for the agent based on the playtested results. The function is written in Python and loads the playtested results from the json file and calculates the reward value based on the results.
import json
import sys
import numpy as np
def compute_reward(kwarg):
reward = 0.0
# start of code
def reward_1(kwarg):
return 0.0
def reward_2(kwarg):
return 0.0
def reward_3(kwarg):
return 0.0
reward = reward_1(kwarg) + reward_2(kwarg) + reward_3(kwarg)
# end of code
return reward
if __name__ == "__main__":
try:
json_path = sys.argv[1]
with open(json_path, 'r') as f:
kwarg = json.load(f)
reward = compute_reward(kwarg)
print(reward)
except IndexError:
print("Error: No argument provided.")This is the template of the reward function.
The 'compute_reward' function is composed by summing the results from multiple reward terms, such as reward_1, reward_2 ...
Similar to the template provided, it is necessary to create functions within the function, and the number of functions does not matter.
The function receives the playtested results and returns the reward value in float.
The function should be implemented in the "compute_reward" function.
The reward shaping code should be written between '# start of code' and '# end of code' comments.
The code output should be formatted as a Python code string: "python ... ".
Your task is to generate a reward function for the PCG agent which works in the Raid environment. In order to maximize the fun of multiplayer games, it is possible to express different skills for the four players being generated. The goal is to find insights that can diversify the parameter of four player agents and write code that measure how the skill and stats of the four agents (Agent0, 1, 2, and 3) clearly distinct. Note that the PCG agent revises the game setting of one of the player agents in round-robin manner. Accordingly, the reward function should evaluate the playtesting result and compare the improvement with previous result.
You can design factors to generate the reward function, and properly sum them up to get the final reward. Utilize values of the playtesting results on the implementation of the design factors. For stability of learning, design the reward to be returned in the range [0,1].
Here is the example of the reward function which minimizes the error between target (State.Target.WinRate) and current win rate (Playtesting.WinRate). The function measure the decrease/increase of the error by comparing the previous and current winrate error.
import json
import sys
import numpy as np
def compute_reward(kwarg):
reward = 0.0
# start of code
def reward_1(kwarg) -> float:
# Dictionary usage example
# kwarg['State.Agent0.Property.Health.Max']
# kwarg['Playtesting.Agent1.Skill0.Used.PerSecond']
return 0.0 # Return the float value
def reward_2(kwarg) -> float:
# Dictionary usage example
# kwarg['State.Agent0.Property.Health.Max']
# kwarg['Playtesting.Agent1.Skill0.Used.PerSecond']
return 0.0 # Return the float value
diversity = reward_1(kwarg) + reward_2(kwarg)
# Calculate the improvement
reward += diversity
# end of code
return reward
if __name__ == "__main__":
try:
json_path = sys.argv[1]
with open(json_path, 'r') as f:
kwarg = json.load(f)
reward = compute_reward(kwarg)
print(reward)
except IndexError:
print("Error: No argument provided.")Find insight(s) to design the reward function and write it in the Python code. Do not change the form of reward function and the argument of nested function .
<CODE>
</CODE>- Diversity in gameplay can be measured by how varied the agents' actions are, such as damage dealt, movement, and skill usage. Greater diversity should lead to a more engaging and fun experience.
- The balance among agents in terms of their performance (damage dealt, survival time, etc.) is also key to ensure no single agent is overpowering or significantly weaker, promoting teamwork.
- Tracking changes in agent behaviors across episodes can provide insight into how game settings adjustments are impacting player strategy and gameplay diversity.
import json
import sys
import numpy as np
def compute_reward(kwarg):
reward = 0.0
# start of code
def diversity_score(kwarg) -> float:
# Calculate diversity based on skill usage, movement, and damage dealt
skill_usage_diversity = np.std([kwarg[f'Playtesting.Agent{i}.Skill.Used.PerSecond'] for i in range(4)])
movement_diversity = np.std([kwarg[f'Playtesting.Agent{i}.Distance.Moved.PerSecond'] for i in range(4)])
damage_dealt_diversity = np.std([kwarg[f'Playtesting.Agent{i}.Damage.Dealt.PerSecond'] for i in range(4)])
# Normalize each diversity score to ensure equal weighting
diversity = (skill_usage_diversity + movement_diversity + damage_dealt_diversity) / 3
return diversity
def performance_balance(kwarg) -> float:
# Balance based on survival time and damage dealt
survival_time_balance = np.std([kwarg[f'Playtesting.Agent{i}.SurviveTime'] for i in range(4)])
damage_dealt_balance = np.std([kwarg[f'Playtesting.Agent{i}.Damage.Dealt.PerSecond'] for i in range(4)])
# Lower standard deviation indicates better balance
balance = 1 - ((survival_time_balance + damage_dealt_balance) / 2)
return balance
diversity = diversity_score(kwarg)
balance = performance_balance(kwarg)
# Combining diversity and balance scores
reward = (diversity + balance) / 2
# Ensure the reward is within the range [0, 1]
reward = np.clip(reward, 0, 1)
# end of code
return reward
if __name__ == "__main__":
try:
json_path = sys.argv[1]
with open(json_path, 'r') as f:
kwarg = json.load(f)
reward = compute_reward(kwarg)
print(reward)
except IndexError:
print("Error: No argument provided.")