ReasonFlux-PRM Reproduce the problem

[Amazing-J]

感谢很有启发性的工作，但是我在复现的时候遇到了几个问题，希望解答一下：

1. ReasonFlux-PRM的loss计算如下：

请问final loss是如何计算的？训练集的数据格式是否如下：
Q + Step_1 + <extract_0> + Step_2 + <extract_0> + ... ... + Step_T + <extract_0>
但是我想知道的是final loss是在哪里计算的loss？

2. Inference时候的数据格式？
   我使用了论文中贴出的case study输入进开源的PRM-7B模型中时，无法给出论文中演示的step-level reward分数。请展示一个inference的数据格式是否可行？
   还有一个疑惑对于 Trajectory-leval 的reward是如何计算的？ 论文中好像没有提到。

[Amazing-J]

@jiaruzouu @YangLing0818 @ZhaochenYu0201 Thank you for the very inspiring work. However, I encountered several problems when reproducing it and hope to get some answers:

1. Excuse me, how is the final loss calculated? Is the data format of the training set as follows: Q + Step_1 + <extract_0> + Step_2 + <extract_0> +... + Step_T + <extract_0>?
   But what I want to know is where the final loss is calculated?

2. What is the data format during Inference? When I input the case study posted in the paper into the open-source PRM-7B model, it can't give the step-level reward score demonstrated in the paper. Is it feasible to show an inference data format? I also have a doubt. How is the Trajectory-level reward calculated? It doesn't seem to be mentioned in the paper.