difference between logits and topic_addition

[xiyan524]

Thanks for your excellent work~

Actually, I rebuild this structure by tensorflow while I met some problem. I discover that logits and topic_addition may own different scale, such as logits distribution are -20-20 and topic_addition may be -1-1. I am not sure add them directly will effect? When I check the training process, I find that the whole netwok rely on logits much more. Is there any wrong?

[dangitstam]

I'm not quite sure, I haven't done any preliminary experiments to see what the difference in scale is. I'm not surprised that the logits from the classifier are so extreme; perhaps it's better to normalize both the logits and topic additions before adding them together but the authors of the paper didn't specify what they did.

[xiyan524]

May you provide your test results? Is that similar to results in paper?
Actually, I have run my reproduction code(written in tf) on other similar datasets, while I did not find significant improvement as mentioned in paper. Also, I can not get in touch with author by email. So, I am sorry to have so much detail questions.

[dangitstam]

I haven't actually been able to reproduce the paper's results unfortunately, perhaps I will try again in the future