Remove special backend treatment

[makoeppel]

Some parts of the code have: if os.environ["KERAS_BACKEND"] == "torch", which should be avoided.