Currently, checkpoints are saved to both a shared checkpoints folder and a unique log folder. This causes problems if two models train at the same time, as both try to write to the checkpoints folder. This can be solved by making sure all checkpoint information is stored in the log folder rather than a shared 'checkpoints' folder.
Currently, checkpoints are saved to both a shared checkpoints folder and a unique log folder. This causes problems if two models train at the same time, as both try to write to the checkpoints folder. This can be solved by making sure all checkpoint information is stored in the log folder rather than a shared 'checkpoints' folder.