"Comms::Receive failed" when inference the whole val dataset to generate proof_log

[zhou745]

When I tried to do inference on the whole validation dataset, I encounter an error caused by receive_int() inside the sandboxee, it always occurs several hours after I start the inference. Does anyone know what caused this issue? and how to solve it? Below is the error screen:

Traceback (most recent call last):
File "/mnt/lustre/zhoujingqiu/./deepmath_bin/main.runfiles/deepmath/deepmath/deephol/main.py", line 28, in
tf.app.run(main)
File "/mnt/lustre/zhoujingqiu/deepmath_bin/main.runfiles/org_tensorflow/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/mnt/lustre/zhoujingqiu/deepmath_bin/main.runfiles/absl_py/absl/app.py", line 300, in run
_run_main(main, args)
File "/mnt/lustre/zhoujingqiu/deepmath_bin/main.runfiles/absl_py/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "/mnt/lustre/zhoujingqiu/./deepmath_bin/main.runfiles/deepmath/deepmath/deephol/main.py", line 24, in main
prover_runner.run_pipeline(*prover_flags.process_prover_flags())
File "/mnt/lustre/zhoujingqiu/deepmath_bin/main.runfiles/deepmath/deepmath/deephol/prover_runner.py", line 56, in run_pipeline
proof_log = this_prover.prove(task)
File "/mnt/lustre/zhoujingqiu/deepmath_bin/main.runfiles/deepmath/deepmath/deephol/prover.py", line 130, in prove
return self.prove_one_wrapper(task)
File "/mnt/lustre/zhoujingqiu/deepmath_bin/main.runfiles/deepmath/deepmath/deephol/prover.py", line 159, in prove_one_wrapper
error_message = self.prove_one(tree, task)
File "/mnt/lustre/zhoujingqiu/deepmath_bin/main.runfiles/deepmath/deepmath/deephol/prover.py", line 284, in prove_one
self.prover_options.tactic_timeout_ms)
File "/mnt/lustre/zhoujingqiu/deepmath_bin/main.runfiles/deepmath/deepmath/deephol/prover_util.py", line 179, in try_tactics
request, score)
File "/mnt/lustre/zhoujingqiu/deepmath_bin/main.runfiles/deepmath/deepmath/deephol/proof_search_tree.py", line 240, in init
response = tree.proof_assistant.ApplyTactic(request)
File "/mnt/lustre/zhoujingqiu/deepmath_bin/main.runfiles/deepmath/deepmath/deephol/public/proof_assistant.py", line 34, in ApplyTactic
return self.stub.ApplyTactic(request)
File "/mnt/lustre/zhoujingqiu/.conda/envs/deephol/lib/python3.5/site-packages/grpc/_channel.py", line 826, in call
return _end_unary_response_blocking(state, call, False, None)
File "/mnt/lustre/zhoujingqiu/.conda/envs/deephol/lib/python3.5/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Failure("Comms::Receive failed")"
debug_error_string = "{"created":"@1595325578.814092705","description":"Error received from peer ipv4:0.0.0.0:2080","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Failure("Comms::Receive failed")","grpc_sta
tus":13}"