[HELP] Repeatability issue of results in multi-threaded environment in 2021 version

[MachineLeaningJoeZhan]

Version
2021.2.0-1877-176bdf51370

Platform
win64

Device
iGPU

Infer model
input: 60 * 60 pixel images
output: 20 probabilities
infer: async-infer

Detailed bug scenarios
I am a software developer. I build a AI Infer program with openvino2021. But recently , I found that when simultaneously starting 5 threads for parallel inference, there may be occasional issues where the same image input has different output results. I can confirm that this is a multi-threaded issue; Because this issue has never occurred during single threaded operation. And it can be fixed when I update the openvino version from 2021 to 2023LTS.
I wonder if there are some known multthreads_problem in openvino2021? Can you provide the solvetion and commit messages for me? Thank You!

[Ganesh-403]

@MachineLeaningJoeZhan The fact that:

• the issue only occurs with OpenVINO 2021.2,
• only in multi-threaded asynchronous inference,
• and disappears after upgrading to 2023 LTS,

strongly suggests you're running into a limitation or bug in the older runtime rather than a problem with your model.

From what I could find, I wasn't able to locate a specific changelog or commit that says "fixed nondeterministic multi-threaded inference on iGPU". However, there have been substantial changes to the Runtime and asynchronous inference implementation between the 2021 and 2023 releases, so it's certainly plausible that your issue was resolved as part of those improvements rather than by a single targeted fix. :contentReference[oaicite:0]{index=0}

A few things I'd check for the 2021 version:

1. InferRequest ownership

   Each thread should have its own InferRequest. Sharing a single InferRequest across multiple threads is not thread-safe, even when using asynchronous inference. The documentation states that wait()/wait_for() are thread-safe, but that does not imply an InferRequest itself can be used concurrently by multiple threads. :contentReference[oaicite:1]{index=1}

2. Input/Output buffers

   Make sure each thread has its own input and output tensors (or buffers). Reusing the same memory across concurrent async requests can lead to race conditions that appear as non-repeatable inference results.

3. GPU plugin

   Since you're using the iGPU, could you confirm whether each thread creates its own InferRequest from a shared executable network, or whether multiple threads share the same request object? That detail would help determine whether this is an application-level synchronization issue or a runtime issue.

Unfortunately, regarding your specific question:

"Can you provide the solution and commit messages?"

I couldn't find a public commit or release note explicitly documenting this exact issue or its fix. If you've already confirmed that upgrading to 2023 LTS eliminates the problem without changing your application logic, I'd consider that the recommended solution unless maintaining 2021.2 compatibility is a hard requirement.

If you do need to stay on 2021.2, sharing a minimal reproducer (how the ExecutableNetwork/InferRequest objects are created and used across the five threads) would make it much easier to determine whether you're hitting a known runtime limitation or an application-level race condition.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.