Make Python async API thread-safe

[pentschev]

The UCXX implementation is intended to allow thread-safe Python operations, at least when using the progress thread. However, at least at the high-level the API is currently not thread-safe.

Consider for example the case of start_notifier_thread/stop_notifier_thread, one thread may pause just before starting the thread, and another destroy the thread object, which would lead to call start() on an object of NoneType.

[DrBinary9]

@pentschev Hi, I would like to work on this. I have previously implemented one-one and many-one threading as a college project. I have experience in this domain. I think we should consider adding a lock in ApplicationContext (self._notifier_lock) and guard all notifier-thread operations.

[pentschev]

Hi @DrBinary9 , if you're willing to work on this that would be great. However, I expect this to be a much more complex and time consuming issue than #350, for example. The description only points out the obvious pieces that are not thread-safe currently, but there may be (potentially many) more, I didn't investigate this in-depth to provide a detailed description of everything that would be necessary. If this is something you're really willing to work on and invest the time on it, I would suggest to start writing a multi-threaded test (there are currently no multi-threaded tests in the codebase) and then based on that verify what issues appear then solve one-by-one. I don't have an exact time estimate, but I think this could be 2-5 days whole days of work time to get done, depending on the issues we end up finding.

Please let me know if this is truly something you'd be interested and be able to dedicate that much time to work on. I'm happy to help you with it, but I don't know a priori exactly what will need to be done.

[DrBinary9]

@pentschev In that case I think I'll be better if I start working on smaller issues. I still have to familiarize myself with this repository. Could you please recommend a specific issue I could take that would take maybe less than a day or two to complete?

[pentschev]

Thanks @DrBinary9 for being willing to make more improvements! I have gone through the current open issues, as well as potentially missing things from UCX-Py, and it looks like there's not a ton that is currently known to be missing or high-priority. The issues that could be worked on are:

• Capability to measure GIL contention in Python benchmarks #502: this is low-priority but good to have that was implemented in UCX-Py and missed in UCXX;
• Bring Python API up-to-speed with recent C++ features #187: there are a bunch of new C++ features that were not yet implemented in Python, although there are currently no known users that are in need of using any of those features this is something we need to address sooner or later.

I'd be happy to help you with any of them if you're willing to contribute more of your time and effort. I'm assuming you're more interested and knowledgeable of Python so this is why I'm proposing those above. If you have some good knowledge of C++ as well and are interested in those too, there are certainly many more things that need to be implemented there.