pike with python multiprocessing fails with "OSError: [Errno 9] Bad file descriptor"

[dinebhar]

Hi,

i could see the pike is not working when i use it with python multiprocessing module.

i am using pike version '0.2.7'

my requirements is to create multiple process to establish the connection and send smb requests to my server.

following are my observations:
1. Without multiprocessing:
`

import pike.model
import multiprocessing

def test():
... client = pike.model.Client()
... connection = client.connect("127.0.0.1", 445).negotiate()
... channel = connection.session_setup("ISI\root%a")
... tree = channel.tree_connect("TestShare")
... print("IT WORKED...")
...

test()
IT WORKED...

`

2. with multiprocessing:
`

class Test(multiprocessing.Process):
... def init(self):
... multiprocessing.Process.init(self)
...
... def run(self):
... client = pike.model.Client()
... connection = client.connect("127.0.0.1", 445).negotiate()
... channel = connection.session_setup("**********") # intentionaly putting * here :-)
... tree = channel.tree_connect("TestShare")
...

Test().start()

ERROR OUTPUT HERE:

Process Test-34:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
File "", line 7, in run
File "/var/crash/tools/lib/python2.7/site-packages/pike/model.py", line 327, in connect
return self.connect_submit(server, port).result()
File "/var/crash/tools/lib/python2.7/site-packages/pike/model.py", line 339, in connect_submit
return Connection(self, server, port).connection_future
File "/var/crash/tools/lib/python2.7/site-packages/pike/model.py", line 499, in init
self.create_socket(family, socktype)
File "/var/crash/tools/lib/python2.7/site-packages/pike/transport.py", line 74, in create_socket
self.set_socket(sock)
File "/var/crash/tools/lib/python2.7/site-packages/pike/transport.py", line 83, in set_socket
self.poller.add_channel(self)
File "/var/crash/tools/lib/python2.7/site-packages/pike/transport.py", line 349, in add_channel
self.kq.control(events, 0)
OSError: [Errno 9] Bad file descriptor

`

could any please take a look and let me know what is the wrong in it.

[isi-mfurer]

This is because pike uses a global poller (in your case here, KQueue) to poll all connected sockets for read/write/error activity. I suspect what's happening is that the forked process is sharing the parent process's kqueue fd but is not sharing the subsequently opened socket fd.

The only chance you have of making this work correctly would be to use an explicitly defined poller in each process -- but this is going to be inconvenient because all of the helper methods which block will have to be changed to async calls, and then you would have to call .loop on that process's poller. Likely this would be more complicated than you're looking to set up.

Typically pike doesn't mix well with threading code because pike follows the async/single-thread/event model -- any threading code that touches it has to be very careful to avoid modifying global state since there is no mutex protection. It can be done, but unless you're willing to dig deeply into the pike model/transport layer, you will likely run into subtle, hard to track down bugs.

What are you trying to accomplish with the multiprocessing module? Pike is designed allow multiple connections and async requests within a single thread/process? Do you think you're hitting a concurrency limit that could be exceeded by using multiple processes?

I'm reluctant to suggest that you try to hack around these limitation in pike -- IMO it would be a better use of your time to understand how pike supports concurrency and attempt to model your test case and needs against that.

[dinebhar]

Thank you for the information.

The thing is, prior to this version i was using pike (i don't remember the version) which was using asyncore module. and it seems it is removed and not present in pike 0.2.7

The reason behind moving to upper version is because of concurrency limit issue i was facing with select system call with multiprocessing.

my code was working fine with that older pike version.

my requirements are -

• i want to create a number of client connection in order to generate stress/load on my server just to test how much oplock break requests my server can handle.

i can do that by single thread which can create number of connections sequentially and then do take oplock and break it. but it will take much longer if the count goes in thousands. that's why i was trying to create them using multiple processes.

i am not sure whether i will be able to updated it to fulfill my requirement but i try.

thanks for the information.

[isi-mfurer]

I would encourage you to try to refactor into the single threaded model. A big reason for removing asyncore in favor of the new pike.transport module was specifically to do away with the limits os the select system call. Asyncore did have a way of working with the poll system call, but unfortunately OneFS does not support this call.

The new transport module works with select, poll, and kqueue and should be able to deliver very good concurrency.

The refactoring also made the entire connection process asynchronous, so, if you structure your code properly with a good callback system, you should be able to get really good concurrency/parallelism. Today is garage day, so I'll try to work up an example of a completely async pike stress test for you to reference

[isi-mfurer]

I had to do some minor refactoring to pike itself to make everything nice, but you can see that work in my branch "concurrent-pike"

master...isi-mfurer:concurrent-pike

Specifically, take a look at how I structured my test in https://github.com/isi-mfurer/pike/blob/97c77d03c02a002e76937e2020bbb9f0451ccfe6/pike/test/concurrent.py. I use a class to store all of my callback functions and various state that I need to keep track of. Then I send the first command to connect to the server, but each command after that is called when the previous command completed via use of the Future.then() callback function. The "Client" will continue on its own until meeting the stopping conditions -- in the case of my test a timeout.

[dinebhar]

Thank you so much. that is really a great help.