miniupnp.tuxfamily.org Forum Index miniupnp.tuxfamily.org
The forum about miniupnp and libnatpmp
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Let python module interact with GIL to prevent freezing

 
Post new topic   Reply to topic    miniupnp.tuxfamily.org Forum Index -> miniupnpc Feature Request
View previous topic :: View next topic  
Author Message
quinox



Joined: 28 Sep 2009
Posts: 5

PostPosted: Mon Sep 28, 2009 12:54 pm    Post subject: Let python module interact with GIL to prevent freezing Reply with quote

<Post intentionally left blank>
Back to top
View user's profile Send private message
quinox



Joined: 28 Sep 2009
Posts: 5

PostPosted: Mon Sep 28, 2009 12:55 pm    Post subject: Reply with quote

First of all I'd like to express my compliments, the MiniUPNP project is a real boon Smile

Now my feature request:

I'd like to use MiniUPnP from a pure-python p2p application. The python module as distributed with 1.3/20090921 works like a charm, except that it doesn't seem to interact with the python GIL. The GIL is the Global Interpreter Lock, python's way of managing what threads are allowed to run (interact with objects etc.) This is a problem for me since without releasing the GIL Python will not allow any other thread to run, causing the GUI (PyGTK) to freeze up. I modified the testupnpigd.py file to include a separate thread with a stopwatch in it. This thread will try to print the time since startup once every second or so. With the python module from version 20090921 this is the output:

Code:

Using real miniupnpc
Discovering... delay=2000ms
--------- Hello ---------
--------- 10.3383970261 seconds elapsed ---------
34 device(s) detected
connect: Operation now in progress
connect: Operation now in progress
recv: Connection reset by peer
connect: Operation now in progress
connect: Operation now in progress
connect: Operation now in progress
connect: Operation now in progress
recv: Connection reset by peer
connect: Network is unreachable
connect: Network is unreachable
connect: Operation now in progress
local ip address :
129.125.101.92
Exception : Miniupnpc Unknown Error
--------- 36.8716480732 seconds elapsed ---------
--------- Byebye (total duration: 37.8722250462 seconds) ---------


(output slightly modified, the "34 devices" was really split over two lines due to the timer thread)

With a probe of 2 seconds the module will take the GIL hostage for 10 seconds with the discover() phase, then another 26 seconds while it's in the selectigd() phase. When I use a fake stub module with precisely the same code the output is as follows:

Code:

Using fake miniupnpc
Discovering... delay=2000ms
--------- Hello ---------
--------- 0.000127077102661 seconds elapsed ---------
--------- 1.0012421608 seconds elapsed ---------
42 device(s) detected
connect: Operation now in progress
--------- 2.00238704681 seconds elapsed ---------
--------- 3.0035071373 seconds elapsed ---------
connect: Operation now in progress
--------- 4.00465202332 seconds elapsed ---------
recv: Connection reset by peer
--------- 5.00579404831 seconds elapsed ---------
connect: Operation now in progress
--------- 6.0068731308 seconds elapsed ---------
--------- 7.00715899467 seconds elapsed ---------
connect: Operation now in progress
--------- 8.00809597969 seconds elapsed ---------
connect: Operation now in progress
--------- 9.00917696953 seconds elapsed ---------
connect: Operation now in progress
--------- 10.0102601051 seconds elapsed ---------
--------- 11.0113451481 seconds elapsed ---------
recv: Connection reset by peer
--------- 12.0118570328 seconds elapsed ---------
connect: Network is unreachable
--------- 13.0129461288 seconds elapsed ---------
connect: Network is unreachable
--------- 14.0140321255 seconds elapsed ---------
--------- 15.0151259899 seconds elapsed ---------
connect: Operation now in progress
--------- 16.0162169933 seconds elapsed ---------
local ip address :
::1
Exception : Dummy doesn't do external
--------- Byebye (total duration: 17.0173101425 seconds) ---------


This is the behaviour that I hope to get from the python module. I've looked at the miniupnpcmodule.c code but I couldn't figure out where to start hacking. Documents that might useful for this feature requests are:

* PEP311, Simplified Global Interpreter Lock Acquisition for Extensions http://www.python.org/dev/peps/pep-0311/
* Python/C API Reference Manual http://docs.python.org/c-api/index.html

The modified testupnpigd.py file I used to get the given output can be found at http://129.125.101.92/tmp/testupnpigd.py (stub module at http://129.125.101.92/tmp/fakeminiupnpc.py)

PS1: The 34 UPnP devices on my network are real. While this is probably not a normal situation it shouldn't freeze up Python if possible
PS2: The "Network is unreachable" messages come from the fact that many devices return a LOCATION value with an IP address belonging to the network behind the device, outside of my reach.
Back to top
View user's profile Send private message
miniupnp
Site Admin


Joined: 14 Apr 2007
Posts: 1452

PostPosted: Sat Oct 10, 2009 7:18 pm    Post subject: Reply with quote

I would have a look at this.
But I'm not a python developper so...
_________________
Main miniUPnP author.
http://miniupnp.tuxfamily.org/
Back to top
View user's profile Send private message Visit poster's website
quinox



Joined: 28 Sep 2009
Posts: 5

PostPosted: Thu Oct 15, 2009 6:54 pm    Post subject: Reply with quote

thanks for trying - if you need someone to test it let me know Smile

would/will?

I'm no C-programmer alas. There is no python code involved in this feature request though - If I understand it correctly you have to call certain functions when you know for sure your module is not going to interact with python objects, and then python does the rest
Back to top
View user's profile Send private message
miniupnp
Site Admin


Joined: 14 Apr 2007
Posts: 1452

PostPosted: Fri Oct 16, 2009 9:09 am    Post subject: Reply with quote

you can test with miniupnpc-20091016.tar.gz to see if the problem is fixed.
I added Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around blocking operations following explanations I found on http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock.
_________________
Main miniUPnP author.
http://miniupnp.tuxfamily.org/
Back to top
View user's profile Send private message Visit poster's website
quinox



Joined: 28 Sep 2009
Posts: 5

PostPosted: Fri Oct 16, 2009 9:18 am    Post subject: Reply with quote

miniupnp wrote:
you can test with miniupnpc-20091016.tar.gz to see if the problem is fixed.



Just ran it with my test script and it looks great. The discovery phase no longer freezes up other python threads, so that part is perfect. Later today I'll do more thorough testing when all of the UPnP devices are online, but I think you nailed it. Thanks!
Back to top
View user's profile Send private message
quinox



Joined: 28 Sep 2009
Posts: 5

PostPosted: Fri Oct 16, 2009 3:53 pm    Post subject: Reply with quote

Tested it more extensively and it seems to work perfectly, thanks a bunch!
Back to top
View user's profile Send private message
kans



Joined: 06 Jan 2010
Posts: 1

PostPosted: Wed Jan 06, 2010 3:21 pm    Post subject: Crashes in windows 7/64 with python 2.5 when using threads Reply with quote

I compiled the python extension module which seems to run fine apart from crashing the python process when I call discover inside a thread- the other methods seem to be OK otherwise- threading or not. Does anyone have any thoughts on what could be going on?
Back to top
View user's profile Send private message
miniupnp
Site Admin


Joined: 14 Apr 2007
Posts: 1452

PostPosted: Thu Jan 14, 2010 3:29 am    Post subject: Re: Crashes in windows 7/64 with python 2.5 when using threa Reply with quote

kans wrote:
I compiled the python extension module which seems to run fine apart from crashing the python process when I call discover inside a thread- the other methods seem to be OK otherwise- threading or not. Does anyone have any thoughts on what could be going on?

Could you post a code sample that trigger the crash ?
_________________
Main miniUPnP author.
http://miniupnp.tuxfamily.org/
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    miniupnp.tuxfamily.org Forum Index -> miniupnpc Feature Request All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group
Protected by Anti-Spam ACP
© 2007 Thomas Bernard, author of MiniUPNP.