Bug 18101

Summary: support multi-threading in pycairo
Product: pycairo Reporter: Sebastian Tusk <freedesktop.20.scyt>
Component: generalAssignee: Steve Chaplin <d74n5pohf9>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: medium    
Version: unspecified   
Hardware: Other   
OS: All   
Whiteboard:
i915 platform: i915 features:

Description Sebastian Tusk 2008-10-17 01:03:21 UTC
Python isn't fully capable of multi-threading. There is a thing called global interpreter lock (GIL) that is hold during calls into C-implemented modules. See http://www.python.org/doc/2.4.4/api/threads.html

Because of this every call to a pycairo function blocks all other threads within python. As cairo should be thread-safe every call into the actual cairo library can be surrounded by Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS to fix this issue.
Comment 1 Steve Chaplin 2008-10-17 02:15:31 UTC
My understanding is that you only need to use Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS around sections of code which you know do blocking I/O operations.
For some cairo functions such as cairo_surface_write_to_png it would be useful, but for many others I don't think it would be needed since the pycairo function would execute instantly and then afterwards the Python interpreter could switch to another thread.
Do you know of other specific cairo functions which do blocking I/O and cause problems for other threads?
Comment 2 Sebastian Tusk 2008-10-17 04:18:02 UTC
You want this for every function not only for those with blocking I/O. This is because the python interpreter executes code completely synchronized.

Say you have two threads A and B. Thread A has the GIL and runs. After executing 100 bytecode instruction it releases the GIL and yields to thread B. After that A waits for the GIL to become available again.

B now has the GIL executes 100 instructions. Then releases the GIL and yields to A.

So no python code ever runs at the same time. My understanding is that it would be very hard to change python to be fully capable of multi-threading. So why use threading at all in python? It possible to release the GIL if you can ensure that no two threads touch a python object or any CPython function at the same time. For instance this is the case if you call into a cairo function.

Of course reading and writing image files can take a lot of time but other image operations can take a lot of time too. In my case I create thumbnails for directories full of image files in a worker thread. Unfortunately this starves my main thread and it starves also on repeated runs where all files are already in system memory.
Comment 3 Steve Chaplin 2008-10-21 04:06:25 UTC
Do you have a small test case that you could attach so I could have a look at the problem?
Comment 4 Steve Chaplin 2008-11-24 07:52:51 UTC
I've added
Py_BEGIN_ALLOW_THREADS
Py_END_ALLOW_THREADS
around all the cairo functions I think are possibly blocking or long running.

I'm not convinced that all cairo function calls need to be wrapped with
Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS.

I had a look at some files Python distribution - fileobject.c and
socketmodule.c - not all of the file/socket library functions are wrapped with
BEGIN_ALLOW and END_ALLOW_THREADS, only (I assume) the possibly blocking and
long-running calls.

If you can, test the new code and report back to say whether is fixes the problem.
Comment 5 Thomas Perl 2008-12-28 12:11:59 UTC
These changes seem to be the cause of bug 19287. Please have a look at the bug report there - it's a crasher related to these thread "brackets".
Comment 6 Steve Chaplin 2008-12-31 22:27:19 UTC
I assume that the reported blocking has been alleviated by the addition of the THREAD wrappers. Closing.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.