Dropping GIL when calling Julia from Python (JuliaCall)

[Moelf]

Similar to what Numba's nogil, it's useful for using Julia as a performant backend of Python

[cjdoris]

Repeating what I said in #108

Yeah calling into Julia from Julia doesn't unlock the GIL. This is because the Julia code could well call back into Python, which would require automatically relocking the GIL, which is all quite complicated and probably slow in general. If you can guarantee your Julia code doesn't call back into Python, you can release the GIL yourself. A future version of JuliaCall will allow something like some_julia_function.jl_call_nogil(x, y, z) to make this a bit easier.

Specifically, you can release the GIL yourself in the Julia code you call like:

pythread = PythonCall.C.PyEval_SaveThread()
try
    # code which doesn't touch Python
finally
    PythonCall.C.PyEval_RestoreThread(pythread)
end

[github-actions]

This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.

[PallHaraldsson]

In case relevant, new in 3.12rc1:

https://peps.python.org/pep-0684/

https://docs.python.org/3.12/whatsnew/3.12.html#pep-684-a-per-interpreter-gil

PEP 684: A Per-Interpreter GIL
Sub-interpreters may now be created with a unique GIL per interpreter. This allows Python programs to take full advantage of multiple CPU cores.

Use the new Py_NewInterpreterFromConfig() function to create an interpreter with its own GIL:

PyInterpreterConfig config = {
    .check_multi_interp_extensions = 1,
    .gil = PyInterpreterConfig_OWN_GIL,
};
PyThreadState *tstate = NULL;
PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);
if (PyStatus_Exception(status)) {
    return -1;
}
/* The new interpeter is now active in the current thread. */
For further examples how to use the C-API for sub-interpreters with a per-interpreter GIL, see Modules/_xxsubinterpretersmodule.c.

New release candidate 2 expected 2023-09-04 then

3.12.0 final: Monday, 2023-10-02

See also new in 3.13:

https://peps.python.org/pep-0554/

PEP 554 – Multiple Interpreters in the Stdlib

A Disclaimer about the GIL
To avoid any confusion up front: This PEP is meant to be independent of any efforts to stop sharing the GIL between interpreters (PEP 684). At most this proposal will allow users to take advantage of any GIL-related work.

Copied from: JuliaPy/PyCall.jl#1046 (comment)

[Moelf]

is this closed by #535?

[cjdoris]

Yep thanks.