如何從boost線程調用Python？

Question

我有一個Python應用程序調用C++ boost python庫，它都可以工作。不過，我有一個回調C++的Python場景，其中來自boost線程的C++調用python，並在C++端獲得訪問衝突。如果我使用python線程完成相同的回調，它可以完美地工作。因此，我懷疑我不能使用boost線程從C++中簡單地回調Python，但需要額外的工作才能使用它。

Answer 1

最有可能的罪魁禍首是Global Interpreter Lock（GIL）在調用Python代碼時未被線程佔用，導致未定義的行爲。驗證所有直接或間接Python調用的路徑，在調用Python代碼之前獲取GIL。

---

GIL是圍繞CPython解釋器的互斥鎖。這個互斥體阻止了在Python對象上執行並行操作。因此，在任何時候，允許最多一個線程（獲得GIL的線程）對Python對象執行操作。當存在多個線程時，調用Python代碼而不保留GIL會導致未定義的行爲。

C或C++線程有時在Python文檔中被稱爲外來線程。 Python解釋器無法控制外來線程。因此，外來線程負責管理GIL以允許與Python線程並行或並行執行。必須仔細考慮：

• 堆棧展開，因爲Boost.Python可能會引發異常。
• 間接調用到Python，如拷貝構造函數和析構函數

一種解決方案是包裝的Python回調與知道GIL管理的自定義類型。

---

使用RAII-style類來管理GIL提供了一個優雅的異常安全解決方案。例如，使用以下with_gil類，創建對象with_gil時，調用線程將獲取GIL。當with_gil對象被破壞時，它恢復GIL狀態。

/// @brief Guard that will acquire the GIL upon construction, and 
///  restore its state upon destruction. 
class with_gil 
{ 
public: 
    with_gil() { state_ = PyGILState_Ensure(); } 
    ~with_gil() { PyGILState_Release(state_); } 

    with_gil(const with_gil&)   = delete; 
    with_gil& operator=(const with_gil&) = delete; 
private: 
    PyGILState_STATE state_; 
}; 

及其用法：

{ 
    with_gil gil;      // Acquire GIL. 
    // perform Python calls, may throw 
}         // Restore GIL. 

與正在能夠通​​過with_gil管理GIL，下一步是創建正確管理GIL函子。下面py_callable類將包裝一個boost::python::object並獲取GIL對於其中Python代碼被調用的所有路徑：

/// @brief Helper type that will manage the GIL for a python callback. 
/// 
/// @detail GIL management: 
///   * Acquire the GIL when copying the `boost::python` object 
///   * The newly constructed `python::object` will be managed 
///    by a `shared_ptr`. Thus, it may be copied without owning 
///    the GIL. However, a custom deleter will acquire the 
///    GIL during deletion 
///   * When `py_callable` is invoked (operator()), it will acquire 
///    the GIL then delegate to the managed `python::object` 
class py_callable 
{ 
public: 

    /// @brief Constructor that assumes the caller has the GIL locked. 
    py_callable(const boost::python::object& object) 
    { 
    with_gil gil; 
    object_.reset(
     // GIL locked, so it is safe to copy. 
     new boost::python::object{object}, 
     // Use a custom deleter to hold GIL when the object is deleted. 
     [](boost::python::object* object) 
     { 
     with_gil gil; 
     delete object; 
     }); 
    } 

    // Use default copy-constructor and assignment-operator. 
    py_callable(const py_callable&) = default; 
    py_callable& operator=(const py_callable&) = default; 

    template <typename ...Args> 
    void operator()(Args... args) 
    { 
    // Lock the GIL as the python object is going to be invoked. 
    with_gil gil; 
    (*object_)(std::forward<Args>(args)...); 
    } 

private: 
    std::shared_ptr<boost::python::object> object_; 
}; 

通過在自由空間管理boost::python::object，可以自由複製該shared_ptr而不必保持GIL。這允許我們安全地使用默認生成的拷貝構造函數，賦值運算符，析構函數等。

一個將使用py_callable如下：

// thread 1 
boost::python::object object = ...; // GIL must be held. 
py_callable callback(object);  // GIL no longer required. 
work_queue.post(callback); 

// thread 2 
auto callback = work_queue.pop(); // GIL not required. 
// Invoke the callback. If callback is `py_callable`, then it will 
// acquire the GIL, invoke the wrapped `object`, then release the GIL. 
callback(...); 

---

下面是一個完整的例子具有Python擴展調用Python對象從一個C++線程回調demonstrating：

#include <memory> // std::shared_ptr 
#include <thread> // std::this_thread, std::thread 
#include <utility> // std::forward 
#include <boost/python.hpp> 

/// @brief Guard that will acquire the GIL upon construction, and 
///  restore its state upon destruction. 
class with_gil 
{ 
public: 
    with_gil() { state_ = PyGILState_Ensure(); } 
    ~with_gil() { PyGILState_Release(state_); } 

    with_gil(const with_gil&)   = delete; 
    with_gil& operator=(const with_gil&) = delete; 
private: 
    PyGILState_STATE state_; 
}; 

/// @brief Helper type that will manage the GIL for a python callback. 
/// 
/// @detail GIL management: 
///   * Acquire the GIL when copying the `boost::python` object 
///   * The newly constructed `python::object` will be managed 
///    by a `shared_ptr`. Thus, it may be copied without owning 
///    the GIL. However, a custom deleter will acquire the 
///    GIL during deletion 
///   * When `py_callable` is invoked (operator()), it will acquire 
///    the GIL then delegate to the managed `python::object` 
class py_callable 
{ 
public: 

    /// @brief Constructor that assumes the caller has the GIL locked. 
    py_callable(const boost::python::object& object) 
    { 
    with_gil gil; 
    object_.reset(
     // GIL locked, so it is safe to copy. 
     new boost::python::object{object}, 
     // Use a custom deleter to hold GIL when the object is deleted. 
     [](boost::python::object* object) 
     { 
     with_gil gil; 
     delete object; 
     }); 
    } 

    // Use default copy-constructor and assignment-operator. 
    py_callable(const py_callable&) = default; 
    py_callable& operator=(const py_callable&) = default; 

    template <typename ...Args> 
    void operator()(Args... args) 
    { 
    // Lock the GIL as the python object is going to be invoked. 
    with_gil gil; 
    (*object_)(std::forward<Args>(args)...); 
    } 

private: 
    std::shared_ptr<boost::python::object> object_; 
}; 

BOOST_PYTHON_MODULE(example) 
{ 
    // Force the GIL to be created and initialized. The current caller will 
    // own the GIL. 
    PyEval_InitThreads(); 

    namespace python = boost::python; 
    python::def("call_later", 
    +[](int delay, python::object object) { 
     // Create a thread that will invoke the callback. 
     std::thread thread(+[](int delay, py_callable callback) { 
     std::this_thread::sleep_for(std::chrono::seconds(delay)); 
     callback("spam"); 
     }, delay, py_callable{object}); 
     // Detach from the thread, allowing caller to return. 
     thread.detach(); 
    }); 
} 

互動用法：

>>> import time 
>>> import example 
>>> def shout(message): 
...  print message.upper() 
... 
>>> example.call_later(1, shout) 
>>> print "sleeping"; time.sleep(3); print "done sleeping" 
sleeping 
SPAM 
done sleeping