使用線程時Django中的數據庫錯誤

Question

我正在使用需要查詢PostgreSQL數據庫的Django Web應用程序。當使用Python threading接口實現併發時，我得到DoesNotExist錯誤的查詢項目。當然，順序執行查詢時不會發生這些錯誤。

讓我告訴一個單元測試，我寫了證明意外的行爲：

class ThreadingTest(TestCase): 
    fixtures = ['demo_city',] 

    def test_sequential_requests(self): 
     """ 
     A very simple request to database, made sequentially. 

     A fixture for the cities has been loaded above. It is supposed to be 
     six cities in the testing database now. We will made a request for 
     each one of the cities sequentially. 
     """ 
     for number in range(1, 7): 
      c = City.objects.get(pk=number) 
      self.assertEqual(c.pk, number) 

    def test_threaded_requests(self): 
     """ 
     Now, to test the threaded behavior, we will spawn a thread for 
     retrieving each city from the database. 
     """ 

     threads = [] 
     cities = [] 

     def do_requests(number): 
      cities.append(City.objects.get(pk=number)) 

     [threads.append(threading.Thread(target=do_requests, args=(n,))) for n in range(1, 7)] 

     [t.start() for t in threads] 
     [t.join() for t in threads] 

     self.assertNotEqual(cities, []) 

正如你所看到的，第一個測試順序執行一些數據庫請求，這確實是沒有問題的工作。然而，第二個測試執行完全相同的請求，但每個請求都在一個線程中產生。這實際上是失敗的，返回一個DoesNotExist異常。

的這個單元測試執行的輸出是這樣的：

test_sequential_requests (cesta.core.tests.threadbase.ThreadingTest) ... ok 
test_threaded_requests (cesta.core.tests.threadbase.ThreadingTest) ... 

Exception in thread Thread-1: 
Traceback (most recent call last): 
    File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner 
    self.run() 
    File "/usr/lib/python2.6/threading.py", line 484, in run 
    self.__target(*self.__args, **self.__kwargs) 
    File "/home/jose/Work/cesta/trunk/src/cesta/core/tests/threadbase.py", line 45, in do_requests 
    cities.append(City.objects.get(pk=number)) 
    File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/manager.py", line 132, in get 
    return self.get_query_set().get(*args, **kwargs) 
    File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/query.py", line 349, in get 
    % self.model._meta.object_name) 
DoesNotExist: City matching query does not exist. 

...其他線程返回一個類似的輸出...

Exception in thread Thread-6: 
Traceback (most recent call last): 
    File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner 
    self.run() 
    File "/usr/lib/python2.6/threading.py", line 484, in run 
    self.__target(*self.__args, **self.__kwargs) 
    File "/home/jose/Work/cesta/trunk/src/cesta/core/tests/threadbase.py", line 45, in do_requests 
    cities.append(City.objects.get(pk=number)) 
    File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/manager.py", line 132, in get 
    return self.get_query_set().get(*args, **kwargs) 
    File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/query.py", line 349, in get 
    % self.model._meta.object_name) 
DoesNotExist: City matching query does not exist. 


FAIL 

====================================================================== 
FAIL: test_threaded_requests (cesta.core.tests.threadbase.ThreadingTest) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
    File "/home/jose/Work/cesta/trunk/src/cesta/core/tests/threadbase.py", line 52, in test_threaded_requests 
    self.assertNotEqual(cities, []) 
AssertionError: [] == [] 

---------------------------------------------------------------------- 
Ran 2 tests in 0.278s 

FAILED (failures=1) 
Destroying test database for alias 'default' ('test_cesta')... 

請記住，這一切都是發生在PostgreSQL數據庫中，該數據庫應該是線程安全的，而不是SQLite或類似的。測試也使用PostgreSQL進行。

在這一點上，我完全失去了什麼可以失敗。任何想法或建議？

謝謝！編輯：

編輯：我寫了一個小視圖，只是爲了檢查它是否能用於測試。下面是這個視圖的代碼：

def get_cities(request): 
    queue = Queue.Queue() 

    def get_async_cities(q, n): 
     city = City.objects.get(pk=n) 
     q.put(city) 

    threads = [threading.Thread(target=get_async_cities, args=(queue, number)) for number in range(1, 5)] 

    [t.start() for t in threads] 
    [t.join() for t in threads] 

    cities = list() 

    while not queue.empty(): 
     cities.append(queue.get()) 

    return render_to_response('async/cities.html', {'cities': cities}, 
     context_instance=RequestContext(request)) 

（請不要考慮寫視圖代碼中的應用程序邏輯的愚蠢記住，這只是一個概念證明，並不會是永遠在真實應用程序中）

結果是代碼運行良好，請求在線程中成功完成，視圖最終在調用其URL後顯示城市。

因此，我認爲使用線程進行查詢只會在您需要測試代碼時出現問題。在生產中，它將毫無問題地工作。

任何有用的建議，測試這種類型的代碼成功？

Answer 1

這聽起來像是交易問題。如果您在當前請求（或測試）中創建元素，那麼它們幾乎可以肯定處於未提交的事務中，而該事務無法通過另一個線程中的單獨連接進行訪問。您可能需要manage your transctions manually才能使其發揮作用。

Answer 2

嘗試使用TransactionTestCase：

class ThreadingTest(TransactionTestCase): 

TestCase的保持內存中的數據，並沒有發出COMMIT數據庫。可能線程試圖直接連接到數據庫，而數據還沒有被提交。Seedescription這裏： https://docs.djangoproject.com/en/dev/topics/testing/?from=olddocs#django.test.TransactionTestCase

TransactionTestCase和測試用例是除了方式 ，其中，數據庫被複位到一個已知的狀態，並測試提交和回滾的影響的能力 測試代碼相同。 A TransactionTestCase在測試運行前重置數據庫 截斷所有表並重新加載初始數據。 TransactionTestCase可能會調用commit和rollback，並觀察這些調用在數據庫上的效果。

Answer 3

成爲與文檔

class LiveServerTestCase(TransactionTestCase): 
    """ 
    ... 
    Note that it inherits from TransactionTestCase instead of TestCase because 
    the threads do not share the same transactions (unless if using in-memory 
    sqlite) and each thread needs to commit all their transactions so that the 
    other thread can see the changes. 
    """ 

現在，該交易尚未提交一個TestCase內的這部分更加清晰，因此變化並不對其他線程可見。