App Engine的批量數據上傳URLError：

我試圖使用appcfg.pyApp Engine的批量數據上傳URLError：

只有少數的記錄上傳一些記錄，以我的本地數據存儲「連接被對方復位」實際上是插入，我也得到了以下輸出（帶有大量錯誤）：

 $ appcfg.py upload_data --config_file=bulkloader.yaml --filename=/output.csv --kind=AutoCompleteIndex --url=http://localhost:8084/remote_api . 
     Application: theapp1; version: dev. 
     Uploading data records. 
     [INFO ] Logging to bulkloader-log-20100808.170223 
     [INFO ] Throttling transfers: 
     [INFO ] Bandwidth: 250000 bytes/second 
     [INFO ] HTTP connections: 8/second 
     [INFO ] Entities inserted/fetched/modified: 20/second 
     [INFO ] Batch Size: 10 
     [INFO ] Opening database: bulkloader-progress-20100808.170223.sql3 
     Please enter login credentials for localhost 
     Email: kjuyg 
     Password for kjuyg: 
     [INFO ] Connecting to localhost:8084/remote_api 
     [INFO ] Starting import; maximum 10 entities per post 
     .......................................[ERROR ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer 
     Traceback (most recent call last): 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork 
      transfer_time = self._TransferItem(thread_pool) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem 
      self.request_manager.PostEntities(self.content) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities 
      datastore.Put(entities) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put 
      'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall 
      rpc.check_success() 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success 
      self.__rpc.CheckSuccess() 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl 
      self.request, self.response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall 
      handler(request, response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put 
      'datastore_v3', 'Put', put_request, put_response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall 
      self._MakeRealSyncCall(service, call, request, response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall 
      encoded_response = self._server.Send(self._path, encoded_request) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send 
      f = self.opener.open(req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open 
      response = self._open(req, data) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open 
      '_open', req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain 
      result = func(*args) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open 
      return self.do_open(httplib.HTTPConnection, req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open 
      raise URLError(err) 
     URLError: <urlopen error [Errno 54] Connection reset by peer> 
     ................................................[ERROR ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer 
     Traceback (most recent call last): 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork 
      transfer_time = self._TransferItem(thread_pool) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem 
      self.request_manager.PostEntities(self.content) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities 
      datastore.Put(entities) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put 
      'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall 
      rpc.check_success() 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success 
      self.__rpc.CheckSuccess() 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl 
      self.request, self.response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall 
      handler(request, response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put 
      'datastore_v3', 'Put', put_request, put_response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall 
      self._MakeRealSyncCall(service, call, request, response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall 
      encoded_response = self._server.Send(self._path, encoded_request) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send 
      f = self.opener.open(req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open 
      response = self._open(req, data) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open 
      '_open', req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain 
      result = func(*args) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open 
      return self.do_open(httplib.HTTPConnection, req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open 
      raise URLError(err) 
     URLError: <urlopen error [Errno 54] Connection reset by peer> 
     .............[ERROR ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer 
     Traceback (most recent call last): 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork 
      transfer_time = self._TransferItem(thread_pool) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem 
      self.request_manager.PostEntities(self.content) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities 
      datastore.Put(entities) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put 
      'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall 
      rpc.check_success() 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success 
      self.__rpc.CheckSuccess() 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl 
      self.request, self.response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall 
      handler(request, response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put 
      'datastore_v3', 'Put', put_request, put_response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall 
      self._MakeRealSyncCall(service, call, request, response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall 
      encoded_response = self._server.Send(self._path, encoded_request) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send 
      f = self.opener.open(req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open 
      response = self._open(req, data) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open 
      '_open', req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain 
      result = func(*args) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open 
      return self.do_open(httplib.HTTPConnection, req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open 
      raise URLError(err) 
     URLError: <urlopen error [Errno 54] Connection reset by peer> 
     ................[ERROR ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer 
     Traceback (most recent call last): 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork 
      transfer_time = self._TransferItem(thread_pool) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem 
      self.request_manager.PostEntities(self.content) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities 
      datastore.Put(entities) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put 
      'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall 
      rpc.check_success() 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success 
      self.__rpc.CheckSuccess() 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl 
      self.request, self.response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall 
      handler(request, response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put 
      'datastore_v3', 'Put', put_request, put_response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall 
      self._MakeRealSyncCall(service, call, request, response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall 
      encoded_response = self._server.Send(self._path, encoded_request) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send 
      f = self.opener.open(req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open 
      response = self._open(req, data) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open 
      '_open', req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain 
      result = func(*args) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open 
      return self.do_open(httplib.HTTPConnection, req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open 
      raise URLError(err) 
     URLError: <urlopen error [Errno 54] Connection reset by peer> 
     .......................................................................................................................................................................[ERROR ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer 
     Traceback (most recent call last): 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork 
      transfer_time = self._TransferItem(thread_pool) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem 
      self.request_manager.PostEntities(self.content) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities 
      datastore.Put(entities) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put 
      'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall 
      rpc.check_success() 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success 
      self.__rpc.CheckSuccess() 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl 
      self.request, self.response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall 
      handler(request, response) 
      File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put 

.... etc ..... 
      return self.do_open(httplib.HTTPConnection, req) 
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open 
      raise URLError(err) 
     URLError: <urlopen error [Errno 54] Connection reset by peer> 
     .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C..[INFO ] An error occurred. Shutting down... 
     .......... 
     [INFO ] 19580 entites total, 0 previously transferred 
     [INFO ] 18570 entities (3593152 bytes) transferred in 225.1 seconds 
     [INFO ] Some entities not successfully transferred

任何想法可能是什麼問題？

來源

2010-08-08 Matthew H

看看你的服務器日誌。你是否得到超過堆大小或OutOfMemmory異常？如果您的上傳量很大，則會在本地開發服務器上發生，因爲開發服務器會將數據存儲保留在主內存中。服務器會掛起一段時間，然後你會重置連接。由於使用大量額外的RAM來組織和存儲元數據，我已經讓開發服務器的內存耗盡了10MB的數據存儲數據。您可以使用-Xmx開關增加堆的大小。

來源

2010-08-09 03:46:31

我們應該在哪裏設置-Xmx開關？ – 2011-07-06 01:10:33

我在我的Mac上有完全相同的問題。我解決它通過創建一個代理，通過把在我的httpd-vhost.conf文件中的以下（在等/ apache2的/額外/）：

<Location /remote_api> 
    ProxyPass http://localhost:8888/remote_api 
</Location>

提供您的測試服務器運行在端口8888

然後我運行：

sudo apachectl restart

重新啓動apache服務器。

現在，這個命令（「調查」是數據項中的一個）：

/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/appcfg.py upload_data/
--kind=Survey --file=Survey.csv/
--url=http://localhost/remote_api [email protected]/
--config_file=config.yml

給出了這樣的輸出：

[INFO ] Logging to bulkloader-log-20121114.100837 
[INFO ] Throttling transfers: 
[INFO ] Bandwidth: 250000 bytes/second 
[INFO ] HTTP connections: 8/second 
[INFO ] Entities inserted/fetched/modified: 20/second 
[INFO ] Batch Size: 10 
Password for [email protected]: 
[INFO ] Opening database: bulkloader-progress-20121114.100837.sql3 
[INFO ] Connecting to localhost/remote_api 
[INFO ] Starting import; maximum 10 entities per post 
.... 
[INFO ] 36 entities total, 0 previously transferred 
[INFO ] 36 entities (15402 bytes) transferred in 1.7 seconds 
[INFO ] All entities successfully transferred

因爲它應該是。

來源

2012-11-14 09:12:50 MTWestra

App Engine的批量數據上傳URLError：

回答

相關問題