如何將在線GAE數據存儲區中的數據加載到本地開發服務器中？

Question

我以前使用過GAE docs中描述的方法來下載我實體在實時數據存儲上的備份。

目前，我有每個實體類型的CSV文件，我得到了寫bulkloader.yaml和使用這個命令：

appcfg.py download_data --config_file=bulkloader.yaml --filename=users.csv --kind=Permission --url=http://your_app_id.appspot.com/_ah/remote_api 

我也有，我得到了使用命令SQL3轉儲文件：

appcfg.py download_data --kind=<kind> --url=http://your_app_id.appspot.com/_ah/remote_api --filename=<data-filename> 

現在，如果我試試這個命令：

appcfg.py upload_data --url=http://your_app_id.appspot.com/_ah/remote_api --kind=<kind> --filename=<data-filename> 

地方更換網址主機：8080，它要求我輸入用戶名/密碼。現在，即使在http://localhost:8080/_ah/remote_api中提供模擬用戶名（test@example.com），並選中「admin」複選框，它總是會給我一個驗證錯誤。

在文檔中提到的另一種方法是使用這樣的：

appcfg.py upload_data --config_file=album_loader.py --filename=album_data.csv --kind=Album --url=http://localhost:8080/_ah/remote_api <app-directory> 

我寫了裝載機，並嘗試過了，它也要求輸入用戶名和密碼，但在這裏接受什麼。輸出如下：

/usr/local/google_appengine/google/appengine/api/search/search.py:232: UserWarning: DocumentOperationResult._code is deprecated. Use OperationResult._code instead. 
    'Use OperationResult.%s instead.' % (name, name)) 
/usr/local/google_appengine/google/appengine/api/search/search.py:232: UserWarning: DocumentOperationResult._CODES is deprecated. Use OperationResult._CODES instead. 
    'Use OperationResult.%s instead.' % (name, name)) 
Application: knowledgetestgame 
Uploading data records. 
[INFO ] Logging to bulkloader-log-20121113.210613 
[INFO ] Throttling transfers: 
[INFO ] Bandwidth: 250000 bytes/second 
[INFO ] HTTP connections: 8/second 
[INFO ] Entities inserted/fetched/modified: 20/second 
[INFO ] Batch Size: 10 
[INFO ] Opening database: bulkloader-progress-20121113.210613.sql3 
Please enter login credentials for localhost 
Email: test@example.com 
Password for test@example.com: 
[INFO ] Connecting to localhost:8080/_ah/remote_api 
[INFO ] Starting import; maximum 10 entities per post 
[ERROR ] [WorkerThread-4] WorkerThread: 
Traceback (most recent call last): 
    File "/usr/local/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 176, in WorkOnItems 
    status, instruction = item.PerformWork(self.__thread_pool) 
    File "/usr/local/google_appengine/google/appengine/tools/bulkloader.py", line 764, in PerformWork 
    transfer_time = self._TransferItem(thread_pool) 
    File "/usr/local/google_appengine/google/appengine/tools/bulkloader.py", line 933, in _TransferItem 
    self.content = self.request_manager.EncodeContent(self.rows) 
    File "/usr/local/google_appengine/google/appengine/tools/bulkloader.py", line 1394, in EncodeContent 
    entity = loader.create_entity(values, key_name=key, parent=parent) 
    File "/usr/local/google_appengine/google/appengine/tools/bulkloader.py", line 2728, in create_entity 
    (len(self.__properties), len(values))) 
AssertionError: Expected 17 columns, found 18. 
[INFO ] [WorkerThread-5] Backing off due to errors: 1.0 seconds 
[INFO ] Unexpected thread death: WorkerThread-4 
[INFO ] An error occurred. Shutting down... 
[ERROR ] Error in WorkerThread-4: Expected 17 columns, found 18. 

[INFO ] 980 entities total, 0 previously transferred 
[INFO ] 0 entities (278 bytes) transferred in 5.9 seconds 
[INFO ] Some entities not successfully transferred 

我總共有〜4000個單位，在這裏說，980被轉移，但實際上我檢查本地數據存儲，我覺得他們都不..

下面是我用的裝載機（我用NDB的guess實體）

import datetime 
from google.appengine.ext import db 
from google.appengine.tools import bulkloader 
from google.appengine.ext.ndb import key 


class Guess(db.Model): 
    pass 

class GuessLoader(bulkloader.Loader): 
    def __init__(self): 
     bulkloader.Loader.__init__(self, 'Guess', 
            [('selectedAssociation', lambda x: x.decode('utf-8')), 
            ('suggestionsList', lambda x: x.decode('utf-8')), 
            ('associationIndexInList', int),          
            ('timeEntered', 
            lambda x: datetime.datetime.strptime(x, '%m/%d/%Y').date()), 
            ('rank', int), 
            ('topicName', lambda x: x.decode('utf-8')), 
            ('topic', int), 
            ('player', int), 
            ('game', int), 
            ('guessString', lambda x: x.decode('utf-8')), 
            ('guessTime', 
            lambda x: datetime.datetime.strptime(x, '%m/%d/%Y').date()), 
            ('accountType', lambda x: x.decode('utf-8')), 
            ('nthGuess', int), 
            ('score', float), 
            ('cutByRoundEnd', bool), 
            ('suggestionsListDelay', int), 
            ('occurrences', float) 
            ]) 

loaders = [GuessLoader] 

編輯：我只注意到這部分錯誤消息[ERROR ] Error in WorkerThread-0: Expected 17 columns, found 18.，而實際上我只是通過整個CSV文件中去，並確保每行有18列。我檢查了加載程序，發現我錯過了key列，我給了它一個類型int，但這不起作用。

Answer 1

試試只按Enter（沒有用戶名/密碼）。這似乎對我來說是訣竅。我的命令（包裹在一個bash腳本，以防止我偶爾收到導入錯誤）是：

#!/bin/bash 

# Modify path 
export PYTHONPATH=$PYTHONPATH:. 

# Load data 
python /path/to/app/config/appcfg.py upload_data \ 
--config_file=<my_loader.py> \ 
--filename=<output.csv> \ 
--kind=<kind> \ 
--application=dev~<application_id> \ 
--url=http://localhost:8088/_ah/remote_api ./ 

當提示Email，我按下回車鍵，所有上傳到開發服務器。在這種情況下，我沒有使用NDB，但我不相信這會有所作爲。

Answer 2

如果您有與認證問題，把你的appengine_config.py如下：

if os.environ.get('SERVER_SOFTWARE','').startswith('Development'): 
    remoteapi_CUSTOM_ENVIRONMENT_AUTHENTICATION = (
    'REMOTE_ADDR', ['127.0.0.1']) 

然後運行

appcfg.py download_data --url=http://APPNAME.appspot.com/_ah/remote_api --filename=dump --kind=EntityName 
appcfg.py upload_data --url=http://localhost:8080/_ah/remote_api --filename=dump --application=dev~APPNAME