爲什麼我不能在數據庫中寫入數據框？

我有32 GB RAM，我使用jupyter和熊貓。我的數據幀不是很大，但是當我想它寫在北極的數據的基礎上我有「的MemoryError」：爲什麼我不能在數據庫中寫入數據框？

而且我希望把它寫：

from arctic import Arctic 
import arctic as arc 
store = Arctic('.....') 
lib = store['myLib'] 
lib.write('quotes', df_q)

MemoryError Traceback (most recent call last) in() 1 memory(df_q) ----> 2 lib.write('quotes', df_q)

/usr/local/lib/python2.7/dist-packages/arctic/decorators.pyc in f_retry(*args, **kwargs) 48 while True: 49 try: ---> 50 return f(*args, **kwargs) 51 except (DuplicateKeyError, ServerSelectionTimeoutError) as e: 52 # Re-raise errors that won't go away.

/usr/local/lib/python2.7/dist-packages/arctic/store/version_store.pyc in write(self, symbol, data, metadata, prune_previous_version, **kwargs) 561 562 handler = self._write_handler(version, symbol, data, **kwargs) --> 563 mongo_retry(handler.write)(self._arctic_lib, version, symbol, data, previous_version, **kwargs) 564 565 # Insert the new version into the version DB

/usr/local/lib/python2.7/dist-packages/arctic/decorators.pyc in f_retry(*args, **kwargs) 48 while True: 49 try: ---> 50 return f(*args, **kwargs) 51 except (DuplicateKeyError, ServerSelectionTimeoutError) as e: 52 # Re-raise errors that won't go away.

/usr/local/lib/python2.7/dist-packages/arctic/store/_pandas_ndarray_store.pyc in write(self, arctic_lib, version, symbol, item, previous_version) 301 def write(self, arctic_lib, version, symbol, item, previous_version): 302 item, md = self.to_records(item) --> 303 super(PandasDataFrameStore, self).write(arctic_lib, version, symbol, item, previous_version, dtype=md) 304 305 def append(self, arctic_lib, version, symbol, item, previous_version):

/usr/local/lib/python2.7/dist-packages/arctic/store/_ndarray_store.pyc in write(self, arctic_lib, version, symbol, item, previous_version, dtype) 385 version['type'] = self.TYPE 386 version['up_to'] = len(item) --> 387 version['sha'] = self.checksum(item) 388 389 if previous_version:

/usr/local/lib/python2.7/dist-packages/arctic/store/_ndarray_store.pyc in checksum(self, item) 370 def checksum(self, item): 371 sha = hashlib.sha1() --> 372 sha.update(item.tostring()) 373 return Binary(sha.digest()) 374

MemoryError:

WTF ？如果我使用df_q.to_csv（）我會等上好幾年....

來源

2016-07-11 ooolllooo

你是否嘗試在[chunks]（https://github.com/manahl/arctic/blob/master/arctic/chunkstore/chunkstore.py）中編寫DF？ – MaxU

您的問題實際上不是內存問題。如果你讀了你的錯誤，看起來你的程序庫在訪問你的數據時遇到了問題......

第一個錯誤：說你的服務器超時了。（ServerSelectionTimeoutError）

第二個錯誤：嘗試更新MongoDB版本。

第三錯誤：重試訪問您的服務器，失敗（ServerSelectionTimeoutError）

等，所以基本上你的問題就出在北極包本身（見最後一個錯誤是一個校驗和錯誤）。您也可以從df_q.to_csv()工作的事實推斷出這一點，但由於它不像Artic那樣進行優化，因此它非常緩慢。我建議嘗試重新安裝北極包

來源

2016-07-11 19:21:45

爲什麼我不能在數據庫中寫入數據框？

回答

相關問題