使用PyMongo將熊貓數據框插入到mongodb

Question

使用PyMongo將最簡單的方法將pandas DataFrame插入到mongodb中是什麼？

嘗試

db.myCollection.insert(df.to_dict()) 

給了一個錯誤InvalidDocument: documents must have only string keys, key was Timestamp('2013-11-23 13:31:00', tz=None)

db.myCollection.insert(df.to_json()) 

給了一個錯誤TypeError: 'str' object does not support item assignment

db.myCollection.insert({id: df.to_json()}) 

給了一個錯誤InvalidDocument: documents must have only string keys, key was <built-in function id>

DF

<class 'pandas.core.frame.DataFrame'> 
DatetimeIndex: 150 entries, 2013-11-23 13:31:26 to 2013-11-23 13:24:07 
Data columns (total 3 columns): 
amount 150 non-null values 
price  150 non-null values 
tid  150 non-null values 
dtypes: float64(2), int64(1) 

Answer 1

我懷疑是有兩個最快和簡單方法。如果你不擔心數據的轉換，你可以做

>>> import json 
>>> df = pd.DataFrame.from_dict({'A': {1: datetime.datetime.now()}}) 
>>> df 
          A 
1 2013-11-23 21:14:34.118531 

>>> records = json.loads(df.T.to_json()).values() 
>>> db.myCollection.insert(records) 

，但如果你試圖load data back，你會得到：

>>> df = read_mongo(db, 'myCollection') 
>>> df 
        A 
0 1385241274118531000 
>>> df.dtypes 
A int64 
dtype: object 

所以你必須將「A」在您的DataFrame中回到datetime s，以及所有不是int,float或str字段。對於這個例子：

>>> df['A'] = pd.to_datetime(df['A']) 
>>> df 
          A 
0 2013-11-23 21:14:34.118531 

Answer 2

這個怎麼樣：

db.myCollection.insert({id: df.to_json()}) 

ID將是該DF的唯一字符串

Answer 3

在這裏你有最快捷的方式。使用pymongo 3的insert_many方法和to_dict方法的'記錄'參數。

db.insert_many(df.to_dict('records')) 

Answer 4

odo可以使用

odo(df, db.myCollection) 

Answer 5

做到這一點如果你的數據幀已經丟失的數據（即無，男），並在文檔中你不想空鍵值：

db.insert_many(df.to_dict("records"))將插入具有空值的鍵。如果你不想在你的文檔中的空鍵值就可以使用熊貓的修改版本低於.to_dict("records")代碼：

from pandas.core.common import _maybe_box_datetimelike 
my_list = [dict((k, _maybe_box_datetimelike(v)) for k, v in zip(df.columns, row) if v != None and v == v) for row in df.values] 
db.insert_many(my_list) 

其中if v != None and v == v我已經添加了檢查，以確保該值不None或nan然後將其放入行的字典中。現在您的.insert_many將僅包含文檔中具有值的鍵（並且不包含數據類型null）。

Answer 6

我覺得這個問題很酷。就我而言，我一直花時間更多地關注大型數據框的移動。在這種情況下，熊貓往往會允許你選擇chunksize（例如pandas.DataFrame.to_sql）。所以我認爲我通過添加我在這個方向使用的函數來貢獻於此。

def write_df_to_mongoDB( my_df,\ 
          database_name = 'mydatabasename' ,\ 
          collection_name = 'mycollectionname', 
          server = 'localhost',\ 
          mongodb_port = 27017,\ 
          chunk_size = 100): 
    #""" 
    #This function take a list and create a collection in MongoDB (you should 
    #provide the database name, collection, port to connect to the remoete database, 
    #server of the remote database, local port to tunnel to the other machine) 
    # 
    #--------------------------------------------------------------------------- 
    #Parameters/Input 
    # my_list: the list to send to MongoDB 
    # database_name: database name 
    # 
    # collection_name: collection name (to create) 
    # server: the server of where the MongoDB database is hosted 
    #  Example: server = '132.434.63.86' 
    # this_machine_port: local machine port. 
    #  For example: this_machine_port = '27017' 
    # remote_port: the port where the database is operating 
    #  For example: remote_port = '27017' 
    # chunk_size: The number of items of the list that will be send at the 
    #  some time to the database. Default is 100. 
    # 
    #Output 
    # When finished will print "Done" 
    #---------------------------------------------------------------------------- 
    #FUTURE modifications. 
    #1. Write to SQL 
    #2. Write to csv 
    #---------------------------------------------------------------------------- 
    #30/11/2017: Rafael Valero-Fernandez. Documentation 
    #""" 



    #To connect 
    # import os 
    # import pandas as pd 
    # import pymongo 
    # from pymongo import MongoClient 

    client = MongoClient('localhost',int(mongodb_port)) 
    db = client[database_name] 
    collection = db[collection_name] 
    # To write 
    collection.delete_many({}) # Destroy the collection 
    #aux_df=aux_df.drop_duplicates(subset=None, keep='last') # To avoid repetitions 
    my_list = my_df.to_dict('records') 
    l = len(my_list) 
    ran = range(l) 
    steps=ran[chunk_size::chunk_size] 
    steps.extend([l]) 

    # Inser chunks of the dataframe 
    i = 0 
    for j in steps: 
     print j 
     collection.insert_many(my_list[i:j]) # fill de collection 
     i = j 

    print('Done') 
    return