格式MongoDB的查詢中使用Pymongo

Question

我想插入我的查詢對象在pymongo連接器創建大熊貓數據幀：

import pandas as pd 
from pymongo import MongoClient 


def _connect_mongo(host, port, username, password, db): 

    if username and password: 
     mongo_uri = 'mongodb://%s:%s@%s:%s/%s' % (username, password, host, port, db) 
     conn = MongoClient(mongo_uri) 
    else: 
     conn = MongoClient(host, port) 


    return conn[db] 


def read_mongo(db, collection, query={}, host='localhost', port=27017, username=None, password=None, no_id=True): 
    """ Read from Mongo and Store into DataFrame """ 

    # Connect to MongoDB 
    db = _connect_mongo(host=host, port=port, username=username, password=password, db=db) 

    # Make a query to the specific DB and Collection 
    cursor = db[collection].find(query) 
    # Expand the cursor and construct the DataFrame 
    df = pd.DataFrame(list(cursor)) 

    # Delete the _id 
    # if no_id: 
    # del df['_id'] 

    return df 

我的查詢被定義爲：

query_1 = "{ 
        "status" : {"$ne" : "deprecated"}, 
        "geoLocationData.date" : { $gte : new ISODate("2016-08-03") } 
       }, 
        { "geoLocationData.date": 1, 
         "geoLocationData.iso": 1, 
         "httpRequestData.ipAddress": 1, 
         "geoLocationData.city": 1, 
         "geoLocationData.latitude": 1, 
         "geoLocationData.longitude": 1 }" 

將其插入 - 獲得一個數據幀大熊貓：

df = read_mongo(db, collection, query_1, host, port, username, password) 

我得到的錯誤：

TypeError: filter must be an instance of dict, bson.son.SON, or other type that inherits from collections.Mapping 

如果我只是省略子文檔，查詢工作得很好，我可以將其轉換爲數據框。

我想這是關於將我的查詢轉換成字典（與子文件）。 我該怎麼做？

Answer 1

您的query_1變量是一個字符串，它似乎包含兩個字典。方法find首先將字典作爲過濾器參數，然後將投影作爲第二個參數。你只是傳遞一個論點。

這應該工作：

def read_mongo(db, collection, filter={}, projection={}, host='localhost', port=27017, username=None, password=None, no_id=True): 
    """ Read from Mongo and Store into DataFrame """ 

    # Connect to MongoDB 
    db = _connect_mongo(host=host, port=port, username=username, password=password, db=db) 

    # Make a query to the specific DB and Collection 
    cursor = db[collection].find(filter, projection) 
    # Expand the cursor and construct the DataFrame 
    df = pd.DataFrame(list(cursor)) 

    # Delete the _id 
    # if no_id: 
    # del df['_id'] 

    return df 

query_filter = { 
        "status" : {"$ne" : "deprecated"}, 
        "geoLocationData.date" : { $gte : new ISODate("2016-08-03") } 
       } 
query_project = { "geoLocationData.date": 1, 
         "geoLocationData.iso": 1, 
         "httpRequestData.ipAddress": 1, 
         "geoLocationData.city": 1, 
         "geoLocationData.latitude": 1, 
         "geoLocationData.longitude": 1 } 

df = read_mongo(db, collection, query_filter, query_project, host, port, username, password) 

不過，我不認爲這種方法可以聯合你想要它做的相當的。這是因爲find操作中的投影只能包含或排除它不以您想要的方式映射它們的字段。你可以遍歷你的遊標來處理數據，然後傳入構造函數DataFrame（注意不要創建巨大的python列表）。更好的方法是不使用find在所有，但使用aggregate：

cursor = db[collection].aggregate([filter, projection]) 

現在你可以讓你的過濾器match流水線階段和投影一個project階段。

query_project = { "geoDate":"$geoLocationData.date", 
        "geoLoc":"$geoLocationData.iso", 
        "ipAddress": "$httpRequestData.ipAddress", 
        "city": "$geoLocationData.city", 
         "lat": "$geoLocationData.latitude", 
         "long":"$geoLocationData.longitude"} 

df = read_mongo(db, collection, {"$match" : query_filter}, {"$project" : query_project}, host, port, username, password)