從下載url導入Kaggle csv到熊貓DataFrame

Question

我一直在嘗試使用不同的方法將SpaceX任務csv fileKaggle直接導入到熊貓DataFrame中，但沒有任何成功。

我需要發送請求登錄。這是我到目前爲止：

import requests 
import pandas as pd 
from io import StringIO 

# Link to the Kaggle data set & name of zip file 
login_url = 'http://www.kaggle.com/account/login?ReturnUrl=/spacex/spacex-missions/downloads/database.csv' 

# Kaggle Username and Password 
kaggle_info = {'UserName': "user", 'Password': "pwd"} 

# Login to Kaggle and retrieve the data. 
r = requests.post(login_url, data=kaggle_info, stream=True) 
df = pd.read_csv(StringIO(r.text)) 

r正在返回頁面的html內容。 df = pd.read_csv(url)給出了CParser錯誤： CParserError: Error tokenizing data. C error: Expected 1 fields in line 13, saw 6

我搜索了一個解決方案，但到目前爲止，沒有我試着努力。

Answer 1

您正在創建一個流並將其直接傳遞給熊貓。我認爲你需要傳遞一個像熊貓一樣的文件。看看this answer尋找一個可能的解決方案（使用帖子，而不是在請求中）。

此外，我認爲您使用的重定向登錄網址不能正常工作。 I know i suggested that here。但我最終沒有使用是因爲後請求調用沒有處理重定向（我懷疑）。

最後我用我的項目中的代碼是這樣的：

def from_kaggle(data_sets, competition): 
    """Fetches data from Kaggle 

    Parameters 
    ---------- 
    data_sets : (array) 
     list of dataset filenames on kaggle. (e.g. train.csv.zip) 

    competition : (string) 
     name of kaggle competition as it appears in url 
     (e.g. 'rossmann-store-sales') 

    """ 
    kaggle_dataset_url = "https://www.kaggle.com/c/{}/download/".format(competition) 

    KAGGLE_INFO = {'UserName': config.kaggle_username, 
        'Password': config.kaggle_password} 

    for data_set in data_sets: 
     data_url = path.join(kaggle_dataset_url, data_set) 
     data_output = path.join(config.raw_data_dir, data_set) 
     # Attempts to download the CSV file. Gets rejected because we are not logged in. 
     r = requests.get(data_url) 
     # Login to Kaggle and retrieve the data. 
     r = requests.post(r.url, data=KAGGLE_INFO, stream=True) 
     # Writes the data to a local file one chunk at a time. 
     with open(data_output, 'wb') as f: 
      # Reads 512KB at a time into memory 
      for chunk in r.iter_content(chunk_size=(512 * 1024)): 
       if chunk: # filter out keep-alive new chunks 
        f.write(chunk) 

使用例：

sets = ['train.csv.zip', 
     'test.csv.zip', 
     'store.csv.zip', 
     'sample_submission.csv.zip',] 
from_kaggle(sets, 'rossmann-store-sales') 

您可能需要解壓縮文件。

def _unzip_folder(destination): 
    """Unzip without regards to the folder structure. 

    Parameters 
    ---------- 
    destination : (str) 
     Local path and filename where file is should be stored. 
    """ 
    with zipfile.ZipFile(destination, "r") as z: 
     z.extractall(config.raw_data_dir) 

所以我從來沒有真正直接裝載入數據幀，而是將其存儲到磁盤第一。但是，您可以修改它以使用臨時目錄，並在讀取它們後刪除這些文件。