轉換CSV文件到大熊貓數據幀

Question

我有以下格式的CSV文件：

DATES, 01-12-2010, 01-12-2010, 01-12-2010, 02-12-2010, 02-12-2010, 02-12-2010 
UNITS, Hz, kV, MW, Hz, kV, MW 
Interval, , , , , ,           
00:15, 49.82, 33.73755, 34.65, 49.92, 33.9009, 36.33, 
00:30, 49.9, 33.7722, 35.34, 49.89, 33.8382, 37.65, 
00:45, 49.94, 33.8316, 33.5, 50.09, 34.07745, 37.41, 
01:00, 49.86, 33.94875, 30.91, 50.18, 34.20945, 36.11, 
01:15, 49.97, 34.2243, 27.28, 50.11, 34.3596, 33.24, 
01:30, 50.02, 34.3332, 26.91, 50.12, 34.452, 31.03, 
01:45, 50.01, 34.1286, 31.26, 50, 33.9306, 38.86, 
02:00, 50.08, 33.9141, 34.96, 50.14, 33.99165, 38.31, 
02:15, 50.07, 33.84975, 35.33, 50.01, 33.9537, 39.78, 
02:30, 49.97, 34.0263, 33.63, 50.07, 33.8547, 41.48, 

我想上面的轉換爲以下格式的數據幀：

    Hz  kV   MW 
DATES_Interval 
01-12-2010 00:15 49.82 33.73755 34.65 
01-12-2010 00:30 49.9 33.7722  35.34 
01-12-2010 00:45 49.94 33.8316  33.5 
01-12-2010 01:00 49.86 33.94875 30.91 
01-12-2010 01:15 49.97 34.2243  27.28 
01-12-2010 01:30 50.02 34.3332  26.91 
01-12-2010 01:45 50.01 34.1286  31.26 
01-12-2010 02:00 50.08 33.9141  34.96 
01-12-2010 02:15 50.07 33.84975 35.33 
01-12-2010 02:30 49.97 34.0263  33.63 
02-12-2010 00:15 49.92 33.9009  36.33 
02-12-2010 00:30 49.89 33.8382  37.65 
02-12-2010 00:45 50.09 34.07745 37.41 
02-12-2010 01:00 50.09 34.07745 37.41 
02-12-2010 01:15 50.11 34.3596  33.24 
02-12-2010 01:30 50.12 34.452  31.03 
02-12-2010 01:45 50  33.9306  38.86 
02-12-2010 02:00 50.14 33.99165 38.31 
02-12-2010 02:15 50.01 33.9537  39.78 
02-12-2010 02:30 50.07 33.8547  41.48 

怎麼辦我用熊貓做這個嗎？

Answer 1

在大熊貓做這種事情的關鍵是stack()方法：

df.stack(level=0) 

不過，我發現得到的地方，你可以利用這一點，至少特定CSV是棘手.. 到退一步說（有幾乎可以肯定這樣做一個更好的辦法！）：

df_data = pd.read_csv('e.csv', sep=',\s+', header=None, skiprows=3)[range(7)].set_index(0) 
df_cols = pd.read_csv('e.csv', sep=',\s+', header=None, nrows=2).set_index(0)[:2] #interval causing problems  
df_ = df_cols.append(df_data).T.set_index(['DATES','UNITS','Interval']).T 
df = df_.stack(level=0) 
df_dates = map(lambda x: pd.to_datetime(' '.join(x[::-1])), df.index) 
df.index = df_dates 

In [7]: df 
Out[7]: 
UNITS     Hz  MW  kV 
2010-01-12 00:15:00 49.82 34.65 33.73755 
2010-02-12 00:15:00 49.92 36.33, 33.9009 
2010-01-12 00:30:00 49.9 35.34 33.7722 
2010-02-12 00:30:00 49.89 37.65, 33.8382 
2010-01-12 00:45:00 49.94 33.5 33.8316 
2010-02-12 00:45:00 50.09 37.41, 34.07745 
2010-01-12 01:00:00 49.86 30.91 33.94875 
2010-02-12 01:00:00 50.18 36.11, 34.20945 
2010-01-12 01:15:00 49.97 27.28 34.2243 
2010-02-12 01:15:00 50.11 33.24, 34.3596 
2010-01-12 01:30:00 50.02 26.91 34.3332 
2010-02-12 01:30:00 50.12 31.03, 34.452 
2010-01-12 01:45:00 50.01 31.26 34.1286 
2010-02-12 01:45:00  50 38.86, 33.9306 
2010-01-12 02:00:00 50.08 34.96 33.9141 
2010-02-12 02:00:00 50.14 38.31, 33.99165 
2010-01-12 02:15:00 50.07 35.33 33.84975 
2010-02-12 02:15:00 50.01 39.78, 33.9537 
2010-01-12 02:30:00 49.97 33.63 34.0263 
2010-02-12 02:30:00 50.07 41.48, 33.8547 

這是一個有點亂，在一些列!:

逗號

def clean(s): 
    try: return float(s.strip(',')) 
    except: return s 

In [9]: df.applymap(clean) 
Out[9]: 
         Hz  MW  kV 
2010-01-12 00:15:00 49.82 34.65 33.73755 
2010-02-12 00:15:00 49.92 36.33 33.90090 
2010-01-12 00:30:00 49.90 35.34 33.77220 
2010-02-12 00:30:00 49.89 37.65 33.83820 
2010-01-12 00:45:00 49.94 33.50 33.83160 
2010-02-12 00:45:00 50.09 37.41 34.07745 
2010-01-12 01:00:00 49.86 30.91 33.94875 
2010-02-12 01:00:00 50.18 36.11 34.20945 
2010-01-12 01:15:00 49.97 27.28 34.22430 
2010-02-12 01:15:00 50.11 33.24 34.35960 
2010-01-12 01:30:00 50.02 26.91 34.33320 
2010-02-12 01:30:00 50.12 31.03 34.45200 
2010-01-12 01:45:00 50.01 31.26 34.12860 
2010-02-12 01:45:00 50.00 38.86 33.93060 
2010-01-12 02:00:00 50.08 34.96 33.91410 
2010-02-12 02:00:00 50.14 38.31 33.99165 
2010-01-12 02:15:00 50.07 35.33 33.84975 
2010-02-12 02:15:00 50.01 39.78 33.95370 
2010-01-12 02:30:00 49.97 33.63 34.02630 
2010-02-12 02:30:00 50.07 41.48 33.85470 

Answer 2

另一種解決方案將是

1. 閱讀日期（CSV的第一行）
2. 讀取的數據的剩餘部分，包括間隔
3. 構建索引根據需要，並將其應用於數據幀

下面是一些示例代碼：

In [1]: from StringIO import StringIO  
In [2]: import pandas as pd 
In [3]: pd.__version__ 
Out[3]: '0.10.1' 

In [4]: CSV_SAMPLE = """ 
DATES, 01-12-2010, 01-12-2010, 01-12-2010, 02-12-2010, 02-12-2010, 02-12-2010 
UNITS, Hz, kV, MW, Hz, kV, MW 
Interval, , , , , , 
00:15, 49.82, 33.73755, 34.65, 49.92, 33.9009, 36.33, 
00:30, 49.9, 33.7722, 35.34, 49.89, 33.8382, 37.65, 
00:45, 49.94, 33.8316, 33.5, 50.09, 34.07745, 37.41, 
01:00, 49.86, 33.94875, 30.91, 50.18, 34.20945, 36.11, 
01:15, 49.97, 34.2243, 27.28, 50.11, 34.3596, 33.24, 
01:30, 50.02, 34.3332, 26.91, 50.12, 34.452, 31.03, 
""" 

#Create one dataframe from just the dates (and we'll grab the units, too) 
In [6]: datesdf = pd.read_csv(StringIO(CSV_SAMPLE), nrows= 2) 
In [7]: dates, units = datesdf.index.droplevel() 

In [9]: dates, units 
Out[9]: 
((' 01-12-2010', 
    ' 01-12-2010', 
    ' 01-12-2010', 
    ' 02-12-2010', 
    ' 02-12-2010', 
    ' 02-12-2010'), 
(' Hz', ' kV', ' MW', ' Hz', ' kV', ' MW')) 

#Create a second dataframe from the rest of the data 
In [11]: data = pd.read_csv(StringIO(CSV_SAMPLE), skiprows=3) 
In [12]: data = data.icol([0,1,2]) 

#Note: Instead, in pandas 0.10, you can use the usecols paramater in read_csv() 
# to combine the above two steps into one. 

In [14]: data.columns = units[:3] 
In [15]: print data 
      Hz  kV  MW 
00:15 49.82 33.73755 34.65 
00:30 49.90 33.77220 35.34 
00:45 49.94 33.83160 33.50 
01:00 49.86 33.94875 30.91 
01:15 49.97 34.22430 27.28 
01:30 50.02 34.33320 26.91 

現在創建所需的索引並應用它。以下是索引的幾種方法。

#We'll need to grab the intervals from this data df 
In [16]: intervals = data.index.tolist() 

In [17]: index1 = pd.MultiIndex.from_arrays([dates,intervals]) 

#This is a multi-index 
In [18]: print index1 
MultiIndex 
[(01-12-2010, 00:15), (01-12-2010, 00:30), (01-12-2010, 00:45), (02-12-2010, 01:00), (02-12-2010, 01:15), (02-12-2 
010, 01:30)] 

#This index is a tuple of date,interval 
In [21]: index2 = pd.Index(zip(dates, intervals)) 
In [22]: print index2 
Index([(01-12-2010, 00:15), (01-12-2010, 00:30), (01-12-2010, 00:45), (02-12-2010, 01:00), (02-12-2010, 01:15), (0 
2-12-2010, 01:30)], dtype=object) 

#This index is based on a string concat of date and interval 
In [23]: def list_join(x,y): 
    ....:   joined = x + ' ' + y 
    ....:   return joined.strip() 
    ....: 

In [24]: index3 = pd.Index(map(list_join, dates, intervals))  
In [25]: print index3 
Simple index: 
Index([01-12-2010 00:15, 01-12-2010 00:30, 01-12-2010 00:45, 02-12-2010 01:00, 02-12-2010 01:15, 02-12-2010 01:30], dtyp 
e=object) 

由於第三種類型的索引是您原始請求中的內容，我將使用它。

In [26]: data.index = index3 
In [27]: print data 
        Hz  kV  MW 
01-12-2010 00:15 49.82 33.73755 34.65 
01-12-2010 00:30 49.90 33.77220 35.34 
01-12-2010 00:45 49.94 33.83160 33.50 
02-12-2010 01:00 49.86 33.94875 30.91 
02-12-2010 01:15 49.97 34.22430 27.28 
02-12-2010 01:30 50.02 34.33320 26.91 

如果您抱怨非唯一索引值，您可能需要修改上面的代碼來處理完整的數據集。在這種情況下，請將Intervals csv列作爲數據列（而不是索引）讀入，並將其作爲數組以創建所需的索引，與上面相同。