如何比較熊貓數據框的聚合部分？

Question

是否可以比較pandas Dataframe中的列的部分？我有以下Dataframe示例，其中有4種語言已保存（en，de，nl，ua），並且每種語言應具有相同的鍵/相同數量的鍵，但具有不同的值（將靜態列留在那裏爲完成，因爲我有一個靜態列，其值始終保持不變）。

static │ langs │ keys │ values 

x  │ en  │ key_1 │ value_en_1 
x  │ en  │ key_2 │ value_en_2 
x  │ en  │ key_3 │ value_en_3 
x  │ de  │ key_1 │ value_de_1 
x  │ de  │ key_2 │ value_de_2 
x  │ de  │ key_3 │ value_de_3 
x  │ nl  │ key_1 │ value_nl_1 
x  │ nl  │ key_2 │ value_nl_2 
x  │ ua  │ key_1 │ value_ua_1 

我需要檢查哪些鍵，多少每種語言的缺失相比，英語一（「恩」在這裏），所以像這將是一個所需的輸出：

│ Lang │ Static │ # Missing │ Keys   │ 
│ de │ x  │ 0   │    │ 
│ nl │ x  │ 1   │ key_3   │ 
│ ua │ x  │ 2   │ key_2, key_3 │ 

這是我目前的進度：

import pandas as pd 

# this is read from a csv, but I'll leave it as list of lists for simplicity 
rows = [ 
    ['x', 'en', 'key_1', 'value_en_1'], 
    ['x', 'en', 'key_2', 'value_en_2'], 
    ['x', 'en', 'key_3', 'value_en_3'], 
    ['x', 'de', 'key_1', 'value_de_1'], 
    ['x', 'de', 'key_2', 'value_de_2'], 
    ['x', 'de', 'key_3', 'value_de_3'], 
    ['x', 'nl', 'key_1', 'value_nl_1'], 
    ['x', 'nl', 'key_2', 'value_nl_2'], 
    ['x', 'ua', 'key_1', 'value_en_1'] 
] 

# create DataFrame out of rows of data 
df = pd.DataFrame(rows, columns=["static", "language", "keys", "values"]) 
# print out DataFrame 
print("Dataframe: ", df) 

# first group by language and the static column 
df_grp = df.groupby(["static", "language"]) 

# try to sum the number of keys and values per each language 
df_summ = df_grp.agg(["count"]) 

# print out the sums 
print() 
print(df_summ) 

# how to compare? 
# how to get the keys? 

這是df_summ的輸出：

    keys values 
       count count 
static language    
x  de   3  3 
     en   3  3 
     nl   2  2 
     ua   1  1 

在這一點上，我不知道如何繼續。我很感激任何幫助/提示。

P.S.這是在Python 3.5上。

Answer 1

編輯：

#get set per groups by static and language 
a = df.groupby(["static",'language'])['keys'].apply(set).reset_index() 
#filter only en language per group by static and create set 
b = df[df['language'] == 'en'].groupby("static")['keys'].apply(set) 
#subtract mapped set b and join 
c = (a['static'].map(b) - a['keys']).str.join(', ').rename('Keys') 
#substract lengths 
m = (a['static'].map(b).str.len() - a['keys'].str.len()).rename('Missing') 

df = pd.concat([a[['static','language']], m, c], axis=1) 
print (df) 
    static language Missing   Keys 
0  x  de  0    
1  x  en  0    
2  x  nl  1   key_3 
3  x  ua  2 key_3, key_2 

編輯：

我嘗試改變數據：

rows = [ 
    ['x', 'en', 'key_1', 'value_en_1'], 
    ['x', 'en', 'key_2', 'value_en_2'], 
    ['x', 'en', 'key_3', 'value_en_3'], 
    ['x', 'de', 'key_1', 'value_de_1'], 
    ['x', 'de', 'key_2', 'value_de_2'], 
    ['x', 'de', 'key_3', 'value_de_3'], 
    ['x', 'nl', 'key_1', 'value_nl_1'], 
    ['x', 'nl', 'key_2', 'value_nl_2'], 
    ['x', 'ua', 'key_1', 'value_en_1'], 
    ['y', 'en', 'key_1', 'value_en_1'], 
    ['y', 'en', 'key_2', 'value_en_2'], 
    ['y', 'de', 'key_4', 'value_en_3'], 
    ['y', 'de', 'key_1', 'value_de_1'], 
    ['y', 'de', 'key_2', 'value_de_2'], 
    ['y', 'de', 'key_3', 'value_de_3'], 
    ['y', 'de', 'key_5', 'value_nl_1'], 
    ['y', 'nl', 'key_2', 'value_nl_2'], 
    ['y', 'ua', 'key_1', 'value_en_1'] 
] 

# create DataFrame out of rows of data 
df = pd.DataFrame(rows, columns=["static", "language", "keys", "values"]) 
# print out DataFrame 
#print(df) 

和輸出：

print (df) 
    static language Missing   Keys 
0  x  de  0    
1  x  en  0    
2  x  nl  1   key_3 
3  x  ua  2 key_3, key_2 
4  y  de  -3    
5  y  en  0    
6  y  nl  1   key_1 
7  y  ua  1   key_2 

問題是de爲y靜態有更多的鍵在en語言。

Answer 2

您可以首先通過分組和計算nans的數量來創建缺失的列。然後創建鍵列並添加靜態列。

df2 = (
    df.groupby('langs')['keys'].apply(lambda x: x.values) 
     .apply(pd.Series) 
     .assign(Missing=lambda x: x.isnull().sum(axis=1)) 
) 

(
    df2[['Missing']].assign(static=df.static.iloc[0], 
          Keys=df2.apply(lambda x: ','.join(df2.loc['en'].loc[x.isnull()]),axis=1))  
) 

Out[44]: 
     Missing   Keys static 
langs        
de   0     x 
en   0     x 
nl   1  key_3  x 
ua   2 key_2,key_3  x 

Answer 3

# First we group with `language` and aggregate `static` with `min` (it's always the same anyway) 
# and `keys` with a lambda function that creates a `set`. 
In [2]: grouped = df.groupby('language').agg({'static': 'min', 'keys': lambda x: set(x)}) 

# Then we get the missing keys... 
In [3]: missing = (grouped['keys']['en'] - grouped['keys']) 

# ... and count them 
In [4]: missing_counts = missing.apply(len).rename('# Missing') 

# Then we join all of this together and replace the keys with a joined string. 
In [5]: grouped.drop('keys', axis=1).join(missing_counts).join(missing.apply(', '.join)).reset_index() 
Out[5]: 
    language static # Missing   keys 
0  de  x   0 
1  en  x   0 
2  nl  x   1   key_3 
3  ua  x   2 key_2, key_3 

Answer 4

因爲你把R標籤在你的問題，這裏是如何與tidyr和dplyr做到這一點：

library(dplyr);library(tidyr) 
df %>% 
    complete(nesting(static, langs), keys) %>% 
    group_by(langs)%>% 
    summarise(Static=max(static), 
      Missing=sum(is.na(values)), 
      Keys=toString(keys[is.na(values)]) 
      ) 

    langs Static Missing   Keys 
    <chr> <chr> <int>  <chr> 
1 de  x  0    
2 en  x  0    
3 nl  x  1  key_3 
4 ua  x  2 key_2, key_3 

數據

df <- read.table(text="static langs keys values 
'x' 'en' 'key_1' 'value_en_1' 
'x' 'en' 'key_2' 'value_en_2' 
'x' 'en' 'key_3' 'value_en_3' 
'x' 'de' 'key_1' 'value_de_1' 
'x' 'de' 'key_2' 'value_de_2' 
'x' 'de' 'key_3' 'value_de_3' 
'x' 'nl' 'key_1' 'value_nl_1' 
'x' 'nl' 'key_2' 'value_nl_2' 
'x' 'ua' 'key_1' 'value_en_1'",header=TRUE,stringsAsFactors = FALSE)