使用R更正/整理數據

我正在處理的交易數據來自兩個來源，兩者都有一些優點和缺點。第一個只有有準確的$和出售單位（DF_Lookup），第二個有正確的人口統計（DFI），但某些售出的單位和單位不正確。所以，我寫了下面的代碼來處理這個問題。使用R更正/整理數據

這裏是我的數據：

DFI

dput(DFI) 
structure(list(PO_ID = c("P1234", "P1234", "P1234", "P1234", 
"P1234", "P1234", "P2345", "P2345", "P3456", "P4567"), SO_ID = c("S1", 
"S1", "S1", "S2", "S2", "S2", "S3", "S4", "S7", "S10"), F_Year = c(2012, 
2012, 2012, 2013, 2013, 2013, 2011, 2011, 2014, 2015), Product_ID = c("385X", 
"385X", "385X", "450X", "450X", "900X", "3700", "3700", "A11U", 
"2700"), Revenue = c(1, 2, 3, 34, 34, 6, 7, 88, 9, 100), Quantity = c(1, 
2, 3, 8, 8, 6, 7, 8, 9, 40), Location1 = c("MA", "NY", "WA", 
"NY", "WA", "NY", "IL", "IL", "MN", "CA")), .Names = c("PO_ID", 
"SO_ID", "F_Year", "Product_ID", "Revenue", "Quantity", "Location1" 
), row.names = c(NA, 10L), class = "data.frame")

DF_Lookup

dput(DF_Lookup) 

structure(list(PO_ID = c("P1234", "P1234", "P1234", "P2345", 
"P2345", "P3456", "P4567"), SO_ID = c("S1", "S2", "S2", "S3", 
"S4", "S7", "S10"), F_Year = c(2012, 2013, 2013, 2011, 2011, 
2014, 2015), Product_ID = c("385X", "450X", "900X", "3700", "3700", 
"A11U", "2700"), Revenue = c(50, 70, 35, 100, -50, 50, 100), 
    Quantity = c(3, 20, 20, 20, -10, 20, 40)), .Names = c("PO_ID", 
"SO_ID", "F_Year", "Product_ID", "Revenue", "Quantity"), row.names = c(NA, 
7L), class = "data.frame")

的第一次嘗試：

策略： - 使用加入覆蓋從DF_Lookup的條目DFI

DF_Generated <- DFI %>% 
    left_join(DF_Lookup,by = c("PO_ID", "SO_ID", "F_Year", "Product_ID")) %>% 
    dplyr::group_by(PO_ID, SO_ID, F_Year, Product_ID) %>% 
    dplyr::mutate(Count = n()) %>% 
    dplyr::ungroup()%>% 
    dplyr::mutate(Revenue = Revenue.y/Count, Quantity = Quantity.y/Count) %>% 
    dplyr::select(PO_ID:Product_ID,Location1,Revenue,Quantity)

預期輸出：

dput(DF_Generated) 
structure(list(PO_ID = c("P1234", "P1234", "P1234", "P1234", 
"P1234", "P1234", "P2345", "P2345", "P3456", "P4567"), SO_ID = c("S1", 
"S1", "S1", "S2", "S2", "S2", "S3", "S4", "S7", "S10"), F_Year = c(2012, 
2012, 2012, 2013, 2013, 2013, 2011, 2011, 2014, 2015), Product_ID = c("385X", 
"385X", "385X", "450X", "450X", "900X", "3700", "3700", "A11U", 
"2700"), Location1 = c("MA", "NY", "WA", "NY", "WA", "NY", "IL", 
"IL", "MN", "CA"), Revenue = c(16.6666666666667, 16.6666666666667, 
16.6666666666667, 35, 35, 35, 100, -50, 50, 100), Quantity = c(1, 
1, 1, 10, 10, 20, 20, -10, 20, 40)), class = c("tbl_df", "tbl", 
"data.frame"), row.names = c(NA, -10L), .Names = c("PO_ID", "SO_ID", 
"F_Year", "Product_ID", "Location1", "Revenue", "Quantity"))

挑戰：這非常適用於較小的數據集。我正在處理的原始數據有大約90萬條記錄。所以，上面的代碼需要永遠。

第二次嘗試： 因此，我認爲只更新那些有$ 和+/- 10％的範圍之外單位行。

這裏是我的代碼：

#Find out whether the numbers are within +/-10% range. 
DF_Mod<-DFI %>% 
    dplyr::group_by(PO_ID, SO_ID, F_Year, Product_ID) %>% 
    dplyr::summarise(Rev_agg = sum(Revenue), Qty_agg = sum(Quantity)) %>% 
    left_join(DF_Lookup) %>% 
    dplyr::rowwise() %>% 
    #check for +/- 10% confidence interval 
    dplyr::mutate(Compute = ifelse((abs(Rev_agg-Revenue)/Revenue <=0.1) & (abs(Qty_agg-Quantity)/Quantity <=0.1),"N","Y")) %>% 
    dplyr::rowwise() %>% 
    dplyr::ungroup() %>% 
    dplyr::select(PO_ID:Product_ID,Compute) %>% 
    dplyr::right_join(DFI) 

#Now, filter Compute == "Y" and then do the join with DF_Lookup. 
DF_Generated_2 <- DF_Mod %>% 
    dplyr::filter(Compute == "Y") %>% 
    left_join(DF_Lookup,by = c("PO_ID", "SO_ID", "F_Year", "Product_ID")) %>% 
    dplyr::group_by(PO_ID, SO_ID, F_Year, Product_ID) %>% 
    dplyr::mutate(Count = n()) %>% 
    dplyr::ungroup()%>% 
    dplyr::mutate(Revenue = Revenue.y/Count, Quantity = Quantity.y/Count) %>% 
    dplyr::select(PO_ID:Product_ID,Location1,Revenue,Quantity) 

#Bind the rows 
DF_Final <- rbind(DF_Generated_2,DFI[DF_Mod$Compute=="N",]) #Expected output

這裏，DF_Final確實是預期輸出。

問題：即使在遵循上述方法之後，由於涉及的連接數太多，性能非常緩慢。無論如何，我們可以加快這個過程嗎？還有其他更好的方法來做我想做的事嗎？

我很感激你的想法。我在這方面花了一天時間，但仍然無處可去。我很困難。

來源

2017-02-23 watchtower

你有沒有嘗試加入其他軟件包的功能？ http://stackoverflow.com/questions/4322219/whats-the-fastest-way-to-merge-join-data-frames-in-r – coletl

@coletl - 謝謝。我不太熟悉使用'data.table'和'sqldf'進行連接。如果有人能指導我，那真的會對我有幫助。 – watchtower

我還沒有大型數據集測試這一點，但它仍然可能是你所需要的（也有可能是這樣做的還有很多更快的方式）：

# load data table library 
library(data.table) 

# convert data frames to data tables 
DFI <- data.table(DFI) 
DF_Lookup <- data.table(DF_Lookup) 

# left join 
df <- merge(DFI, DF_Lookup, all.x = TRUE, by = c("PO_ID", "SO_ID", "F_Year", "Product_ID")) 

# Calculate the strange quantity and revenue 
df2 <- df[, list(Revenue = Revenue.y/.N, Quantity = Quantity.y/.N), 
       by = list(PO_ID, SO_ID, F_Year, Product_ID)]

來源

2017-02-23 09:26:51 ira

希望我正確地解釋你的問題。

使用來自DFI的正確人口統計數據來執行使用數據的查找。表

library(data.table) 
setDT(DFI) 
setDT(DF_Lookup) 
cols <- c("PO_ID", "SO_ID", "F_Year", "Product_ID") 
DF_Lookup[DFI, Location:=Location1, on=cols]

計算由PO_ID，SO_ID，F_Year，PRODUCT_ID，地點的平均收入和數量

DF_Lookup[, list(AveRevenue=mean(Revenue), AveQuantity=mean(Quantity)), 
    by=c(cols, "Location")]

，你可能還需要探索其他包mean更快的版本（搜索SO）

來源

2017-02-23 10:20:59 chinsoon12

使用R更正/整理數據

回答

相關問題