大熊貓每行加入兩個不同時間範圍的數據幀

我想計算某個公司在其收入日期的一年內在新聞中出現的次數，並在同一時間框架內比較其他人的次數。我有兩個熊貓數據框，一個是收益日期，另一個是新聞。我的方法很慢。有更好的熊貓/ numpy方式嗎？大熊貓每行加入兩個不同時間範圍的數據幀

import pandas as pd 

companies = pd.DataFrame({'CompanyName': ['A', 'B', 'C'], 'EarningsDate': ['2013/01/15', '2015/03/25', '2017/05/03']}) 
companies['EarningsDate'] = pd.to_datetime(companies.EarningsDate) 

news = pd.DataFrame({'CompanyName': ['A', 'A', 'A', 'B', 'B', 'C'], 
        'NewsDate': ['2012/02/01', '2013/01/10', '2015/05/13' , '2012/05/23', '2013/01/03', '2017/05/01']}) 
news['NewsDate'] = pd.to_datetime(news.NewsDate)

companies看起來像

CompanyName EarningsDate 
0 A   2013-01-15 
1 B   2015-03-25 
2 C   2017-05-03

news看起來像

CompanyName NewsDate 
0 A  2012-02-01 
1 A  2013-01-10 
2 A  2015-05-13 
3 B  2012-05-23 
4 B  2013-01-03 
5 C  2017-05-01

我如何改寫呢？這有效，但是每個數據幀大於500k行非常慢。

company_count = [] 
other_count = [] 

for _, company in companies.iterrows(): 
    end_date = company.EarningsDate 
    start_date = end_date - pd.DateOffset(years=1) 
    subset = news[(news.NewsDate > start_date) & (news.NewsDate < end_date)] 

    mask = subset.CompanyName==company.CompanyName 
    company_count.append(subset[mask].shape[0]) 
    other_count.append(subset[~mask].groupby('CompanyName').size().mean()) 

companies['12MonCompanyNewsCount'] = pd.Series(company_count) 
companies['12MonOtherNewsCount'] = pd.Series(other_count).fillna(0)

最終結果，companies看起來像

CompanyName EarningsDate 12MonCompanyNewsCount 12MonOtherNewsCount 
0 A   2013-01-15  2      2 
1 B   2015-03-25  0      0 
2 C   2017-05-03  1      0

來源

2017-08-29 E.K.

試試這個：https://stackoverflow.com/questions/22391433/count-the-frequency-that-a-value-occurs-in-a-dataframe-column – RetardedJoker

'value_counts（）'在這裏不起作用。我必須加入兩個不同窗口的數據框來進行聚合。 –

好吧，在這裏。

用於獲取12MonCompanyNewsCount，您可以使用merge_asof，這實在是整潔：

companies['12MonCompanyNewsCount'] = pd.merge_asof(
    news, 
    companies, 
    by='CompanyName', 
    left_on='NewsDate', 
    right_on='EarningsDate', 
    tolerance=pd.Timedelta('365D'), 
    direction='forward' 
).groupby('CompanyName').count().NewsDate

其中作爲當前執行的兩倍快的工作（並會變得更好）

對於12MonOtherNewsCount，我如果不循環事物，就無法真正找到一種方法。我想這是更簡潔：

companies['12MonOtherNewsCount'] = companies.apply(
    lambda x: len(
     news[ 
      (news.NewsDate.between(x.EarningsDate-pd.Timedelta('365D'), x.EarningsDate, inclusive=False)) 
      &(news.CompanyName!=x.CompanyName) 
     ] 
    ), 
    axis=1 
)

而且它確實看起來更快一點。

來源

2017-08-29 15:33:47

好的，'merge_asof'正是我在找的東西。它看起來在版本0.19.0中是新的。我升級了我的熊貓，我很高興去！非常感謝！ –

'merge_asof'最近介紹給我一個類似的問題。這真的是一個救星！ –

我不能找到一種方法，不是在companies行迭代。但是，您可以設置companies的開始日期列，遍歷companies的行併爲符合您的標準的日期和公司名稱news創建布爾指數。然後只需執行一個布爾值and操作並對得到的布爾數組進行求和。

我發誓，它看起來更有意義的代碼。

# create the start date column and the 12 month columns, 
# fill the 12 month columns with zeros for now 
companies['startdate'] = companies.EarningsDate - pd.DateOffset(years=1) 
companies['12MonCompanyNewsCount'] = 0 
companies['12MonOtherNewsCount'] = 0 

# iterate the rows of companies and hold the index 
for i, row in companies.iterrows(): 
    # create a boolean index when the news date is after the start date 
    # and when the news date is before the end date 
    # and when the company names match 
    ix_start = news.NewsDate >= row.startdate 
    ix_end = news.NewsDate <= row.EarningsDate 
    ix_samename = news.CompanyName == row.CompanyName 
    # set the news count value for the current row of `companies` using 
    # boolean `and` operations on the indices. first when the names match 
    # and again when the names don't match. 
    companies.loc[i,'12MonCompanyNewsCount'] = (ix_start & ix_end & ix_samename).sum() 
    companies.loc[i,'12MonOtherNewsCount'] = (ix_start & ix_end & ~ix_samename).sum() 

companies 
#returns: 

    CompanyName EarningsDate startdate 12MonCompanyNewsCount \ 
0   A 2013-01-15 2012-01-15      1 
1   B 2015-03-25 2014-03-25      0 
2   C 2017-05-03 2016-05-03      1 

    12MonOtherNewsCount 
0     2 
1     1 
2     0

來源

2017-08-29 15:14:45 James

大熊貓每行加入兩個不同時間範圍的數據幀

回答

相關問題