下面顯示的代碼部分我從twitter獲取推文,並將它們最初存儲在「backup.txt」中。我還創建了一個文件「tweets3.csv」,並保存每個推文的一些特定字段。但我意識到一些推文具有完全相同的文本(重複)。我怎樣才能從我的csv文件中刪除這些文件?從csv文件中刪除重複tweet
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
import time
import json
import csv
ckey = ''
csecret = ''
atoken = ''
asecret = ''
class listener(StreamListener):
def on_data(self, data):
try:
all_data = json.loads(data)
with open("backup.txt", 'a') as backup:
backup.write(str(all_data) + "\n")
backup.close()
text = str(all_data["text"]).encode("utf-8")
id = str(all_data["id"]).encode("utf-8")
timestamp = str(all_data["timestamp_ms"]).encode("utf-8")
sn = str(all_data["user"]["screen_name"]).encode("utf-8")
user_id = str(all_data["user"]["id"]).encode("utf-8")
create = str(all_data["created_at"]).encode("utf-8")
follower = str(all_data["user"]["followers_count"]).encode("utf-8")
following = str(all_data["user"]["following"]).encode("utf-8")
status = str(all_data["user"]["statuses_count"]).encode("utf-8")
# text = data.split(',"text":"')[1].split('","source')[0]
# name = data.split(',"screen_name":"')[1].split('","location')[0]
contentlist = []
contentlist.append(text)
contentlist.append(id)
contentlist.append(timestamp)
contentlist.append(sn)
contentlist.append(user_id)
contentlist.append(create)
contentlist.append(follower)
contentlist.append(following)
contentlist.append(status)
print contentlist
f = open("tweets3.csv", 'ab')
wrt = csv.writer(f, dialect='excel')
try:
wrt.writerow(contentlist)
except UnicodeEncodeError, UnicodeEncodeError:
return True
return True
except BaseException, e:
print 'failed on data',type(e),str(e)
time.sleep(3)
def on_error(self, status):
print "Error status:" + str(status)
auth = OAuthHandler(ckey, csecret)
auth.set_access_token(atoken, asecret)
twitterStream = Stream(auth, listener())
twitterStream.filter(track=["zikavirus"], languages=['en'])
我想你可以做一個列表變量,每次你去鳴叫,你遍歷列表並檢查,如果這個ID存在與否。如果是的話,什麼也不做。如果否,請將該ID添加到列表中。 – Fusseldieb