刪除重複

Question

我有不幸的是包含重複，像這樣的元組的列表：

[(67, u'top-coldestcitiesinamerica'), (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), (65, u'a-b-c-ca-d-ab-ea-d-c-c'), (64, u'a-b-c-ca-d-ab-ea-d-c-c'), (63, u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), (62, u'ghgemissions'), (61, u'top-coldestcitiesinamerica'), (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), (57, u'culture'), (55, u'cas-k-ihaveanidea'), (54, u'trendsfor'), (53, u'batteryimpedance'), (52, u'evs-howey-full'), (51, u'bericht'), (49, u'classiccarinsurance'), (47, u'uploaded_file'), (46, u'x_file'), (45, u's-s-main'), (44, u'vehicle-propulsion'), (43, u'x_file')] 

的問題是，元組的第一個元素（0基於排序）是我想要的條目檢查重複。所以，我可以看到：

(67, u'top-coldestcitiesinamerica') 
(61, u'top-coldestcitiesinamerica') 

..are重複，我想刪除其中的一個（類似於set）。因此，在最後，我想有元組的，象這樣沒有重複（即元組的第一個元素沒有重複）乾淨的列表：

[(67, u'top-coldestcitiesinamerica'), (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), (65, u'a-b-c-ca-d-ab-ea-d-c-c') (63, u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), (62, u'ghgemissions'), (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), (57, u'culture'), (55, u'cas-k-ihaveanidea'), (54, u'trendsfor'), (53, u'batteryimpedance'), (52, u'evs-howey-full'), (51, u'bericht'), (49, u'classiccarinsurance'), (47, u'uploaded_file'), (46, u'x_file'), (45, u's-s-main'), (44, u'vehicle-propulsion')] 

我怎樣才能在Python的實現這一目標辦法？ 謝謝！

Answer 1

您可以使用set方法從How do you remove duplicates from a list in whilst preserving order?，使用x[1]作爲唯一標識符：

def unique_second_element(seq): 
    seen = set() 
    seen_add = seen.add 
    return [x for x in seq if not (x[1] in seen or seen_add(x[1]))] 

注意，OrderedDict做法也顯示也將工作，如果你想保留最後發生;對於第一次發生，您必須將輸入反向，然後再次反向輸出。

你可以讓這個更通用的支持key功能：

def unique_preserve_order(seq, key=None): 
    if key is None: 
     key = lambda elem: elem 
    seen = set() 
    seen_add = seen.add 
    augmented = ((key(x), x) for x in seq) 
    return [x for k, x in augmented if not (k in seen or seen_add(k))] 

然後用

import operator 

unique_preserve_order(yourlist, key=operator.itemgetter(1)) 

演示：

>>> def unique_preserve_order(seq, key=None): 
...  if key is None: 
...   key = lambda elem: elem 
...  seen = set() 
...  seen_add = seen.add 
...  augmented = ((key(x), x) for x in seq) 
...  return [x for k, x in augmented if not (k in seen or seen_add(k))] 
... 
>>> from pprint import pprint 
>>> import operator 
>>> yourlist = [(67, u'top-coldestcitiesinamerica'), (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), (65, u'a-b-c-ca-d-ab-ea-d-c-c'), (64, u'a-b-c-ca-d-ab-ea-d-c-c'), (63, u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), (62, u'ghgemissions'), (61, u'top-coldestcitiesinamerica'), (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), (57, u'culture'), (55, u'cas-k-ihaveanidea'), (54, u'trendsfor'), (53, u'batteryimpedance'), (52, u'evs-howey-full'), (51, u'bericht'), (49, u'classiccarinsurance'), (47, u'uploaded_file'), (46, u'x_file'), (45, u's-s-main'), (44, u'vehicle-propulsion'), (43, u'x_file')] 
>>> pprint(unique_preserve_order(yourlist, operator.itemgetter(1))) 
[(67, u'top-coldestcitiesinamerica'), 
(66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), 
(65, u'a-b-c-ca-d-ab-ea-d-c-c'), 
(63, 
    u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), 
(62, u'ghgemissions'), 
(58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), 
(57, u'culture'), 
(55, u'cas-k-ihaveanidea'), 
(54, u'trendsfor'), 
(53, u'batteryimpedance'), 
(52, u'evs-howey-full'), 
(51, u'bericht'), 
(49, u'classiccarinsurance'), 
(47, u'uploaded_file'), 
(46, u'x_file'), 
(45, u's-s-main'), 
(44, u'vehicle-propulsion')] 

Answer 2

作爲一個備選答案，你可以使用itertools.groupby()如果你有一個巨大的列表，這可能會有幫助，但是不如set：

>>> from itertools import groupby 
>>> from operator import itemgetter 
>>> [next(g) for _,g in groupby(sorted(l,key=itemgetter(1)),itemgetter(1))] 
[(65, u'a-b-c-ca-d-ab-ea-d-c-c'), (63, u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), (53, u'batteryimpedance'), (51, u'bericht'), (55, u'cas-k-ihaveanidea'), (49, u'classiccarinsurance'), (57, u'culture'), (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), (52, u'evs-howey-full'), (62, u'ghgemissions'), (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), (45, u's-s-main'), (67, u'top-coldestcitiesinamerica'), (54, u'trendsfor'), (47, u'uploaded_file'), (44, u'vehicle-propulsion'), (46, u'x_file')] 

Answer 3

1. 定義檢查列表變量添加關鍵。
2. 迭代輸入列表中的每個項目。
3. 檢查密鑰是否存在或不在檢查列表中。
4. 如果不存在，則將項目添加到結果列表並更新檢查列表。
5. 打印結果。

代碼：

input_list = [(67, u'top-coldestcitiesinamerica'), (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), (65, u'a-b-c-ca-d-ab-ea-d-c-c'), (64, u'a-b-c-ca-d-ab-ea-d-c-c'), (63, u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), (62, u'ghgemissions'), (61, u'top-coldestcitiesinamerica'), (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), (57, u'culture'), (55, u'cas-k-ihaveanidea'), (54, u'trendsfor'), (53, u'batteryimpedance'), (52, u'evs-howey-full'), (51, u'bericht'), (49, u'classiccarinsurance'), (47, u'uploaded_file'), (46, u'x_file'), (45, u's-s-main'), (44, u'vehicle-propulsion'), (43, u'x_file')] 

check_list = set() 
result = [] 
for i in input_list: 
    if not i[1] in check_list: 
     result.append(i) 
     check_list.add(i[1]) 

import pprint 
pprint.pprint(result) 

輸出：

$ python task4.py 
[(67, u'top-coldestcitiesinamerica'), 
(66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), 
(65, u'a-b-c-ca-d-ab-ea-d-c-c'), 
(63, 
    u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), 
(62, u'ghgemissions'), 
(58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), 
(57, u'culture'), 
(55, u'cas-k-ihaveanidea'), 
(54, u'trendsfor'), 
(53, u'batteryimpedance'), 
(52, u'evs-howey-full'), 
(51, u'bericht'), 
(49, u'classiccarinsurance'), 
(47, u'uploaded_file'), 
(46, u'x_file'), 
(45, u's-s-main'), 
(44, u'vehicle-propulsion')] 

Answer 4

我做了一個很樸實和簡單的方法。

lst=[(67, u'top-coldestcitiesinamerica'), (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), (65, u'a-b-c-ca-d-ab-ea-d-c-c'), (64, u'a-b-c-ca-d-ab-ea-d-c-c'), (63, u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), (62, u'ghgemissions'), (61, u'top-coldestcitiesinamerica'), (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), (57, u'culture'), (55, u'cas-k-ihaveanidea'), (54, u'trendsfor'), (53, u'batteryimpedance'), (52, u'evs-howey-full'), (51, u'bericht'), (49, u'classiccarinsurance'), (47, u'uploaded_file'), (46, u'x_file'), (45, u's-s-main'), (44, u'vehicle-propulsion'), (43, u'x_file')] 

lst2 = [] #empty list to fill with unique tuples 
lst_banned = [] #empty list to fill with banned elements 

for tup in lst: 
    if tup[-1] not in lst_banned: 
     lst_banned.append(tup[-1]) 
     lst2.append(tup) 

lst=lst2 
del lst2 
del lst_banned