從用python編寫的蜘蛛獲取csv中的重複內容

Question

我創建了一個按照我的預期收集數據的蜘蛛。我現在面臨的唯一問題是結果有很多重複。不過，我想動搖重複的功能關閉，CSV寫結果：

下面是代碼：

import csv 
import requests 
from lxml import html 

def Startpoint(): 
    global writer 
    outfile=open('Data.csv','w',newline='') 
    writer=csv.writer(outfile) 
    writer.writerow(["Name","Price"]) 
    address = "https://www.sephora.ae/en/stores/" 
    page = requests.get(address) 
    tree = html.fromstring(page.text) 
    titles=tree.xpath('//li[contains(@class,"level0")]') 
    for title in titles: 
     href = title.xpath('.//a[contains(@class,"level0")]/@href')[0] 
     Layer2(href) 

def Layer2(address): 
    global writer 
    page = requests.get(address) 
    tree = html.fromstring(page.text) 
    titles=tree.xpath('//li[contains(@class,"amshopby-cat")]') 
    for title in titles: 
     href = title.xpath('.//a/@href')[0] 
     Endpoint(href) 

def Endpoint(address): 
    global writer 
    page = requests.get(address) 
    tree = html.fromstring(page.text) 
    titles=tree.xpath('//div[@class="product-info"]') 
    for title in titles: 
     Name = title.xpath('.//div[contains(@class,"h3")]/a[@title]/text()')[0] 
     Price = title.xpath('.//span[@class="price"]/text()')[0] 
     metco=(Name,Price) 
     print(metco) 
     writer.writerow(metco) 

Startpoint() 

Answer 1

你不需要csv模塊wrtie CSV文件。指定擴展就足夠了。因此，把你的代碼變成

import requests 
from lxml import html 

delimiter = ";" 
file_name = 'data.csv' 

def Startpoint(): 
    address = "https://www.sephora.ae/en/stores/" 
    page = requests.get(address) 
    tree = html.fromstring(page.text) 
    titles=tree.xpath('//li[contains(@class,"level0")]') 
    for title in titles: 
     href = title.xpath('.//a[contains(@class,"level0")]/@href')[0] 
     Layer2(href) 

def Layer2(address): 
    page = requests.get(address) 
    tree = html.fromstring(page.text) 
    titles=tree.xpath('//li[contains(@class,"amshopby-cat")]') 
    for title in titles: 
     href = title.xpath('.//a/@href')[0] 
     Endpoint(href) 

def Endpoint(address): 
    page = requests.get(address) 
    tree = html.fromstring(page.text) 
    titles=tree.xpath('//div[@class="product-info"]') 
    for title in titles: 
     Name = title.xpath('.//div[contains(@class,"h3")]/a[@title]/text()')[0] 
     Price = title.xpath('.//span[@class="price"]/text()')[0] 
     metco=(Name,Price) 
     print(metco) 
     with open(file_name,'a') as outfile: 
      outfile.write(delimiter.join(metco).encode('utf8') + '\n') 

with open(file_name,'w') as outfile: 
    outfile.write(delimiter.join(["Product Name", "Price"])+'\n') 
Startpoint() 

應該做的伎倆。請注意0​​部分，它可以防止您從UnicodeEncodeError開始的寫入過程。此外，請注意open函數中使用的參數'w'和'a'。第一種意思是「寫」，第二種意思是「附加」。但是，即使從啓發式的角度來看，這些代碼的工作原理，遠不是「好」的想法。