循環卡在第一頁

Question

一直使用美麗的湯來遍歷頁面，但無論出於什麼原因，我無法讓循​​環超出第一頁。它似乎應該很容易，因爲它是一個文本字符串，但它似乎循環回來，也許它是我的結構而不是我的文本字符串？

這是我有：

import csv 
import urllib2 
from bs4 import BeautifulSoup 

f = open('nhlstats.csv', "w") 


groups=['points', 'shooting', 'goaltending', 'defensive', 'timeonice', 'faceoffs', 'minor-penalties', 'major-penalties'] 

year = ["2016", "2015","2014","2013","2012"] 

for yr in year: 
    for gr in groups: 
     url = "http://www.espn.com/nhl/statistics/player/_/stat/points/year/"+str(yr) 
    #www.espn.com/nhl/statistics/player/_/stat/points/year/2014/ 
    page = urllib2.urlopen(url) 
    soup=BeautifulSoup(page, "html.parser") 
    pagecount = soup.findAll(attrs= {"class":"page-numbers"})[0].string 
    pageliteral = int(pagecount[5:]) 
    for i in range(0,pageliteral): 
     number = int(((i*40) + 1)) 
     URL = "http://www.espn.com/nhl/statistics/player/_/stat/points/sort/points/year/"+str(yr) + "/count/"+str(number) 
     page = urllib2.urlopen(url) 
     soup=BeautifulSoup(page, "html.parser") 
     for tr in soup.select("#my-players-table tr[class*=player]"): 
      row =[] 
      for ob in range(1,15): 
       player_info = tr('td')[ob].get_text(strip=True) 
       row.append(player_info) 
      f.write(str(yr) +","+",".join(row) + "\n") 

f.close() 

這個遍地得到相同的前40條記錄。

我嘗試使用this solution彷彿一個並沒有發現這樣做

prevLink = soup.select('a[rel="nofollow"]')[0] 
newurl = "http:" + prevLink.get('href') 

沒有更好的工作，但我不知道怎麼做循環，它前進的方式？可能只是累了，但我的循環仍然只是去下一組記錄和卡住那之一。請幫我解決我的環

UPDATE

我的格式是在複製粘貼丟失，我實際的代碼如下所示：

import csv 
import urllib2 
from bs4 import BeautifulSoup 

f = open('nhlstats.csv', "w") 


groups=['points', 'shooting', 'goaltending', 'defensive', 'timeonice', 'faceoffs', 'minor-penalties', 'major-penalties'] 

year = ["2016", "2015","2014","2013","2012"] 


for yr in year: 
    for gr in groups: 
     url = "http://www.espn.com/nhl/statistics/player/_/stat/points/year/"+str(yr) 
    #www.espn.com/nhl/statistics/player/_/stat/points/year/2014/ 
     page = urllib2.urlopen(url) 
     soup=BeautifulSoup(page, "html.parser") 
     pagecount = soup.findAll(attrs= {"class":"page-numbers"})[0].string 
     pageliteral = int(pagecount[5:]) 
     for i in range(0,pageliteral): 
      number = int(((i*40) + 1)) 
      URL = "http://www.espn.com/nhl/statistics/player/_/stat/points/sort/points/year/"+str(yr) + "/count/"+str(number) 
      page = urllib2.urlopen(url) 
      soup=BeautifulSoup(page, "html.parser") 
      for tr in soup.select("#my-players-table tr[class*=player]"): 
       row =[] 
       for ob in range(1,15): 
        player_info = tr('td')[ob].get_text(strip=True) 
        row.append(player_info) 
       f.write(str(yr) +","+",".join(row) + "\n") 

f.close() 

Answer 1

您的代碼縮進主要是有過錯的。另外，實際使用您導入的CSV庫會很明智，它會自動將玩家名稱包含在引號中，以避免任何逗號破壞csv結構。

這可以通過查找指向下一頁的鏈接並提取開始計數。然後用它來構建你的下一頁get。如果找不到下一頁，則轉到下一年的組。請注意，計數不是頁數，而是起始條目數。

import csv 
import urllib2 
from bs4 import BeautifulSoup 


groups= ['points', 'shooting', 'goaltending', 'defensive', 'timeonice', 'faceoffs', 'minor-penalties', 'major-penalties'] 
year = ["2016", "2015", "2014", "2013", "2012"] 

with open('nhlstats.csv', "wb") as f_output: 
    csv_output = csv.writer(f_output) 

    for yr in year: 
     for gr in groups: 
      start_count = 1 
      while True: 
       #print "{}, {}, {}".format(yr, gr, start_count)  # show progress 

       url = "http://www.espn.com/nhl/statistics/player/_/stat/points/sort/points/year/{}/count/{}".format(yr, start_count) 
       page = urllib2.urlopen(url) 
       soup = BeautifulSoup(page, "html.parser") 

       for tr in soup.select("#my-players-table tr[class*=player]"): 
        row = [yr] 
        for ob in range(1, 15): 
         player_info = tr('td')[ob].get_text(strip=True) 
         row.append(player_info) 

        csv_output.writerow(row) 

       try: 
        start_count = int(soup.find(attrs= {"class":"page-numbers"}).find_next('a')['href'].rsplit('/', 1)[1]) 
       except: 
        break 

使用with也會在最後自動關閉您的文件。

這會給你一個CSV文件開始，如下所示：

2016,"Patrick Kane, RW",CHI,82,46,60,106,17,30,1.29,287,16.0,9,17,20 
2016,"Jamie Benn, LW",DAL,82,41,48,89,7,64,1.09,247,16.6,5,17,13 
2016,"Sidney Crosby, C",PIT,80,36,49,85,19,42,1.06,248,14.5,9,10,14 
2016,"Joe Thornton, C",SJ,82,19,63,82,25,54,1.00,121,15.7,6,8,21 

Answer 2

您正在改變URL多次要打開前這是第一次，由於縮進錯誤。試試這個：

for gr in groups: url = "...some_url..." page = urllib2.urlopen(url) ...everything else should be indented....