Beautifulsoup - 刮網頁 - 動態加載

Question

頁

我想刮一個網頁：https://www.justdial.com/Mumbai/Dairy-Product-Retailers-in-Thane/nct-10152687

我需要所有的商店名稱的數據，TEL-NUM和他們的地址

但我只能做到這一點高達10 原因加載其他項目需要滾動網頁

我的代碼：

import requests 
import bs4 

crawl_url = requests.get('https://www.justdial.com/Mumbai/Dairy-Product- 
Retailers-in-Thane/nct-10152687', headers={'User-Agent': 'Mozilla/5.0'}) 
crawl_url.raise_for_status() 


soup = bs4.BeautifulSoup(crawl_url.text, 'lxml') 

for elems in soup.find_all('span', class_="jcn"): 
    select_a = elems.select('a') 
    for links in select_a: 
     href = links.get('href') 
     res = requests.get(href, headers={'User-Agent': 'Mozilla/5.0'}) 

     xsoup = bs4.BeautifulSoup(res.text, 'lxml') 

     Name = xsoup.select('.fn') 
     tel = xsoup.select('.tel') 
     add = xsoup.select('.adrstxtr') 
     a = Name[0] 
     b = tel[0] 
     c = add[0] 
     print(a.getText()) 
     print("--"*10) 
     print(b.getText()) 
     print("--"*10) 
     print(c.getText()) 
     print("=="*25) 

當我們向下滾動霸GE等物品裝入了 所以我想知道如何得到數據的任何數字/項目我想

我試圖this

但沒有聲張很好理解的，也是我沒得到了POST方法：/

如果需要更多的信息告訴我的tmadam給出

Answer 1

解決方案工作 這裏是代碼

import requests 
import bs4 

def spider(max_pages): 
    page = 1 
    while page < max_pages: 
     url = "https://www.justdial.com/Mumbai/Dairy-Product-Retailers-in- 
Thane/nct-10152687/page-%s" % page 
     crawl_url = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'}) 
     crawl_url.raise_for_status() 
     soup = bs4.BeautifulSoup(crawl_url.text, 'lxml') 
     for elems in soup.find_all('span', class_="jcn"): 
      select_a = elems.select('a') 
      for links in select_a: 
       href = links.get('href') 
       res = requests.get(href, headers={'User-Agent': 
'Mozilla/5.0'}) 
       xsoup = bs4.BeautifulSoup(res.text, 'lxml') 
       Name = xsoup.select('.fn') 
       tel = xsoup.select('.tel') 
       add = xsoup.select('.adrstxtr') 
       a = Name[0] 
       b = tel[0] 
       c = add[0] 
       print(a.getText()) 
       print("--"*10) 
       print(b.getText()) 
       print("--"*10) 
       print(c.getText()) 
       print("=="*25) 
     page += 1 


spider(3)