從單擊按鈕時生成的網站獲取完整的源代碼

Question

我想要源代碼的網頁包含一些產品。

當我簡單地加載它時，它只給出了前幾個產品的源代碼。

我需要的是點擊後的完整源代碼顯示140-250項其中生成幾次，得到的代碼時，有沒有像顯示說明X - Y項目了。正在生成的按鈕的id是see-more-products。

我試過urllib，requests模塊，這些模塊並沒有如預期般完成工作。

指向該特定頁面的鏈接在註釋中。

任何幫助表示讚賞。謝謝:)

Answer 1

有幾種不同的方式，讓你開始

import requests 
from bs4 import BeautifulSoup 
for i in range(0, 3339, 48): 
    r = requests.get("http://www.snapdeal.com/acors/json/product/get/search/175/{}/48".format(i)) 
    print([a["href"] for a in BeautifulSoup(r.content).select("div.product-desc-rating.title-section-expand a")]) 

哪樣給你所有的手機鏈，其中一個樣本是：

['http://www.snapdeal.com/product/micromax-canvas-spark-2-4/658765871888', 'http://www.snapdeal.com/product/alpha-cod/639184667095', 'http://www.snapdeal.com/product/apple-iphone-5s-16-gb/1204769399', 'http://www.snapdeal.com/product/micromax-canvas-a1-aq4502-8/630310793485', 'http://www.snapdeal.com/product/infocus-m370-8gb/685769734109', 'http://www.snapdeal.com/product/lenovo-a1000-8gb-black/634810874852', 'http://www.snapdeal.com/product/infocus-m350-16gb/667944684160', 'http://www.snapdeal.com/product/motorola-moto-x-force-32gb/677683041597', 'http://www.snapdeal.com/product/infocus-m812i-16gb-silver/672166899922', 'http://www.snapdeal.com/product/xolo-black-1x-32gb/633488491545', 'http://www.snapdeal.com/product/meizu-m2-m578h-4g-16gb/632749055332', 'http://www.snapdeal.com/product/iphone-6s-16gb/663413326062', 'http://www.snapdeal.com/product/apple-iphone-6-16-gb/1270529654', 'http://www.snapdeal.com/product/apple-iphone-5s-16-gb/347830397', 'http://www.snapdeal.com/product/micromax-s301-4gb-black/668491526095', 'http://www.snapdeal.com/product/infocus-m260-zl2-8gb/619518605458', 'http://www.snapdeal.com/product/micromax-juice-2-aq5001-dual/625046032091', 'http://www.snapdeal.com/product/swipe-konnect-51-8gb-sand/626439596420', 'http://www.snapdeal.com/product/lenovo-a6000-black/670500124351', 'http://www.snapdeal.com/product/redmi-2-prime-16-gb/683138068892', 'http://www.snapdeal.com/product/asus-zenfone-go-8gb/646618265597', 'http://www.snapdeal.com/product/karbonn-aura-8gb-3g/627156357326', 'http://www.snapdeal.com/product/panasonic-eluga-turbo-32gb-marine/644981636428', 'http://www.snapdeal.com/product/asus-zenfone-2-laser-ze550kl/633082651487', 'http://www.snapdeal.com/product/redmi-2-8-gb-grey/628016512272', 'http://www.snapdeal.com/product/infocus-m680-16gb-4g/649492687203', 'http://www.snapdeal.com/product/lava-iris-atom-8gb-white/665532304471', 'http://www.snapdeal.com/product/micromax-x073-black/632400291797', 'http://www.snapdeal.com/product/intex-aqua-3g-pro-4gb/661217697971', 'http://www.snapdeal.com/product/micromax-x605-black/663717815370', 'http://www.snapdeal.com/product/micromax-x088-red/550645299', 'http://www.snapdeal.com/product/micromax-unite-3-q372/639750556852', 'http://www.snapdeal.com/product/micromax-canvas-selfie-2-q340/673685061115', 'http://www.snapdeal.com/product/samsung-j7-16gb-espresso-brown/661359071561', 'http://www.snapdeal.com/product/hpl-a50-white-dual-core/752819001', 'http://www.snapdeal.com/product/intex-intex-aqua-power-hd/677955090549', 'http://www.snapdeal.com/product/samsung-galaxy-core-2-4gb/95934411', 'http://www.snapdeal.com/product/samsung-galaxy-j2-8gb/637947695307', 'http://www.snapdeal.com/product/micromax-bolt-x088/1252626810', 'http://www.snapdeal.com/product/lenovo-a7000-turbo-16gb-matte/676482356343', 'http://www.snapdeal.com/product/nokia-105-dual-sim-black/625877378012', 'http://www.snapdeal.com/product/lenovo-vibe-x2ap/619606468183', 'http://www.snapdeal.com/product/xolo-hive-8x1000i-16-gb/620631303689', 'http://www.snapdeal.com/product/panasonic-p41-black-android-dual/1202820682', 'http://www.snapdeal.com/product/asus-zenfone-2-laser-ze500kl/686932521097', 'http://www.snapdeal.com/product/samsung-galaxy-grand-max/647437791381', 'http://www.snapdeal.com/product/panasonic-p55-novo-8-gb/630374183875', 'http://www.snapdeal.com/product/samsung-tizen-z1/214936653'] 
['http://www.snapdeal.com/product/micromax-canvas-juice-3-q392/675338203115', 'http://www.snapdeal.com/product/micromax-x853-256-mb-gray/678836590088', 'http://www.snapdeal.com/product/iball-prince-2-black/683878915532', 'http://www.snapdeal.com/product/intex-cloud-swift-4g-16gb/651487709065', 'http://www.snapdeal.com/product/htc-desire-820-white/2096877771', 'http://www.snapdeal.com/product/celkon-2gb-star-16gb/668257971604', 'http://www.snapdeal.com/product/lava-iris-atom-8gb-black/671938280548', 'http://www.snapdeal.com/product/asus-zenfone-max-zc550kl-16gb/659567821492', 'http://www.snapdeal.com/product/micromax-x084/1189906548', 'http://www.snapdeal.com/product/infocus-bingo-21-m430-8gb/685620250128', 'http://www.snapdeal.com/product/xolo-black-4g-16gb-with/670501820959', 'http://www.snapdeal.com/product/nokia-130-dual-sim-black/1238158129', 'http://www.snapdeal.com/product/phicomm-e670-energy-2-16gb/642536836073', 'http://www.snapdeal.com/product/lenovo-a7000-8-gbwhite/660909822842', 'http://www.snapdeal.com/product/infocus-m260-8gb-yellow-3g/658001307243', 'http://www.snapdeal.com/product/gionee-m5l-32gb-golden-4g/664586124833', 'http://www.snapdeal.com/product/spice-gaming-mobile-x2-red/673912591179', 'http://www.snapdeal.com/product/lenovo-lenovo-a1000-8gb-white/656199391476', 'http://www.snapdeal.com/product/xolo-cube-50-8gb-black/681508479062', 'http://www.snapdeal.com/product/microsoft-lumia-540-8-gb/641216010217', 'http://www.snapdeal.com/product/infocus-m810-16gb-gold-4g/683493537817', 'http://www.snapdeal.com/product/samsung-guru-music-2-sm/672045180352', 'http://www.snapdeal.com/product/samsung-guru-music-2-duos/197236301', 'http://www.snapdeal.com/product/intex-cloud-cube-8gb-gray/678043979304', 'http://www.snapdeal.com/product/samsung-tizen-z3-8gb/638555381978', 'http://www.snapdeal.com/product/micromax-canvas-spark/634407752835', 'http://www.snapdeal.com/product/moto-e-2nd-gen-3g/627496057276', 'http://www.snapdeal.com/product/intex-cloud-breeze-8gb-grey/671476050074', 'http://www.snapdeal.com/product/samsung-j5-8gb-espresso-brown/659785514599', 'http://www.snapdeal.com/product/lg-google-nexus-5-16/848745269', 'http://www.snapdeal.com/product/mi4i-16gb/654856488809', 'http://www.snapdeal.com/product/micromax-joy-1850-35-kb/673275470330', 'http://www.snapdeal.com/product/phicomm-energy-653-4g-8gb/621455892516', 'http://www.snapdeal.com/product/panasonic-eluga-icon-16gb/650547748150', 'http://www.snapdeal.com/product/lenovo-vibe-x2ap-32gb-white/660083187849', 'http://www.snapdeal.com/product/xolo-prime/683763827810', 'http://www.snapdeal.com/product/meizu-mx5-silver-black/618643364985', 'http://www.snapdeal.com/product/karbonn-titanium-s205-2gb-16gb/662465421207', 'http://www.snapdeal.com/product/micromax-canvas-nitro-4g-e455/667663492017', 'http://www.snapdeal.com/product/i-kall-k88-gray/659450396588', 'http://www.snapdeal.com/product/karbonn-mach-two-titanium-s360/653888615715', 'http://www.snapdeal.com/product/asus-zenfone-2-laser-55/681651791446', 'http://www.snapdeal.com/product/i-kall-k55-fruity-orange/621445557898', 'http://www.snapdeal.com/product/yu-yunique-8gb/621238835100', 'http://www.snapdeal.com/product/htc-desire-326g-dual-sim/673825142381', 'http://www.snapdeal.com/product/micromax-h375-below-256-mb/626813860848', 'http://www.snapdeal.com/product/intex-intex-aqua-star-2/677150759819', 'http://www.snapdeal.com/product/micromax-canvas-nitro-2/674813358269'] 
['http://www.snapdeal.com/product/karbonn-s-15-plus-8gb/678547994280', 'http://www.snapdeal.com/product/huawei-honor-bee-8gb-3g/623753457685', 'http://www.snapdeal.com/product/intex-cloud-m6-16-gb/652438065304', 'http://www.snapdeal.com/product/karbonn-s310/2046781351', 'http://www.snapdeal.com/product/htc-desire-826/673132800168', 'http://www.snapdeal.com/product/oneplus-one-64gb/683926070447', 'http://www.snapdeal.com/product/samsung-galaxy-grand-prime-4g/682950239423', 'http://www.snapdeal.com/product/panasonic-panasonic-eluga-switch-32gb/651502528631', 'http://www.snapdeal.com/product/infocus-f120-dual-sim-gsm/647132622326', 'http://www.snapdeal.com/product/samsung-j5-8gb-white/625632131578', 'http://www.snapdeal.com/product/micromax-canvas-juice-3-q394/647451126738', 'http://www.snapdeal.com/product/micromax-x610-black/677067716681', 'http://www.snapdeal.com/product/videocon-v1393-dual-sim-mobile/519151652', 'http://www.snapdeal.com/product/nokia-105/1397084', 'http://www.snapdeal.com/product/micromax-canvas-blaze-4g-q400/635578549162', 'http://www.snapdeal.com/product/lenovo-a-6000-shot-16gb/652479804547', 'http://www.snapdeal.com/product/micromax-x597-black/666857291079', 'http://www.snapdeal.com/product/lenovo-a2010-16gb-black/641561931090', 'http://www.snapdeal.com/product/micromax-x597-blue/635774497484', 'http://www.snapdeal.com/product/lenovo-a1000-8gb-white-3g/682465160925', 'http://www.snapdeal.com/product/oorie-gsm-4gb-gray/644950735714', 'http://www.snapdeal.com/product/videocon-v1429w-black/659876102140', 'http://www.snapdeal.com/product/wham-q4-8gb-black/683618702751', 'http://www.snapdeal.com/product/lenovo-s60-8gb-white/682858759745', 'http://www.snapdeal.com/product/lenovo-a6000-white-8gb-white/687038221568', 'http://www.snapdeal.com/product/k2-air-8gb-black/662606076269', 'http://www.snapdeal.com/product/infocus-m680-16gb-4g/623852692656', 'http://www.snapdeal.com/product/micromax-x707-gray/675166544662', 'http://www.snapdeal.com/product/micromax-bolt-a-24-champagne/318667695', 'http://www.snapdeal.com/product/htc-desire-620-g/2140463474', 'http://www.snapdeal.com/product/whitecherry-k9-gray/634412911071', 'http://www.snapdeal.com/product/intex-neo-v-plus-black/49904988', 'http://www.snapdeal.com/product/intex-aqua-power-hd-mobile/639191044460', 'http://www.snapdeal.com/product/i-kall-k88-black/624856889024', 'http://www.snapdeal.com/product/apple-iphone-6-64-gb/131798351', 'http://www.snapdeal.com/product/infocus-f135-dual-sim-gsm/662122536540', 'http://www.snapdeal.com/product/apple-iphone-5s-32-gb/936530455', 'http://www.snapdeal.com/product/micromax-bolt-q338-8gb-black/633509849227', 'http://www.snapdeal.com/product/micromax-x084-black/676316493911', 'http://www.snapdeal.com/product/infocus-f110-dual-sim-gsm/668147567472', 'http://www.snapdeal.com/product/kenxinda-watch-mobile-dual-sim/149800122', 'http://www.snapdeal.com/product/panasonic-eluga-s/1270286256', 'http://www.snapdeal.com/product/swipe-virtue-16gb-white-3g/633490528209', 'http://www.snapdeal.com/product/micromax-canvas-play-q355-dual/663710099854', 'http://www.snapdeal.com/product/samsung-tizen-z1-4gb-gold/683743955020', 'http://www.snapdeal.com/product/asus-zenfone-go-45-zc451tg/632893288226', 'http://www.snapdeal.com/product/karbonn-a6-turbo/677649317861', 'http://www.snapdeal.com/product/karbonn-titanium-delight-s22-blackgrey/679556902984'] 

你可能有稍微調整這一點，但這應該接近獲取所有你想要的鏈接：

import requests 

from bs4 import BeautifulSoup 
def get_all_links(): 
    r = requests.get("http://www.snapdeal.com/acors/json/product/get/search/175/0/48") 
    total = int((BeautifulSoup(r.content).select("div.jsNumberFound.hidden")[0].text)) 
    yield from (a["href"] for a in BeautifulSoup(r.content).select("div.product-desc-rating.title-section-expand a")) 
    print("There are {} total items".format(total)) 
    for pge in range(50, total+1, 48): 
     r = requests.get("http://www.snapdeal.com/acors/json/product/get/search/175/{}/48".format(pge)) 
     yield from (a["href"] for a in BeautifulSoup(r.content).select("div.product-desc-rating.title-section-expand a")) 


There are 3340 total items 
http://www.snapdeal.com/product/micromax-canvas-spark/634407752835 
http://www.snapdeal.com/product/iphone-6s-16gb/663413326062 
http://www.snapdeal.com/product/apple-iphone-5s-16-gb/1204769399 
http://www.snapdeal.com/product/moto-e-2nd-gen-3g/627496057276 
http://www.snapdeal.com/product/redmi-2-prime-16-gb/683138068892 
http://www.snapdeal.com/product/micromax-juice-2-aq5001-dual/625046032091 
http://www.snapdeal.com/product/alpha-cod/639184667095 
http://www.snapdeal.com/product/intex-cloud-breeze-8gb-grey/671476050074 
http://www.snapdeal.com/product/micromax-canvas-spark-2-4/658765871888 
http://www.snapdeal.com/product/infocus-m370-8gb/685769734109 
http://www.snapdeal.com/product/infocus-m350-16gb/667944684160 
http://www.snapdeal.com/product/lg-google-nexus-5-16/848745269 
http://www.snapdeal.com/product/lenovo-a6000-black/670500124351 
http://www.snapdeal.com/product/apple-iphone-6-16-gb/1270529654 
http://www.snapdeal.com/product/meizu-m2-m578h-4g-16gb/632749055332 
http://www.snapdeal.com/product/micromax-canvas-a1-aq4502-8/630310793485 
http://www.snapdeal.com/product/apple-iphone-5s-16-gb/347830397 
http://www.snapdeal.com/product/infocus-bingo-21-m430-8gb/685620250128 
http://www.snapdeal.com/product/samsung-j7-16gb-espresso-brown/661359071561 
http://www.snapdeal.com/product/apple-iphone-6-64-gb/131798351 
http://www.snapdeal.com/product/panasonic-p55-novo-8-gb/630374183875 
http://www.snapdeal.com/product/lenovo-a1000-8gb-black/634810874852 
http://www.snapdeal.com/product/infocus-m260-zl2-8gb/619518605458 
http://www.snapdeal.com/product/asus-zenfone-2-laser-ze500kl/686932521097 
http://www.snapdeal.com/product/meizu-mx5-silver-black/618643364985 
http://www.snapdeal.com/product/moto-e-2nd-gen-4g/682444504031 
http://www.snapdeal.com/product/xolo-black-1x-32gb/633488491545 
http://www.snapdeal.com/product/panasonic-eluga-turbo-32gb-marine/644981636428 
http://www.snapdeal.com/product/mi4i-16gb/654856488809 
http://www.snapdeal.com/product/samsung-j5-8gb-espresso-brown/659785514599 
http://www.snapdeal.com/product/intex-cloud-swift-4g-16gb/651487709065 
http://www.snapdeal.com/product/lenovo-vibe-x2ap-32gb-white/660083187849 
http://www.snapdeal.com/product/panasonic-eluga-icon-16gb/650547748150 
http://www.snapdeal.com/product/micromax-s301-4gb-black/668491526095 
http://www.snapdeal.com/product/apple-iphone-5s-32-gb/936530455 
http://www.snapdeal.com/product/lenovo-a7000-turbo-16gb-matte/676482356343 
http://www.snapdeal.com/product/samsung-tizen-z3-8gb/638555381978 
http://www.snapdeal.com/product/oneplus-one-64gb/683926070447 
http://www.snapdeal.com/product/asus-zenfone-2-laser-55/681651791446 
http://www.snapdeal.com/product/htc-desire-826/673132800168 
http://www.snapdeal.com/product/gionee-m5l-32gb-golden-4g/664586124833 
http://www.snapdeal.com/product/intex-intex-aqua-power-hd/677955090549 
http://www.snapdeal.com/product/samsung-a7-2016-16gb-4g/632787468738 
http://www.snapdeal.com/product/micromax-unite-3-q372/639750556852