Ruby Net :: HTTP :: Post提交表格

我正在創建一個網站刮板。有一種表單用於更改當前頁面。Ruby Net :: HTTP :: Post提交表格

這是我爲POST請求提交表單的方式，但它似乎一次又一次地提取相同的頁面。

下面是一些示例代碼：

pages = { 
"total_pages" => 19, 
"p1" => '1234/1456/78990/123324345/12143343214345/231432143/12432412/435435/', 
"p2" => '1432424/123421421/345/435/6/65/5/34/3/2/21/1243', 
.. 
.. 
..  
} 


idx = 1 
p_count = pages["total_pages"] 

#set up the HTTP request to change pages to get all the auction results 
uri = URI.parse("http://somerandomwebsite.com?listings") 
http = Net::HTTP.new(uri.host, uri.port) 
req = Net::HTTP::Post.new(uri.request_uri) 

p_count.times do 
    puts "On loop sequence: #{idx}" 
    pg_num = "p#{idx}" 
    pg_content = pages["#{pg_num}"] 
    req.set_form_data({"page" => "#{pg_num}", "#{pg_num}" => "#{pg_content}"}) 

    response = http.request(req) 
    page = Nokogiri::HTML(response.body) 
    idx = idx + 1 
end

它看起來像page永遠不會改變。有沒有辦法看到每次我想要確保正確的參數都通過時，完整的請求是什麼樣的？似乎幾乎不可能確定有關req的任何內容。

來源

2017-02-09 Zack Herbert

請閱讀「[mcve]」。你的代碼不會運行，我們必須改變它來測試以確定問題。這浪費了我們的時間。我建議不要使用Net :: HTTP，而應使用Ruby中存在的許多HTTP客戶端之一。如果你正在發明一種新的服務器類型，Net :: HTTP非常棒，但對於正常的HTTP工作來說，它是非常低級的，特別是當你只是請求頁面時。至於看到請求，http://httpbin.org可能非常有用。 –

一個偉大的方式來調試HTTP是採取http://httpbin.org優勢：

require 'net/http' 
uri = URI('http://httpbin.org/post') 
res = Net::HTTP.post_form(uri, 'q' => 'ruby', 'max' => '50') 
puts res.body

將返回：

# >> { 
# >> "args": {}, 
# >> "data": "", 
# >> "files": {}, 
# >> "form": { 
# >>  "max": "50", 
# >>  "q": "ruby" 
# >> }, 
# >> "headers": { 
# >>  "Accept": "*/*", 
# >>  "Accept-Encoding": "gzip;q=1.0,deflate;q=0.6,identity;q=0.3", 
# >>  "Content-Length": "13", 
# >>  "Content-Type": "application/x-www-form-urlencoded", 
# >>  "Host": "httpbin.org", 
# >>  "User-Agent": "Ruby" 
# >> }, 
# >> "json": null, 
# >> "origin": "216.69.191.1", 
# >> "url": "http://httpbin.org/post" 
# >> }

這就是說，我建議不使用網:: HTTP。 Ruby有很多優秀的HTTP客戶端，這將使編寫代碼變得更加容易。例如下面是使用同樣的事情HTTPClient：

require 'httpclient' 
clnt = HTTPClient.new 
res = clnt.post('http://httpbin.org/post', 'q' => 'ruby', 'max' => '50') 
puts res.body 

# >> { 
# >> "args": {}, 
# >> "data": "", 
# >> "files": {}, 
# >> "form": { 
# >>  "max": "50", 
# >>  "q": "ruby" 
# >> }, 
# >> "headers": { 
# >>  "Accept": "*/*", 
# >>  "Content-Length": "13", 
# >>  "Content-Type": "application/x-www-form-urlencoded", 
# >>  "Date": "Thu, 09 Feb 2017 20:03:57 GMT", 
# >>  "Host": "httpbin.org", 
# >>  "User-Agent": "HTTPClient/1.0 (2.8.3, ruby 2.4.0 (2016-12-24))" 
# >> }, 
# >> "json": null, 
# >> "origin": "216.69.191.1", 
# >> "url": "http://httpbin.org/post" 
# >> }

這是未經測試的代碼，因爲你沒有告訴我們遠遠不夠的，但它是在那裏我會開始做你在做什麼：

require 'httpclient' 

BASE_URL = 'http://somerandomwebsite.com?listings' 
PAGES = [ 
'1234/1456/78990/123324345/12143343214345/231432143/12432412/435435/', 
'1432424/123421421/345/435/6/65/5/34/3/2/21/1243', 
] 

clnt = HTTPClient.new 

PAGES.each.with_index(1) do |page, idx| 
    puts "On loop sequence: #{idx}" 

    response = clnt.post(BASE_URL, 'page' => idx, idx => page) 

    doc = Nokogiri::HTML(response.body) 
    # ... 
end

來源

2017-02-09 19:58:36

Ruby Net :: HTTP :: Post提交表格

回答

相關問題