如何在使用HttpWebRequest在C＃中獲取http響應時處理JavaScript？

Question

我有一個C＃代碼（它是一個Web應用程序，它託管在IIS上），我使用HttpWebRequest來獲取HttpWebResponse。我向任何網站提出請求&以字符串的形式得到響應，然後我分析響應字符串。但是最近我在頁面加載到瀏覽器中後，JavaScript執行數據提取的響應。

我試圖在firebug中調試這個&看到在響應的底部有一個JavaScript函數，它在頁面加載後更新dom元素。有什麼辦法可以在我的C＃代碼中做同樣的事情嗎？我在網上搜索了這個到現在爲止找不到的解決方案。

以下是我使用的代碼：

HttpWebResponse response = (HttpWebResponse)request.GetResponse(); 

     foreach (Cookie cook in response.Cookies) 
     { 
      Cookie cookie = new Cookie(); 
      cookie.Name = cook.Name; 
      cookie.Value = cook.Value; 
      cookie.Domain = cook.Domain; 
      cookie.Expires = DateTime.Now.AddDays(10); 
      cookieList.Add(cookie); 
     } 

     string postData = string.Format("username=" + txtUserID.Text + "&password=" + txtPwd.Text + "&url=https://example.com/&game="); 
     byte[] postBytes = Encoding.UTF8.GetBytes(postData); 
     HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://login.example.com/Login/authenticate"); 
     req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0"; 
     req.KeepAlive = true; 
     req.AutomaticDecompression = DecompressionMethods.GZip; 

     ////set the cookie 
     req.CookieContainer = new CookieContainer(); 
     foreach (Cookie cook in cookieList) 
     { 
      Cookie cookie = new Cookie(); 
      cookie.Name = cook.Name; 
      cookie.Value = cook.Value; 
      cookie.Domain = cook.Domain; 
      cookie.Expires = DateTime.Now.AddDays(10); 
      req.CookieContainer.Add(cookie); 
     } 

     req.Headers.Add("Accept-Encoding", "gzip, deflate"); 
     req.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");//en-GB,en-US;q=0.8,en;q=0.6 
     req.Method = "POST"; 
     req.Host = "login.example.com"; 
     req.Referer = "https://login.example.com/Login/logout"; 
     req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"; 

     req.ContentType = "application/x-www-form-urlencoded;"; 
     req.ContentLength = postBytes.Length; 

     //getting the request stream and posting data 
     StreamWriter requestwriter = new StreamWriter(req.GetRequestStream(), System.Text.Encoding.ASCII); 
     requestwriter.Write(postData); 
     requestwriter.Close(); 
     HttpWebResponse myHttpWebResponse = (HttpWebResponse)req.GetResponse(); 
     Stream responseStream = myHttpWebResponse.GetResponseStream(); 
     StreamReader myStreamReader = new StreamReader(responseStream, Encoding.ASCII); 
     string responseString = myStreamReader.ReadToEnd(); 
     myStreamReader.Close(); 
     responseStream.Close(); 
     myHttpWebResponse.Close(); 

Answer 1

我終於得到了簡單的解決方案，以我的需要。以下是我跟着鏈接： Link to tutorial

以下是會得到的結果代碼：

首先，您需要輸入以下內容：

using System.Drawing; 
using OpenQA.Selenium; 
using OpenQA.Selenium.PhantomJS; 
using System.Text.RegularExpressions; 
using System.IO; 
using HtmlAgilityPack; 

現在代碼：

 var options = new PhantomJSOptions(); 
     var driver = new PhantomJSDriver(options); 
     driver.Manage().Window.Size = new Size(1360, 728); 
     var size = driver.Manage().Window.Size; 

     driver.Navigate().GoToUrl("https://example.com/"); 
     string url = driver.Url; 
     //the driver can now provide you with what you need (it will execute the script) 
     //get the source of the page 
     var source = driver.PageSource; 
     //fully navigate the dom 
     var pathElement1 = driver.FindElementByName("username"); 
     var pathElement2 = driver.FindElementByName("password"); 
     var pathElement3 = driver.FindElementByXPath("//button[@class='SubmitButton']"); 

     pathElement1.Clear(); 
     pathElement1.SendKeys("username"); 
     pathElement2.Clear(); 
     pathElement2.SendKeys("password"); 
     pathElement3.Click(); 

     //Now get the response after login button click 
     source = driver.PageSource;