用BufferedWriter在Java中獲取353個網頁的更快方法

Question

我檢查了其他鏈接，但是我找不到解決方案。 我可以連接到353個鏈接，並在大約7分鐘內抓取我需要的數據。我需要把時間縮短到不到1分鐘。

我在下面列入了我的代碼。

URL urlChartLink; 
URLConnection urlconn; 

try { 
    Class.forName(driver).newInstance(); 
    Connection mysqlconn = DriverManager.getConnection(url + dbName, userName, password); 
    Statement st1 = mysqlconn.createStatement(); 
    Connection mysqlconn2 = DriverManager.getConnection(url + dbName, userName, password); 
    ResultSet rs1 = st1.executeQuery(strSQL); 

    while (rs1.next()) 
    { 
     sElementID = rs1.getString(1); 
     sSymbol = rs1.getString(2); 
     sChartLink = rs1.getString(3); 

     urlChartLink = new URL(sChartLink); 
     urlconn = urlChartLink.openConnection(); 
     urlconn.addRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"); 

     sCurrentPrice = ""; 
     sPriceChange = ""; 

     try { 
      BufferedReader in = new BufferedReader(new InputStreamReader(urlconn.getInputStream(), "UTF-8"));     
      String currentLine; 

      int iLine = 0; 

      while ((currentLine = in.readLine()) != null) { 
       //Get data from page 
       iLine += 1; 

      } 
      in.close(); 
     } catch (IOException e) { 

    } 

    st1.close(); 

    mysqlconn.close(); 
    mysqlconn2.close(); 

} 

我試過了沒有URLConnection，但我得到了403錯誤。

如果有人能給我一個更好的解決方案，這將是偉大的！

Eddi Rae

Answer 1

對線程使用ExecutorService。此代碼不是100％語法錯誤免費，但它應該給你正確的想法。

Class.forName(driver).newInstance(); 
Connection mysqlconn = DriverManager.getConnection(url + dbName, userName, password); 
Statement st1 = mysqlconn.createStatement(); 
Connection mysqlconn2 = DriverManager.getConnection(url + dbName, userName, password); 
final ResultSet rs1 = st1.executeQuery(strSQL); 
ExecutorService service = ExecutorService.newFixedThreadPool(30); 

while (rs1.next()) 
{ 
    service.execute(new Runnable() { 
     public void run() { 
      String sElementID = rs1.getString(1); 
      String sSymbol = rs1.getString(2); 
      String sChartLink = rs1.getString(3); 

      URL urlChartLink = new URL(sChartLink); 
      URLConnection urlconn = urlChartLink.openConnection(); 
      urlconn.addRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"); 

      sCurrentPrice = ""; 
      sPriceChange = ""; 

      try { 
       BufferedReader in = new BufferedReader(new InputStreamReader(urlconn.getInputStream(), "UTF-8"));     
       String currentLine; 


       while ((currentLine = in.readLine()) != null) { 
        //Call syncronized method to perform operations that need to be thread safe 
        addLine() 

       } 
       in.close(); 
      } catch (IOException e) { 

      } 
     } 
    }); 
} 

executor.shutdown(); 
while (!executer.isShutdown() { 
    Thread.sleep(100); 
} 

st1.close(); 

mysqlconn.close(); 
mysqlconn2.close(); 

public void addLine() { 
    syncronized (OBJECT_LOCK) { 
     iLine++; 
    } 
}