Jsoup將內容保存到數據庫中

Question

我有一個url數組，我想從URL中存儲信息，我在數據庫中讀取它。我的問題是我的數據列表太大的URL如果讀取從存儲在數據庫中的上面的露珠序列化每個URL將需要時間。

我知道有一種方法可以使用線程來操作，但我不知道該怎麼做，請幫助我。或者任何你的方法

try { 
    String lstUrls = "http://www.java2s.com/Tutorials/Java/Scala/index.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0020__Scala_Variables.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0040__Scala_Variable_Declarations.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0060__Scala_Semicolons.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0080__Scala_Code_Blocks.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0090__Scala_Comments.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0100__Scala_Type_Hierarchy.htm\n"; 
    String[] urls = lstUrls.split("\n"); 
    for (String url : urls) { 
     Document doc = Jsoup.connect(url).userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36").get(); 
     Elements select = doc.select("div.row"); 
     String html = select.html(); 
     System.out.println(html); 
     /* 
     insert html to database 
     */ 
    } 
} catch (IOException ex) { 
    ex.printStackTrace(); 
} 

Answer 1

要使用多線程檢索數據，你可以做這樣的事情：

Executor ex = Executors.newFixedThreadPool(3); 
    String lstUrls = "http://www.java2s.com/Tutorials/Java/Scala/index.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0020__Scala_Variables.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0040__Scala_Variable_Declarations.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0060__Scala_Semicolons.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0080__Scala_Code_Blocks.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0090__Scala_Comments.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0100__Scala_Type_Hierarchy.htm\n"; 
    String[] urls = lstUrls.split("\n"); 
    for (final String url : urls) { 
     try { 
      ex.execute(new Runnable() { 
       @Override 
       public void run() { 
        try { 
         Document doc = Jsoup 
           .connect(url) 
           .userAgent(
             "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36") 
           .get(); 
         Elements select = doc.select("div.row"); 
         String html = select.html(); 
         System.out.println(html); 
         /* 
         * insert html to database 
         */ 
        } catch (Exception e) { 
         e.printStackTrace(); 
        } 
       } 
      }); 
     } catch (Exception e) { 
      e.printStackTrace(); 
     } 
    } 

這將使用3個線程同時處理的URL，如果你想使用更多然後3個線程更改此行Executor ex = Executors.newFixedThreadPool(3);並用您想要的任何數字替換3。

你可以找到更多關於Executors here

Answer 2

我建議在插入數據庫之前壓縮數據。

//PreparedStatement.setBytes(1,compress(html)); 

public static byte[] compress(String str) throws Exception { 
    if (str == null || str.length() == 0) { 
     return null; 
    } 
    ByteArrayOutputStream obj = new ByteArrayOutputStream(); 
    GZIPOutputStream gzip = new GZIPOutputStream(obj); 
    gzip.write(str.getBytes("UTF-8")); 
    gzip.close(); 
    return obj.toByteArray(); 
} 

public static String decompress(byte[] bytes) throws Exception { 
    GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(bytes)); 
    BufferedReader bf = new BufferedReader(new InputStreamReader(gis,"UTF-8")); 
    String outStr = ""; 
    String line; 
    while ((line = bf.readLine()) != null) { 
     outStr += line; 
    } 
    return outStr; 
} 

第二種方法，將html數據保存到一個文件中並且只在數據庫中存儲文件路徑。

long ts = System.currentTimeMillis(); 
String filePath = String.valueOf(ts)+".gz"; 
saveToFile(filePath ,html); 
--------  
public static void saveToFile(String filePath, String text) { 
    try { 
     GZIPOutputStream gzos = new GZIPOutputStream(new FileOutputStream(filePath)); 
     gzos.write(text.getBytes("UTF-8")); 
     gzos.finish(); 
     gzos.close(); 

    } catch (IOException ex) { 
     ex.printStackTrace(); 
    } 
}