SAX解析器爲一個非常巨大的XML文件

Question

我正在處理一個非常巨大的XML文件，4 GB，我總是得到一個內存不足錯誤，我的java堆已經達到最大，這是爲什麼代碼：

Handler h1 = new Handler("post"); 
     Handler h2 = new Handler("comment"); 
     posts = new Hashtable<Integer, Posts>(); 
     comments = new Hashtable<Integer, Comments>(); 
     edges = new Hashtable<String, Edges>(); 
     try { 
       output = new BufferedWriter(new FileWriter("gephi.gdf")); 
       SAXParser parser = SAXParserFactory.newInstance().newSAXParser(); 
       SAXParser parser1 = SAXParserFactory.newInstance().newSAXParser(); 


       parser.parse(new File("G:\\posts.xml"), h1); 
       parser1.parse(new File("G:\\comments.xml"), h2); 
      } catch (Exception ex) { 
       ex.printStackTrace(); 
      } 

    @Override 
     public void startElement(String uri, String localName, String qName, 
        Attributes atts) throws SAXException { 
       if(qName.equalsIgnoreCase("row") && type.equals("post")) { 
        post = new Posts(); 
        post.id = Integer.parseInt(atts.getValue("Id")); 
        post.postTypeId = Integer.parseInt(atts.getValue("PostTypeId")); 
        if (atts.getValue("AcceptedAnswerId") != null) 
         post.acceptedAnswerId = Integer.parseInt(atts.getValue("AcceptedAnswerId")); 
        else 
         post.acceptedAnswerId = -1; 
        post.score = Integer.parseInt(atts.getValue("Score")); 
        if (atts.getValue("OwnerUserId") != null) 
         post.ownerUserId = Integer.parseInt(atts.getValue("OwnerUserId")); 
        else 
         post.ownerUserId = -1; 
        if (atts.getValue("ParentId") != null) 
         post.parentId = Integer.parseInt(atts.getValue("ParentId")); 
        else 
         post.parentId = -1; 
       } 
       else if(qName.equalsIgnoreCase("row") && type.equals("comment")) { 
        comment = new Comments(); 
        comment.id = Integer.parseInt(atts.getValue("Id")); 
        comment.postId = Integer.parseInt(atts.getValue("PostId")); 
        if (atts.getValue("Score") != null) 
         comment.score = Integer.parseInt(atts.getValue("Score")); 
        else 
         comment.score = -1; 
        if (atts.getValue("UserId") != null) 
         comment.userId = Integer.parseInt(atts.getValue("UserId")); 
        else 
         comment.userId = -1; 
       } 
      } 



public void endElement(String uri, String localName, String qName) 
     throws SAXException { 
      if(qName.equalsIgnoreCase("row") && type.equals("post")){ 
       posts.put(post.id, post); 
       //System.out.println("Size of hash table is " + posts.size()); 
      }else if (qName.equalsIgnoreCase("row") && type.equals("comment")) 
       comments.put(comment.id, comment); 
     } 

有沒有什麼辦法可以優化這段代碼，使我不會耗盡內存？可能使用流？如果是的話，你會怎麼做？

Answer 1

SAX解析器對故障有效。

帖子，評論和邊緣HashMaps立即跳出我作爲潛在的問題。我懷疑你需要定期從內存中清空這些地圖以避免OOME。

Answer 2

看一看一個叫SaxDoMix http://www.devsphere.com/xml/saxdomix/

項目它可以讓你解析一個大的XML文件，並有返回解析DOM實體的某些元素。比購買SAX解析器更容易。