與Apache提卡解析HTML的問題

Question

我爬行網頁和爬行它提取該網頁的所有鏈接，然後我將嘗試使用下面的代碼，以便對某些URL，就來解析使用Apache提卡和BoilerPipe所有網址後解析得很好，但對於一些我得到這樣的錯誤。它顯示了上HTMLParser.java一些錯誤：行號102。這是在HTMLParser.java

String parsedText = tika.parseToString(htmlStream, md); 

我所提供的HTMLParse代碼也行號102。

org.apache.tika.exception.TikaException: TIKA-198: Illegal IOException from org.apache.tika.parser.microsoft.ooxml.OOXMLParser@67c28a6a 
     at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:203) 
     at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:197) 
     at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:135) 
     at org.apache.tika.Tika.parseToString(Tika.java:357) 
     at edu.uci.ics.crawler4j.crawler.HTMLParser.parse(HTMLParser.java:102) 
     at edu.uci.ics.crawler4j.crawler.WebCrawler.handleHtml(WebCrawler.java:227) 
     at edu.uci.ics.crawler4j.crawler.WebCrawler.processPage(WebCrawler.java:299) 
     at edu.uci.ics.crawler4j.crawler.WebCrawler.run(WebCrawler.java:118) 
     at java.lang.Thread.run(Unknown Source) 
Caused by: java.util.zip.ZipException: invalid block type 
     at java.util.zip.InflaterInputStream.read(Unknown Source) 
     at java.util.zip.ZipInputStream.read(Unknown Source) 
     at java.io.FilterInputStream.read(Unknown Source) 
     at org.apache.poi.openxml4j.util.ZipInputStreamZipEntrySource$FakeZipEntry.<init>(ZipInputStreamZipEntrySource.java:114) 
     at org.apache.poi.openxml4j.util.ZipInputStreamZipEntrySource.<init>(ZipInputStreamZipEntrySource.java:55) 
     at org.apache.poi.openxml4j.opc.ZipPackage.<init>(ZipPackage.java:82) 
     at org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:220) 
     at org.apache.poi.extractor.ExtractorFactory.createExtractor(ExtractorFactory.java:152) 
     at org.apache.tika.parser.microsoft.ooxml.OOXMLExtractorFactory.parse(OOXMLExtractorFactory.java:65) 
     at org.apache.tika.parser.microsoft.ooxml.OOXMLParser.parse(OOXMLParser.java:67) 
     at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:197) 
     ... 8 more 

這是我HTMLParser.java文件 -

public void parse(String htmlContent, String contextURL) { 

     InputStream htmlStream = null; 
     text = null; 
     title = null; 
     metaData = new HashMap<String, String>(); 

     urls = new HashSet<String>(); 
     char[] chars = htmlContent.toCharArray(); 

     bulletParser.setCallback(textExtractor); 
     bulletParser.parse(chars); 

     try { 
      text = articleExtractor.getText(htmlContent); 
     } catch (BoilerpipeProcessingException e) { 
      e.printStackTrace(); 
     } 

     if (text == null){ 
      text = textExtractor.text.toString().trim(); 
     } 

     title = textExtractor.title.toString().trim(); 
     try { 
      Metadata md = new Metadata(); 
      String utfHtmlContent = new String(htmlContent.getBytes(),"UTF-8"); 
      htmlStream = new ByteArrayInputStream(utfHtmlContent.getBytes()); 
    //The below line is at the line number 102 according to error above 
       String parsedText = tika.parseToString(htmlStream, md); 
       //very unlikely to happen 
       if (text == null){ 
        text = parsedText.trim(); 
       } 
       processMetaData(md); 
      } catch (Exception e) { 
       e.printStackTrace(); 
      } finally { 
       IOUtils.closeQuietly(htmlStream); 
      } 
      bulletParser.setCallback(linkExtractor); 
      bulletParser.parse(chars); 
      Iterator<String> it = linkExtractor.urls.iterator(); 

      String baseURL = linkExtractor.base(); 
      if (baseURL != null) { 
       contextURL = baseURL; 
      } 

      int urlCount = 0; 
      while 

(it.hasNext()) { 
      String href = it.next(); 
      href = href.trim(); 
      if (href.length() == 0) { 
       continue; 
      } 
      String hrefWithoutProtocol = href.toLowerCase(); 
      if (href.startsWith("http://")) { 
       hrefWithoutProtocol = href.substring(7); 
      } 
      if (hrefWithoutProtocol.indexOf("javascript:") < 0 
        && hrefWithoutProtocol.indexOf("@") < 0) { 
       URL url = URLCanonicalizer.getCanonicalURL(href, contextURL); 
       if (url != null) { 
        urls.add(url.toExternalForm()); 
        urlCount++; 
        if (urlCount > MAX_OUT_LINKS) { 
         break; 
        } 
       }    
      } 
     } 
    } 

任何建議將不勝感激。

Answer 1

聽起來像格式錯誤OOXML文檔（.docx，.XLSX等）。要檢查問題是否仍與最新版本蒂卡發生，你可以download的蒂卡-應用罐子，像這樣運行：

java -jar tika-app-1.0.jar --text http://url.of.the/troublesome/document.docx 

這應該打印出包含在文檔中的文本。如果不起作用，請將bug report與煩人文檔的URL一起提交（或者如果文檔不公開，請附上文檔）。

Answer 2

我有同樣的問題，我發現，我試圖解析文檔（DOCX）文件本來就不是簡單的文件，它的形式與標籤文本旁邊的文本和輸入字段的Microsoft Word開發。

我刪除的文件夾下的所有文件到Solr引擎解析和索引的休息後這樣的文件，它的工作。