2013-07-29 24 views
5

我有一個Web應用程序,用戶上傳圖片。我們通過ImageIO.read()驗證圖像數據,並在將其保存到磁盤之前對生成的BufferedImage執行一些簡單的轉換。ImageIO是同步的

在執行負載測試中,我們意識到,當許多要求在同一時間進來,他們被擋在ImageIO.read()調用。深入挖掘,我們注意到JPEGImageReader已同步,並且一次只創建一個BufferedImage。

有沒有人遇到過這個?我一直在使用谷歌搜索幾天,並沒有遇到過這個問題的另一個人,所以也許我做錯了什麼。我不能拿出任何合乎邏輯的理由。這似乎與不能爲每個圖像創建單獨的讀者和作者有關內存泄漏問題,但這種解釋似乎相當薄。

編輯:Here是一個性能工具,它打破了這麼久什麼是服用。我相信這是由於所有的線程都在等待同步鎖,JPEGImageReader源。

編輯:JAI庫會工作,除了OpenJDK的已刪除了它的關鍵部件,明確JPEG編解碼器的支持。

解決方案:給定的時間我花了試圖找到一個替代的解決方案,並沒有做這樣的量,我最好的解決辦法是異步處理的圖像,相對於該請求。因此,當請求進入時,原始圖像數據被存儲爲假定有效的圖像;那麼請求線程之外的異步進程將一次處理一個圖像。 由於ImageIO庫的同步性,嘗試一次執行多個操作沒有任何好處。考慮到庫不同步,圖像可以並行處理,效率很低。

一邊做處理異步地增加了複雜的程度,它可能是一個好主意,關於修改圖像。不起作用的是,我們無法在每個請求中處理原始圖像,這意味着我們的系統必須假設每個圖像都是有效的圖像數據。當異步處理器開始處理圖像時,如果數據不好,系統中可能會出現不一致。

+1

您在運行多線程?你是說多個'JPEGImageReader'實例在類上序列化(即單線程)?你怎麼確定這是事實?如果是這樣,他們都在鎖定/等待的顯示器在哪裏? –

+0

它絕對是多線程的,因爲它們是Tomcat中的單個請求。我開始相信ImageIO只是在底層創建一個底層的JPEGImageReader。請參閱:[JPEGImageReader](http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/com/sun/imageio/plugins/jpeg/JPEGImageReader.java)。我們99%確定這是發生了什麼事情,因爲當我們註釋掉創建BufferedImage並執行轉換並將InputStream內容保存到磁盤的部分時,CPU實際上正在徵稅。否則,他們在負載時的使用率約爲5%。 – user1236874

+1

聽起來你需要在ImageIO中下一層,並使用創建讀者的方法,而不是方便的方法。 – EJP

回答

2

在執行負載測試中,我們意識到,當許多要求在同一時間進來,他們被擋在ImageIO.read()調用。深入挖掘,我們注意到JPEGImageReader已同步,並且一次只創建一個BufferedImage。

有沒有人遇到過這個?

正如我在評論部分中提到:從性能上分析,它看起來像在OpenJDK的(LCMS)的色彩管理系統性能問題,因爲顏色轉換不應該採取這種長。我的猜測是,無法進一步調試,即使ImageIO能夠並行解碼,此問題也會使解碼(出現)同步。

這是一個SSCCE,它顯示ImageIO完美的能夠一次解碼多個圖像。

import javax.imageio.ImageIO; 
import javax.imageio.ImageReader; 
import javax.imageio.event.IIOReadProgressListener; 
import javax.imageio.stream.ImageInputStream; 
import java.io.File; 
import java.util.Iterator; 
import java.util.concurrent.ExecutorService; 
import java.util.concurrent.Executors; 
import java.util.concurrent.atomic.AtomicInteger; 

class MultipleJPEGDecoding { 
    private static int threads = Runtime.getRuntime().availableProcessors(); 
    private static ExecutorService executorService = Executors.newFixedThreadPool(threads * 4); 

    public static void main(final String[] args) throws InterruptedException { 
     for (int i = 0; i < 100; i++) { 
      final int index = i; 

      executorService.submit(new Runnable() { 
       public void run() { 
        try { 
         ImageInputStream stream = ImageIO.createImageInputStream(new File(args[index % args.length])); 
         try { 
          Iterator<ImageReader> readers = ImageIO.getImageReaders(stream); 
          if (!readers.hasNext()) { 
           System.err.println("No reader!"); 
           return; 
          } 

          ImageReader reader = readers.next(); 
          reader.setInput(stream); 
          reader.addIIOReadProgressListener(new ProgressListener(index)); 

          try { 
           reader.read(0); 
          } 
          finally { 
           reader.dispose(); 
          } 
         } 
         finally { 
          stream.close(); 
         } 
        } 
        catch (Exception e) { 
         System.err.printf("Error reading %d\n", index); 
         e.printStackTrace(); 
        } 
       } 
      }); 
     } 

     executorService.shutdown(); 
    } 

    static class ProgressListener implements IIOReadProgressListener { 
     final static AtomicInteger simultaneous = new AtomicInteger(0); 

     final int index; 
     int nextProgress = 25; 

     public ProgressListener(int index) { 
      this.index = index; 
     } 

     public void imageStarted(ImageReader source, int imageIndex) { 
      int inProgress = simultaneous.incrementAndGet(); 
      System.err.printf("Started reading image %d (now decoding %d images simultaneous)...\n", index, inProgress); 
     } 

     public void imageComplete(ImageReader source) { 
      int inProgress = simultaneous.decrementAndGet(); 
      System.err.printf("Done reading image %d%s.\n", index, inProgress > 0 ? String.format(" (still decoding %d other images)", inProgress) : ""); 
     } 

     public void imageProgress(ImageReader source, float percentageDone) { 
      if (percentageDone > nextProgress) { 
       int inProgress = simultaneous.get(); 
       System.err.printf("Progress on image %d (now decoding %d images simultaneous)...\n", index, inProgress); 
       nextProgress += 25; 
      } 
     } 

     public void sequenceStarted(ImageReader source, int minIndex) { 
     } 

     public void sequenceComplete(ImageReader source) { 
     } 

     public void thumbnailStarted(ImageReader source, int imageIndex, int thumbnailIndex) { 
     } 

     public void thumbnailProgress(ImageReader source, float percentageDone) { 
     } 

     public void thumbnailComplete(ImageReader source) { 
     } 

     public void readAborted(ImageReader source) { 
     } 
    } 
} 

輸出的樣子:

Started reading image 5 (now decoding 2 images simultaneous)... 
Started reading image 0 (now decoding 16 images simultaneous)... 
Started reading image 14 (now decoding 15 images simultaneous)... 
Started reading image 4 (now decoding 14 images simultaneous)... 
Started reading image 13 (now decoding 1 images simultaneous)... 
Started reading image 11 (now decoding 13 images simultaneous)... 
Started reading image 2 (now decoding 12 images simultaneous)... 
Started reading image 15 (now decoding 4 images simultaneous)... 
Started reading image 8 (now decoding 11 images simultaneous)... 
Started reading image 7 (now decoding 10 images simultaneous)... 
Started reading image 9 (now decoding 9 images simultaneous)... 
Started reading image 6 (now decoding 8 images simultaneous)... 
Started reading image 10 (now decoding 7 images simultaneous)... 
Started reading image 1 (now decoding 6 images simultaneous)... 
Started reading image 3 (now decoding 5 images simultaneous)... 
Started reading image 12 (now decoding 3 images simultaneous)... 
Progress on image 11 (now decoding 16 images simultaneous)... 
Progress on image 15 (now decoding 16 images simultaneous)... 
Progress on image 13 (now decoding 16 images simultaneous)... 
Progress on image 13 (now decoding 16 images simultaneous)... 
Progress on image 9 (now decoding 16 images simultaneous)... 
Progress on image 9 (now decoding 16 images simultaneous)... 
Progress on image 1 (now decoding 16 images simultaneous)... 
Progress on image 3 (now decoding 16 images simultaneous)... 
Progress on image 1 (now decoding 16 images simultaneous)... 
Progress on image 3 (now decoding 16 images simultaneous)... 
Progress on image 1 (now decoding 16 images simultaneous)... 
Progress on image 13 (now decoding 16 images simultaneous)... 
Progress on image 5 (now decoding 16 images simultaneous)... 
Progress on image 9 (now decoding 16 images simultaneous)... 
Progress on image 3 (now decoding 16 images simultaneous)... 
Done reading image 3 (still decoding 15 other images). 
Started reading image 16 (now decoding 15 images simultaneous)... 
Progress on image 11 (now decoding 15 images simultaneous)... 
Done reading image 13 (still decoding 14 other images). 
Started reading image 17 (now decoding 15 images simultaneous)... 
Progress on image 5 (now decoding 15 images simultaneous)... 
Progress on image 4 (now decoding 15 images simultaneous)... 
Progress on image 11 (now decoding 15 images simultaneous)... 
Done reading image 9 (still decoding 14 other images). 
Progress on image 5 (now decoding 15 images simultaneous)... 
Progress on image 17 (now decoding 15 images simultaneous)... 
Done reading image 11 (still decoding 14 other images). 
Started reading image 19 (now decoding 15 images simultaneous)... 
Progress on image 17 (now decoding 15 images simultaneous)... 
Done reading image 5 (still decoding 14 other images). 
Started reading image 18 (now decoding 15 images simultaneous)... 
Progress on image 15 (now decoding 15 images simultaneous)... 
Done reading image 1 (still decoding 14 other images). 
Started reading image 21 (now decoding 15 images simultaneous)... 
Progress on image 15 (now decoding 15 images simultaneous)... 
Progress on image 21 (now decoding 15 images simultaneous)... 
Done reading image 15 (still decoding 14 other images). 
Progress on image 6 (now decoding 14 images simultaneous)... 
Progress on image 21 (now decoding 14 images simultaneous)... 
Progress on image 7 (now decoding 14 images simultaneous)... 
Progress on image 21 (now decoding 14 images simultaneous)... 
Progress on image 7 (now decoding 14 images simultaneous)... 
Progress on image 19 (now decoding 14 images simultaneous)... 
Progress on image 19 (now decoding 14 images simultaneous)... 
Progress on image 7 (now decoding 14 images simultaneous)... 
Progress on image 19 (now decoding 14 images simultaneous)... 
Done reading image 7 (still decoding 13 other images). 
Started reading image 23 (now decoding 14 images simultaneous)... 
Progress on image 10 (now decoding 14 images simultaneous)... 
Progress on image 23 (now decoding 14 images simultaneous)... 
Progress on image 14 (now decoding 14 images simultaneous)... 
Started reading image 22 (now decoding 15 images simultaneous)... 
Progress on image 0 (now decoding 15 images simultaneous)... 
Done reading image 21 (still decoding 14 other images). 
Started reading image 24 (now decoding 15 images simultaneous)... 
Started reading image 20 (now decoding 16 images simultaneous)... 
Progress on image 2 (now decoding 16 images simultaneous)... 
Progress on image 8 (now decoding 16 images simultaneous)... 
Progress on image 17 (now decoding 15 images simultaneous)... 
Done reading image 17 (still decoding 14 other images). 
Started reading image 25 (now decoding 15 images simultaneous)... 
Progress on image 23 (now decoding 15 images simultaneous)... 
Done reading image 19 (still decoding 15 other images). 
Started reading image 26 (now decoding 16 images simultaneous)... 
Progress on image 23 (now decoding 16 images simultaneous)... 
Done reading image 23 (still decoding 15 other images). 
Started reading image 27 (now decoding 16 images simultaneous)... 
Progress on image 4 (now decoding 16 images simultaneous)... 
Progress on image 27 (now decoding 16 images simultaneous)... 
Progress on image 27 (now decoding 16 images simultaneous)... 
Progress on image 6 (now decoding 16 images simultaneous)... 
Progress on image 12 (now decoding 16 images simultaneous)... 
Progress on image 20 (now decoding 16 images simultaneous)... 
Progress on image 0 (now decoding 16 images simultaneous)... 
Progress on image 25 (now decoding 16 images simultaneous)... 
Progress on image 25 (now decoding 16 images simultaneous)... 
Progress on image 25 (now decoding 16 images simultaneous)... 
Progress on image 14 (now decoding 16 images simultaneous)... 
Progress on image 10 (now decoding 16 images simultaneous)... 
Progress on image 8 (now decoding 16 images simultaneous)... 
Progress on image 18 (now decoding 16 images simultaneous)... 
Done reading image 25 (still decoding 15 other images). 

[...]

Progress on image 75 (now decoding 12 images simultaneous)... 
Started reading image 73 (now decoding 13 images simultaneous)... 
Progress on image 75 (now decoding 13 images simultaneous)... 
Progress on image 73 (now decoding 13 images simultaneous)... 
Progress on image 75 (now decoding 13 images simultaneous)... 
Started reading image 74 (now decoding 14 images simultaneous)... 
Progress on image 66 (now decoding 14 images simultaneous)... 
Progress on image 64 (now decoding 14 images simultaneous)... 
Progress on image 73 (now decoding 14 images simultaneous)... 
Progress on image 60 (now decoding 14 images simultaneous)... 
Progress on image 74 (now decoding 14 images simultaneous)... 
Progress on image 58 (now decoding 14 images simultaneous)... 
Done reading image 75 (still decoding 13 other images). 
Progress on image 73 (now decoding 13 images simultaneous)... 
Started reading image 77 (now decoding 14 images simultaneous)... 
Done reading image 73 (still decoding 13 other images). 
Progress on image 77 (now decoding 13 images simultaneous)... 
Started reading image 78 (now decoding 14 images simultaneous)... 
Progress on image 60 (now decoding 14 images simultaneous)... 
Done reading image 48 (still decoding 13 other images). 
Progress on image 77 (now decoding 13 images simultaneous)... 
Started reading image 79 (now decoding 14 images simultaneous)... 
Started reading image 70 (now decoding 15 images simultaneous)... 
Progress on image 52 (now decoding 15 images simultaneous)... 
Progress on image 71 (now decoding 15 images simultaneous)... 
Started reading image 72 (now decoding 16 images simultaneous)... 
Progress on image 71 (now decoding 16 images simultaneous)... 
Progress on image 54 (now decoding 16 images simultaneous)... 
Progress on image 68 (now decoding 16 images simultaneous)... 
Progress on image 64 (now decoding 16 images simultaneous)... 
Progress on image 66 (now decoding 16 images simultaneous)... 
Progress on image 62 (now decoding 16 images simultaneous)... 
Progress on image 79 (now decoding 16 images simultaneous)... 
Progress on image 79 (now decoding 16 images simultaneous)... 
Progress on image 79 (now decoding 16 images simultaneous)... 
Progress on image 77 (now decoding 16 images simultaneous)... 
Progress on image 68 (now decoding 16 images simultaneous)... 
Done reading image 79 (still decoding 15 other images). 
Done reading image 77 (still decoding 14 other images). 
Started reading image 81 (now decoding 15 images simultaneous)... 
Progress on image 74 (now decoding 15 images simultaneous)... 
Progress on image 81 (now decoding 15 images simultaneous)... 
Progress on image 81 (now decoding 15 images simultaneous)... 
Progress on image 78 (now decoding 15 images simultaneous)... 
Done reading image 60 (still decoding 14 other images). 
Started reading image 82 (now decoding 15 images simultaneous)... 
Started reading image 80 (now decoding 16 images simultaneous)... 
Progress on image 76 (now decoding 14 images simultaneous)... 
Progress on image 66 (now decoding 14 images simultaneous)... 
Progress on image 70 (now decoding 14 images simultaneous)... 
Done reading image 52 (still decoding 14 other images). 
Done reading image 71 (still decoding 15 other images). 
Progress on image 81 (now decoding 16 images simultaneous)... 
Started reading image 84 (now decoding 15 images simultaneous)... 
Started reading image 83 (now decoding 16 images simultaneous)... 
Progress on image 58 (now decoding 16 images simultaneous)... 
Progress on image 83 (now decoding 16 images simultaneous)... 
Progress on image 83 (now decoding 16 images simultaneous)... 
Progress on image 83 (now decoding 16 images simultaneous)... 
Done reading image 81 (still decoding 15 other images). 
Started reading image 85 (now decoding 16 images simultaneous)... 
Progress on image 85 (now decoding 16 images simultaneous)... 
Progress on image 74 (now decoding 16 images simultaneous)... 
Done reading image 66 (still decoding 15 other images). 
Started reading image 86 (now decoding 16 images simultaneous)... 
Progress on image 64 (now decoding 16 images simultaneous)... 
Progress on image 70 (now decoding 16 images simultaneous)... 
Progress on image 78 (now decoding 16 images simultaneous)... 
Progress on image 54 (now decoding 16 images simultaneous)... 
Done reading image 58 (still decoding 15 other images). 
Started reading image 87 (now decoding 16 images simultaneous)... 
Progress on image 87 (now decoding 16 images simultaneous)... 
Progress on image 84 (now decoding 16 images simultaneous)... 
Progress on image 87 (now decoding 16 images simultaneous)... 
Done reading image 64 (still decoding 15 other images). 
Started reading image 88 (now decoding 16 images simultaneous)... 
Progress on image 76 (now decoding 16 images simultaneous)... 
Done reading image 83 (still decoding 15 other images). 
Progress on image 62 (now decoding 15 images simultaneous)... 
Progress on image 70 (now decoding 16 images simultaneous)... 
Progress on image 85 (now decoding 16 images simultaneous)... 
Started reading image 89 (now decoding 16 images simultaneous)... 
Progress on image 72 (now decoding 16 images simultaneous)... 
Progress on image 85 (now decoding 16 images simultaneous)... 
Progress on image 89 (now decoding 16 images simultaneous)... 
Done reading image 85 (still decoding 15 other images). 
Progress on image 89 (now decoding 15 images simultaneous)... 
Progress on image 82 (now decoding 15 images simultaneous)... 
Progress on image 80 (now decoding 15 images simultaneous)... 
Done reading image 74 (still decoding 14 other images). 
Started reading image 91 (now decoding 15 images simultaneous)... 
Started reading image 90 (now decoding 16 images simultaneous)... 
Done reading image 62 (still decoding 15 other images). 
Progress on image 87 (now decoding 15 images simultaneous)... 
Progress on image 68 (now decoding 15 images simultaneous)... 
Done reading image 87 (still decoding 14 other images). 
Progress on image 91 (now decoding 14 images simultaneous)... 
Started reading image 93 (now decoding 15 images simultaneous)... 
Progress on image 93 (now decoding 15 images simultaneous)... 
Progress on image 91 (now decoding 15 images simultaneous)... 
Progress on image 91 (now decoding 15 images simultaneous)... 
Progress on image 80 (now decoding 15 images simultaneous)... 
Started reading image 92 (now decoding 16 images simultaneous)... 
Done reading image 91 (still decoding 15 other images). 
Started reading image 94 (now decoding 16 images simultaneous)... 
Progress on image 93 (now decoding 16 images simultaneous)... 
Progress on image 72 (now decoding 16 images simultaneous)... 
Progress on image 93 (now decoding 16 images simultaneous)... 
Done reading image 93 (still decoding 15 other images). 
Started reading image 95 (now decoding 16 images simultaneous)... 
Progress on image 95 (now decoding 16 images simultaneous)... 
Progress on image 95 (now decoding 16 images simultaneous)... 
Progress on image 76 (now decoding 16 images simultaneous)... 
Progress on image 95 (now decoding 16 images simultaneous)... 
Progress on image 72 (now decoding 16 images simultaneous)... 
Done reading image 95 (still decoding 15 other images). 
Started reading image 96 (now decoding 16 images simultaneous)... 
Progress on image 94 (now decoding 16 images simultaneous)... 
Progress on image 89 (now decoding 15 images simultaneous)... 
Done reading image 89 (still decoding 14 other images). 
Done reading image 54 (still decoding 15 other images). 
Started reading image 97 (now decoding 14 images simultaneous)... 
Started reading image 98 (now decoding 15 images simultaneous)... 
Done reading image 70 (still decoding 13 other images). 
Started reading image 99 (now decoding 16 images simultaneous)... 
Progress on image 82 (now decoding 16 images simultaneous)... 
Progress on image 99 (now decoding 16 images simultaneous)... 
Progress on image 99 (now decoding 16 images simultaneous)... 
Progress on image 97 (now decoding 16 images simultaneous)... 
Progress on image 97 (now decoding 16 images simultaneous)... 
Progress on image 97 (now decoding 16 images simultaneous)... 
Done reading image 68 (still decoding 15 other images). 
Done reading image 97 (still decoding 14 other images). 
Progress on image 78 (now decoding 14 images simultaneous)... 
Progress on image 99 (now decoding 14 images simultaneous)... 
Done reading image 99 (still decoding 13 other images). 
Progress on image 86 (now decoding 13 images simultaneous)... 
Done reading image 72 (still decoding 12 other images). 
Progress on image 82 (now decoding 12 images simultaneous)... 
Progress on image 98 (now decoding 12 images simultaneous)... 
Progress on image 84 (now decoding 12 images simultaneous)... 
Progress on image 90 (now decoding 12 images simultaneous)... 
Done reading image 76 (still decoding 11 other images). 
Progress on image 92 (now decoding 11 images simultaneous)... 
Progress on image 80 (now decoding 11 images simultaneous)... 
Progress on image 94 (now decoding 11 images simultaneous)... 
Progress on image 88 (now decoding 11 images simultaneous)... 
Progress on image 84 (now decoding 11 images simultaneous)... 
Progress on image 90 (now decoding 11 images simultaneous)... 
Progress on image 92 (now decoding 11 images simultaneous)... 
Progress on image 86 (now decoding 11 images simultaneous)... 
Progress on image 94 (now decoding 10 images simultaneous)... 
Done reading image 84 (still decoding 9 other images). 
Progress on image 92 (now decoding 9 images simultaneous)... 
Done reading image 78 (still decoding 10 other images). 
Progress on image 88 (now decoding 9 images simultaneous)... 
Progress on image 90 (now decoding 9 images simultaneous)... 
Done reading image 80 (still decoding 8 other images). 
Done reading image 82 (still decoding 7 other images). 
Progress on image 86 (now decoding 7 images simultaneous)... 
Progress on image 96 (now decoding 7 images simultaneous)... 
Progress on image 88 (now decoding 7 images simultaneous)... 
Done reading image 90 (still decoding 6 other images). 
Done reading image 92 (still decoding 5 other images). 
Progress on image 98 (now decoding 5 images simultaneous)... 
Done reading image 94 (still decoding 4 other images). 
Done reading image 86 (still decoding 3 other images). 
Progress on image 96 (now decoding 3 images simultaneous)... 
Done reading image 88 (still decoding 2 other images). 
Progress on image 98 (now decoding 2 images simultaneous)... 
Progress on image 96 (now decoding 2 images simultaneous)... 
Done reading image 98 (still decoding 1 other images). 
Done reading image 96. 
+0

這並不能證明什麼。您已經證明的是,您可以將請求排入解碼器。你還沒有證明它們是同時解碼的。該類的源代碼清楚地表明這是不可能的。 – pstanton

+0

@pstanton源代碼表示*實例*是同步的(但API文檔已經指定讀者不被線程共享)。 ImageIO將使用不同的實例進行多次調用,因此它一次解碼多個圖像(如我的代碼所示)。請求沒有排隊,它們被同時解碼。我會更新代碼以使其更清晰。 – haraldK

+0

我承認我沒有測試過這個,但是記錄調用你的代碼永遠不會顯示真正發生的事情。測試這個的唯一方法是斷點'JPEGImageReader'的內部。我不想看起來像是捍衛我的職位,我不是,我希望你是對的。斷點第一次命中'JPEGImageReader.readInternal',看看它是否停止所有其他'simultanious'的讀取... – pstanton

4

編輯:首先,haraldK的回答是正確的,我的原始答案是錯誤的。它不能被刪除,但只有編輯OP應重新分配勾號。

的ThreadLock機制是有停止線程從自己的訪問不同的讀者,而不是獨立地和同時放訪問自己的讀者與其他線程停止線程。

因此ImageIO的JPEGImageReader應該可以非常好地工作。

+0

ps我知道這不是一個完整的「解決方案」,對此感到抱歉 – pstanton

+0

這很好。如果我得到它的工作,我會給你檢查並解決複選標記,並用解決方案更新我的問題。謝謝! – user1236874

+0

我認爲這是正確的,因爲正如我所能說的那樣,這是最優化的方法。問題在於OpenJDK,它不再支持JAI,但我發現所有其他庫都依賴於ImageIO。 – user1236874