如何使用MediaCodec從視頻獲取位圖（幀）

Question

我試圖從使用MediaCodec的視頻文件獲取所有幀。如果我嘗試在SurfaceView上顯示視頻，一切都可以。但是如果surface爲null，並且當我嘗試從字節數組中獲取Bitmap時，alwaus會獲得null或運行時異常。

這是我的代碼：

private class PlayerThread extends Thread { 
    private MediaExtractor extractor; 
    private MediaCodec decoder; 
    private Surface surface; 

    public PlayerThread(Surface surface) { 
     this.surface = surface; 
    } 

    @Override 
    public void run() { 
     extractor = new MediaExtractor(); 
     extractor.setDataSource(videoPath); 

     for (int i = 0; i < extractor.getTrackCount(); i++) { 
      MediaFormat format = extractor.getTrackFormat(i); 
      String mime = format.getString(MediaFormat.KEY_MIME); 
      if (mime.startsWith("video/")) { 
       extractor.selectTrack(i); 
       decoder = MediaCodec.createDecoderByType(mime); 
       decoder.configure(format, /*surface*/ null, null, 0); 
       break; 
      } 
     } 

     if (decoder == null) { 
      Log.e("DecodeActivity", "Can't find video info!"); 
      return; 
     } 

     decoder.start(); 

     ByteBuffer[] inputBuffers = decoder.getInputBuffers(); 
     ByteBuffer[] outputBuffers = decoder.getOutputBuffers(); 
     BufferInfo info = new BufferInfo(); 
     boolean isEOS = false; 

     while (!Thread.interrupted()) { 
      ++numFrames; 
      if (!isEOS) { 
       int inIndex = decoder.dequeueInputBuffer(10000); 
       if (inIndex >= 0) { 
        ByteBuffer buffer = inputBuffers[inIndex]; 
        int sampleSize = extractor.readSampleData(buffer, 0); 
        if (sampleSize < 0) { 
         // We shouldn't stop the playback at this point, 
         // just pass the EOS 
         // flag to decoder, we will get it again from the 
         // dequeueOutputBuffer 
         Log.d("DecodeActivity", 
           "InputBuffer BUFFER_FLAG_END_OF_STREAM"); 
         decoder.queueInputBuffer(inIndex, 0, 0, 0, 
           MediaCodec.BUFFER_FLAG_END_OF_STREAM); 
         isEOS = true; 
        } else { 
         decoder.queueInputBuffer(inIndex, 0, sampleSize, 
           extractor.getSampleTime(), 0); 
         extractor.advance(); 
        } 
       } 
      } 

      int outIndex = decoder.dequeueOutputBuffer(info, 10000); 
      switch (outIndex) { 
      case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED: 
       Log.d("DecodeActivity", "INFO_OUTPUT_BUFFERS_CHANGED"); 
       outputBuffers = decoder.getOutputBuffers(); 
       break; 
      case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED: 
       Log.d("DecodeActivity", 
         "New format " + decoder.getOutputFormat()); 
       break; 
      case MediaCodec.INFO_TRY_AGAIN_LATER: 
       Log.d("DecodeActivity", "dequeueOutputBuffer timed out!"); 
       break; 
      default: 
       // I tried get Bitmap on few ways 
       //1. 
       //ByteBuffer buffer = outputBuffers[outIndex]; 
       //byte[] ba = new byte[buffer.remaining()]; 
       //buffer.get(ba); 
       //final Bitmap bmp = BitmapFactory.decodeByteArray(ba, 0, ba.length);// this return null 
       //2. 
       //ByteBuffer buffer = outputBuffers[outIndex]; 
       //final Bitmap bmp = Bitmap.createBitmap(1920, 1080, Config.ARGB_8888);//using MediaFormat object I know width and height 
       //int a = bmp.getByteCount(); //8294400 
       //buffer.rewind(); 
       //int b = buffer.capacity();//3137536 
       //int c = buffer.remaining();//3137536 
       //bmp.copyPixelsFromBuffer(buffer); // java.lang.RuntimeException: Buffer not large enough for pixels 
       //I know what exception mean, but i don't know why xception occurs 

       //In this place I need bitmap 

       // We use a very simple clock to keep the video FPS, or the 
       // video 
       // playback will be too fast 
       while (info.presentationTimeUs/1000 > System 
         .currentTimeMillis() - startMs) { 
        try { 
         sleep(10); 
        } catch (InterruptedException e) { 
         e.printStackTrace(); 
         break; 
        } 
       } 
       decoder.releaseOutputBuffer(outIndex, true); 
       break; 
      } 

      // All decoded frames have been rendered, we can stop playing 
      // now 
      if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) { 
       Log.d("DecodeActivity", 
         "OutputBuffer BUFFER_FLAG_END_OF_STREAM"); 
       break; 
      } 
     } 

     decoder.stop(); 
     decoder.release(); 
     extractor.release(); 
    } 
} 

我不知道如何解決它。

薩莫斯

Answer 1

有幾個問題與您的代碼（或可以說，與MediaCodec）。

首先，MediaCodec中的ByteBuffer處理很差，因此您必須從由dequeueOutputBuffer()填寫的BufferInfo對象中的值中手動設置緩衝區參數。

其次，來自MediaCodec的值是YUV格式，而不是RGB。從Android 4.4開始，Android框架不提供將輸出轉換爲位圖的功能。您需要提供您自己的YUV到RGB轉換器（複數 - 有多種格式）。一些設備使用專有的，未記錄的顏色格式。

您可以看到一個在CTS EncodeDecodeTest緩衝區到緩衝區方法（例如checkFrame()）中提取和檢查MediaCodec解碼器緩衝區內容的示例。

解決這個問題的一個更可靠的方法是返回到Surface的解碼，但用OpenGL ES提取像素。大牌ExtractMpegFramesTest顯示瞭如何做到這一點。

Answer 2

簡短的回答是：

在您的switch語句的默認部分，你需要重新設置ByteBuffer的位置，所以不是：

ByteBuffer buffer = outputBuffers[outIndex]; 
byte[] ba = new byte[buffer.remaining()]; 
buffer.get(ba); 

你應該有類似

ByteBuffer buffer = outputBuffers[outIndex]; 
buffer.position(info.offset); 
buffer.limit(info.offset + info.size); 
byte[] ba = new byte[buffer.remaining()]; 
buffer.get(ba); 

在您的原始代碼中，您會發現您的ByteBuffer的剩餘（）爲0.