長篇解碼：如何處理原始UDP數據包，使他們可以通過解碼器過濾器在一個DirectShow源過濾
不久：
我如何才能通過H264/MPEG-4解碼器過濾器，解碼處理那些原始的UDP數據？ 任何人都可以清楚地確定與H264/MPEG流有關的步驟嗎？
額外的信息：
我可以能夠與這FFmpeg的......我真的不能弄清楚如何FFmpeg的過程中的原始數據，以便它可以由一個解碼器解碼。
和平的蛋糕！
1.獲取
正如我所看到的，你已經知道如何做到這一點的數據（啓動RTSP會話，設置一個RTP/AVP/UDP;unicast;運輸，並獲得用戶數據報）...但如果你是在懷疑，問。
無論傳輸（UDP或TCP）的數據格式主要是一樣的：
[RTP Header - 12bytes][Video data][RTP Data][$ - 1byte][Transport Channel - 1byte][RTP data length - 2bytes][RTP data]所以要從UDP獲取數據，您只需剝離表示RTP頭的前12個字節。但要小心，您需要它來獲取視頻時間信息，而對於MPEG4來說，這是包裝信息！
對於TCP，您需要先讀取第一個字節，直到獲得字節$。然後讀取下一個字節，這將是以下數據所屬的傳輸通道（當服務器響應SETUP請求時它說：Transport: RTP/AVP/TCP;unicast;interleaved=0-1這意味着VIDEO DATA將具有TRANSPORT_CHANNEL = 0並且VIDEO RTCP DATA將具有TRANSPORT_CHANNEL = 1）。你想得到VIDEO DATA，所以我們期望0 ...然後讀取一個短的（2字節），它表示後面的RTP數據的長度，所以讀取那麼多字節，現在和UDP一樣。
2.解包數據
H264和MPEG4數據通常被分組（SDP中有packetization-mode參數可以有值0，1和2他們每個人的手段，以及如何解包，你可以看到HERE），因爲一個端點可以通過稱爲MTU的TCP或UDP發送一定的網絡限制。它通常是1500字節或更少。所以如果視頻幀比這個大（通常是這樣），它需要被分段（打包）成MTU大小的片段。這可以通過編碼器/流媒體在TCP和UDP傳輸上完成，或者您可以在IP上中繼以在另一側對視頻幀進行分段和重新組合......如果您想要通過UDP平滑地出現容易出錯的視頻，首先要好得多和TCP。
H264：要檢查是否RTP數據（到達了UDP，或交織在TCP）舉行一個更大的H264視頻幀的片段，你必須知道什麼時候被打包的片段的外觀：
H264片段
First byte: [ 3 NAL UNIT BITS | 5 FRAGMENT TYPE BITS]
Second byte: [ START BIT | END BIT | RESERVED BIT | 5 NAL UNIT BITS]
Other bytes: [... VIDEO FRAGMENT DATA...]
現在，拿到第一視頻數據的字節數組稱爲Data並獲得以下信息：
int fragment_type = Data[0] & 0x1F;
int nal_type = Data[1] & 0x1F;
int start_bit = Data[1] & 0x80;
int end_bit = Data[1] & 0x40;
如果fragment_type == 28那麼其後的視頻數據代表視頻幀片段。下一個檢查是start_bit集，如果是，那麼該片段是序列中的第一個。通過從第一個有效載荷字節（3 NAL UNIT BITS）取前3個位並將它們與來自第二個有效載荷字節（5 NAL UNIT BITS）的最後5個位組合起來，您可以使用它來重建IDR的NAL字節，因此您將得到像這樣的字節[3 NAL UNIT BITS | 5 NAL UNIT BITS]。然後將該NAL字節首先寫入一個清零緩衝區，其中VIDEO FRAGMENT DATA來自該片段。
如果start_bit和end_bit爲0，則只需將VIDEO FRAGMENT DATA（跳過標識片段的前兩個有效負載字節）寫入緩衝區。
如果start_bit是0和end_bit是1，這意味着它是最後的片段，你只寫了VIDEO FRAGMENT DATA（跳過識別片段中的前兩個字節）的緩衝，現在你有你的視頻幀重建！
請記住，RTP數據在前12個字節中包含RTP頭，並且如果幀被分段，則永遠不會在碎片整理緩衝區中寫入前兩個字節，並且您需要重建NAL字節並先寫入。如果你在這裏搞砸了一些東西，那麼圖片將會是局部的（一半會變成灰色或黑色，否則你會看到文物）。
MPEG4： 這是一個容易的。您需要檢查RTP標題中的MARKER_BIT。如果視頻數據表示整個視頻幀，則該字節被設置（1），並且視頻數據的0是一個視頻幀片段。因此，爲了將其拆包，你需要看看MARKER_BIT是什麼。如果它是1那就是它，只是讀取視頻數據字節。
整個幀：
[MARKER = 1]
打包幀：
[MARKER = 0], [MARKER = 0], [MARKER = 0], [MARKER = 1]
具有MARKER_BIT=0是所述第一視頻幀的片段，所有其他跟隨包括第一次與MARKER_BIT=1首先包是相同視頻幀的片段。所以，你需要做的是：在解包緩衝
MARKER_BIT=0地方視頻數據MARKER_BIT=1到相同的緩衝3.解碼器的處理數據（NAL字節流）
當你有拆包視頻幀，你需要製作NAL字節流。它具有以下格式：
0x000001[SPS], 0x000001[PPS], 0x000001[VIDEO FRAME], 0x000001...0x000001[Visual Object Sequence Start], 0x000001[VIDEO FRAME]規則：
0x000001 3字節代碼前綴事情沒有編解碼器sprop-parameter-sets在SDP），以及用於MPEG4的VOS的幀（在SDP config參數）所以你需要建立用於H264一個配置緩衝器和MPEG4前面加上3個字節0x000001，先發送它，然後用相同的3個字節預先加載每個拆包的視頻幀，然後發送給解碼器。
如果你需要任何澄清只是評論... :)
利用UDP數據包，您可以接收H.264數據流的各個比特，您可以將它們解包爲H.264 NAL Units，而這些數據反過來說，您通常是從您的過濾器推入DirectShow管道。
NAL單元將被格式化爲DirectShow媒體樣本，也可能作爲媒體類型的一部分（SPS/PPS NAL單元）。
拆包步驟在RFC 6184 - RTP Payload Format for H.264 Video中描述。這是RTP流量的有效載荷部分，由RFC 3550 - RTP: A Transport Protocol for Real-Time Applications定義。
明確，但不是很短。
我有這個@https://net7mma.codeplex.com/
實現下面是相關代碼
/// <summary>
/// Implements Packetization and Depacketization of packets defined in <see href="https://tools.ietf.org/html/rfc6184">RFC6184</see>.
/// </summary>
public class RFC6184Frame : Rtp.RtpFrame
{
/// <summary>
/// Emulation Prevention
/// </summary>
static byte[] NalStart = { 0x00, 0x00, 0x01 };
public RFC6184Frame(byte payloadType) : base(payloadType) { }
public RFC6184Frame(Rtp.RtpFrame existing) : base(existing) { }
public RFC6184Frame(RFC6184Frame f) : this((Rtp.RtpFrame)f) { Buffer = f.Buffer; }
public System.IO.MemoryStream Buffer { get; set; }
/// <summary>
/// Creates any <see cref="Rtp.RtpPacket"/>'s required for the given nal
/// </summary>
/// <param name="nal">The nal</param>
/// <param name="mtu">The mtu</param>
public virtual void Packetize(byte[] nal, int mtu = 1500)
{
if (nal == null) return;
int nalLength = nal.Length;
int offset = 0;
if (nalLength >= mtu)
{
//Make a Fragment Indicator with start bit
byte[] FUI = new byte[] { (byte)(1 << 7), 0x00 };
bool marker = false;
while (offset < nalLength)
{
//Set the end bit if no more data remains
if (offset + mtu > nalLength)
{
FUI[0] |= (byte)(1 << 6);
marker = true;
}
else if (offset > 0) //For packets other than the start
{
//No Start, No End
FUI[0] = 0;
}
//Add the packet
Add(new Rtp.RtpPacket(2, false, false, marker, PayloadTypeByte, 0, SynchronizationSourceIdentifier, HighestSequenceNumber + 1, 0, FUI.Concat(nal.Skip(offset).Take(mtu)).ToArray()));
//Move the offset
offset += mtu;
}
} //Should check for first byte to be 1 - 23?
else Add(new Rtp.RtpPacket(2, false, false, true, PayloadTypeByte, 0, SynchronizationSourceIdentifier, HighestSequenceNumber + 1, 0, nal));
}
/// <summary>
/// Creates <see cref="Buffer"/> with a H.264 RBSP from the contained packets
/// </summary>
public virtual void Depacketize() { bool sps, pps, sei, slice, idr; Depacketize(out sps, out pps, out sei, out slice, out idr); }
/// <summary>
/// Parses all contained packets and writes any contained Nal Units in the RBSP to <see cref="Buffer"/>.
/// </summary>
/// <param name="containsSps">Indicates if a Sequence Parameter Set was found</param>
/// <param name="containsPps">Indicates if a Picture Parameter Set was found</param>
/// <param name="containsSei">Indicates if Supplementatal Encoder Information was found</param>
/// <param name="containsSlice">Indicates if a Slice was found</param>
/// <param name="isIdr">Indicates if a IDR Slice was found</param>
public virtual void Depacketize(out bool containsSps, out bool containsPps, out bool containsSei, out bool containsSlice, out bool isIdr)
{
containsSps = containsPps = containsSei = containsSlice = isIdr = false;
DisposeBuffer();
this.Buffer = new MemoryStream();
//Get all packets in the frame
foreach (Rtp.RtpPacket packet in m_Packets.Values.Distinct())
ProcessPacket(packet, out containsSps, out containsPps, out containsSei, out containsSlice, out isIdr);
//Order by DON?
this.Buffer.Position = 0;
}
/// <summary>
/// Depacketizes a single packet.
/// </summary>
/// <param name="packet"></param>
/// <param name="containsSps"></param>
/// <param name="containsPps"></param>
/// <param name="containsSei"></param>
/// <param name="containsSlice"></param>
/// <param name="isIdr"></param>
internal protected virtual void ProcessPacket(Rtp.RtpPacket packet, out bool containsSps, out bool containsPps, out bool containsSei, out bool containsSlice, out bool isIdr)
{
containsSps = containsPps = containsSei = containsSlice = isIdr = false;
//Starting at offset 0
int offset = 0;
//Obtain the data of the packet (without source list or padding)
byte[] packetData = packet.Coefficients.ToArray();
//Cache the length
int count = packetData.Length;
//Must have at least 2 bytes
if (count <= 2) return;
//Determine if the forbidden bit is set and the type of nal from the first byte
byte firstByte = packetData[offset];
//bool forbiddenZeroBit = ((firstByte & 0x80) >> 7) != 0;
byte nalUnitType = (byte)(firstByte & Common.Binary.FiveBitMaxValue);
//o The F bit MUST be cleared if all F bits of the aggregated NAL units are zero; otherwise, it MUST be set.
//if (forbiddenZeroBit && nalUnitType <= 23 && nalUnitType > 29) throw new InvalidOperationException("Forbidden Zero Bit is Set.");
//Determine what to do
switch (nalUnitType)
{
//Reserved - Ignore
case 0:
case 30:
case 31:
{
return;
}
case 24: //STAP - A
case 25: //STAP - B
case 26: //MTAP - 16
case 27: //MTAP - 24
{
//Move to Nal Data
++offset;
//Todo Determine if need to Order by DON first.
//EAT DON for ALL BUT STAP - A
if (nalUnitType != 24) offset += 2;
//Consume the rest of the data from the packet
while (offset < count)
{
//Determine the nal unit size which does not include the nal header
int tmp_nal_size = Common.Binary.Read16(packetData, offset, BitConverter.IsLittleEndian);
offset += 2;
//If the nal had data then write it
if (tmp_nal_size > 0)
{
//For DOND and TSOFFSET
switch (nalUnitType)
{
case 25:// MTAP - 16
{
//SKIP DOND and TSOFFSET
offset += 3;
goto default;
}
case 26:// MTAP - 24
{
//SKIP DOND and TSOFFSET
offset += 4;
goto default;
}
default:
{
//Read the nal header but don't move the offset
byte nalHeader = (byte)(packetData[offset] & Common.Binary.FiveBitMaxValue);
if (nalHeader > 5)
{
if (nalHeader == 6)
{
Buffer.WriteByte(0);
containsSei = true;
}
else if (nalHeader == 7)
{
Buffer.WriteByte(0);
containsPps = true;
}
else if (nalHeader == 8)
{
Buffer.WriteByte(0);
containsSps = true;
}
}
if (nalHeader == 1) containsSlice = true;
if (nalHeader == 5) isIdr = true;
//Done reading
break;
}
}
//Write the start code
Buffer.Write(NalStart, 0, 3);
//Write the nal header and data
Buffer.Write(packetData, offset, tmp_nal_size);
//Move the offset past the nal
offset += tmp_nal_size;
}
}
return;
}
case 28: //FU - A
case 29: //FU - B
{
/*
Informative note: When an FU-A occurs in interleaved mode, it
always follows an FU-B, which sets its DON.
* Informative note: If a transmitter wants to encapsulate a single
NAL unit per packet and transmit packets out of their decoding
order, STAP-B packet type can be used.
*/
//Need 2 bytes
if (count > 2)
{
//Read the Header
byte FUHeader = packetData[++offset];
bool Start = ((FUHeader & 0x80) >> 7) > 0;
//bool End = ((FUHeader & 0x40) >> 6) > 0;
//bool Receiver = (FUHeader & 0x20) != 0;
//if (Receiver) throw new InvalidOperationException("Receiver Bit Set");
//Move to data
++offset;
//Todo Determine if need to Order by DON first.
//DON Present in FU - B
if (nalUnitType == 29) offset += 2;
//Determine the fragment size
int fragment_size = count - offset;
//If the size was valid
if (fragment_size > 0)
{
//If the start bit was set
if (Start)
{
//Reconstruct the nal header
//Use the first 3 bits of the first byte and last 5 bites of the FU Header
byte nalHeader = (byte)((firstByte & 0xE0) | (FUHeader & Common.Binary.FiveBitMaxValue));
//Could have been SPS/PPS/SEI
if (nalHeader > 5)
{
if (nalHeader == 6)
{
Buffer.WriteByte(0);
containsSei = true;
}
else if (nalHeader == 7)
{
Buffer.WriteByte(0);
containsPps = true;
}
else if (nalHeader == 8)
{
Buffer.WriteByte(0);
containsSps = true;
}
}
if (nalHeader == 1) containsSlice = true;
if (nalHeader == 5) isIdr = true;
//Write the start code
Buffer.Write(NalStart, 0, 3);
//Write the re-construced header
Buffer.WriteByte(nalHeader);
}
//Write the data of the fragment.
Buffer.Write(packetData, offset, fragment_size);
}
}
return;
}
default:
{
// 6 SEI, 7 and 8 are SPS and PPS
if (nalUnitType > 5)
{
if (nalUnitType == 6)
{
Buffer.WriteByte(0);
containsSei = true;
}
else if (nalUnitType == 7)
{
Buffer.WriteByte(0);
containsPps = true;
}
else if (nalUnitType == 8)
{
Buffer.WriteByte(0);
containsSps = true;
}
}
if (nalUnitType == 1) containsSlice = true;
if (nalUnitType == 5) isIdr = true;
//Write the start code
Buffer.Write(NalStart, 0, 3);
//Write the nal heaer and data data
Buffer.Write(packetData, offset, count - offset);
return;
}
}
}
internal void DisposeBuffer()
{
if (Buffer != null)
{
Buffer.Dispose();
Buffer = null;
}
}
public override void Dispose()
{
if (Disposed) return;
base.Dispose();
DisposeBuffer();
}
//To go to an Image...
//Look for a SliceHeader in the Buffer
//Decode Macroblocks in Slice
//Convert Yuv to Rgb
}
也有實現各種其他RFC有助於讓媒體在MediaElement或其他軟件中播放，或者只是將其保存到磁盤。
寫入容器格式正在進行中。
感謝您的詳細解釋...我會嘗試看看會發生什麼... – Novalis
它適用於H264 ...順便說一下，我應該檢查其他fragment_type比其他28 ... – Novalis
那麼如果它不是28，比它沒有打包片段！然後你就可以直接使用VIDEO DATA。投票？ ：D – Cipi