#mediacodec

One performance issue we faced on Horizon for Android was that MediaMuxer's writeSampleData, would ocassionaly block for a couple of milliseconds to several seconds. The problem appeared on older devices and when writing to the SD card. In other words, when the medium's writing speed was low, MediaMuxer's buffering was not big enough to compensate for the delay and it would block until the data were actually written.

Before moving on with the solution, let's first present a simple approach on feeding MediaCodec's output to MediaMuxer:

This method is called when we drain MediaCodec's output (usually in a dedicated thread). We have to call releaseOutputBuffer as soon as possible, so that MediaCodec can reuse this buffer to encode new packets. If muxer delays, MediaCodec will eventually ran out of buffers and when we will try to enqueue new video or audio packets on the other end, those methods will block repsectively: SwapBuffers will block, dequeueInputBuffer will block or return -1.

The solution

We have to copy the encoded data to a temporary ByteBuffer and call writeSampleData asynchronously on a dedicated thread:

This approach works. However, it requires that we allocate a new ByteBuffer for every incoming packet. Allocating direct buffers has bigger overhead than allocating a simple ByteBuffer. Also, we are hoping that the garbage collector will kick in at some point and reclaim the memory of the unused temporary byte buffers. Looking at ADB's memory graph, it does.

However, since this method will run about 73 times/sec (for audio & video packets) it would be best if we could re-use the same memory somehow. The problem in implementing this approach, is that each video/audio packet has a different size. In order to make this work, we would have to allocate a large enough ByteBuffer that can hold a couple of seconds of encoded video/audio packets and write each new packet after the end of the previous one. Of course, we'll have to do all the bookkeeping for the actual position of each packet in the big ByteBuffer.

Towards this approach, I used Grafika's CircularBuffer and changed it so that it works for our case. You can find the resulting CircularBuffer here. First, we have to allocate the CircularBuffer using the MediaCodec's MediaFormat:

mCircularBuffer = new CircularBuffer(trackFormat, 2000); // allocate 2 seconds buffer

This is how it's used:

The above code, copies the incoming packet to the CircularBuffer and releases it almost instantly. Then, it asynchronosuly retrieves the packet from the CircularBuffer and feeds the MediaMuxer. There's always the possibility that our CircularBuffer is full. In this case, the add() method returns -1 and we just block until enough room is made.

I have also created a increaseSize() method, that allocates more space on Circular's buffer interal buffer. This way we don't have to block. Here's the updated code:

In the above code, when we can't add a packet, we try to increase its size instead of blocking. If we fail to do that because we are out of memory, we just block. Tha'ts the best approach because we can start with a small internal buffer and -if the device is slow- we gradually increase it as needed.