RTSP in Stagefright (2)

A basic sequence diagram of setting up RTSP connection in Stagefright:

[1] Real Time Streaming Protocol (RTSP)

RTSP in Stagefright (1)

RTSP support was added into Stagefright on GingerBread release:

• A preliminary foundation module, provides a generic way to asynchronously handle commands and events/messages sequentially in the schedule thread. Each message has a target id, indicating its corresponding handler, which is then registered in a looper. A looper spawns a thread, and schedules registered handlers to process any messages posted to it.
• ARTSPController acts as MediaExtractor for RTSP playback, which then delegates RTSP related stuff to ARTSPConnection, and payload parsing related to ARTPConnection/ARTPSource. ARTPSource leverages ARTPAssembler to re-assemble RTP packets to frames. AAMRAssembler, AH263Assembler, etc. all inherit from ARTPAssember according to the corresponding protocols. Buffers parsed are sent back via AMessage and queued in APacketSource, which acts as the MediaSource for downstream components.
• There is also ARTPSession and ARTPWriter which re-uses the existing RTP protocol for VOIP(Gtalk?) solution.

Native mediatest updated for FroYo

I had almost had to update the small program for every major release, so I've uploaded it to github for better version control:
git clone http://github.com/freepine/MediaTest.git

No wonder Android engineers keep warning us from accessing unpublished code:)

Media framework change in Froyo

Here is a Google I/O session by Dave Sparks, who talked about new APIs of media framework introduced in Froyo, and also some light on the future of Stagefright, OpenCORE, etc in the Q&A section.

Analyze memory leak of Android native process

Android libc_debug.so has a built-in function to dump all heap allocations with its backtrace, which is very useful to debug memory leaks of native processes. Below are the steps summarized during my investigation of mediaserver process:

1. apply the patch in ./frameworks/base, which registers a memory dumper service in mediaserver process, then rebuild
2. (*)flash new system.img, replace libc.so with libc_debug.so, then reboot
   
   • $ adb remount
   • $ adb shell mv /system/lib/libc_debug.so /system/lib/libc.so
   • $ adb reboot
3. run memorydumper to get the initial heap allocations of mediaserver process
   
   • $ adb shell /system/bin/memorydumper
4. play several files, save the process maps during playback, then get the memory dump again
   
   • $ adb pull /proc/<mediaserver_pid>/maps .
   • $ adb shell /system/bin/memorydumper
5. get the diff file of memory allocations
   
   • $ adb pull /data/memstatus_1136.0 .
   • $ adb pull /data/memstatus_1136.1 .
   • $ diff memstatus_1136.0 memstatus_1136.1 >diff_0_1
6. run the script to resolve symbols from the backtrace addresses in the diff file
   
   • $ ./addr2func.py --root-dir=../ --maps-file=./maps diff_0_1

[Update: 07/05/2010]
In Froyo release, there is no need to replace libc.so with libc_debug.so. Try below steps instead of original step 2:

• adb shell setprop libc.debug.malloc 1
• adb shell ps mediaserver
• adb shell kill <mediaserver_pid>

And I also updated the script to skip libc_debug.so while parsing symbols.

[Update: 06/22/2013]
Actually Android has a built-in service which could dump mediaserver's memory directly, so you can replace memorydumper related steps with below command. Sorry that I wasn't aware of this tool when I wrote this article :)
 #dumpsys "media.player" -m

Current status of Stagefright

• Local file playback (AMR/MP3/MP4/...) and basic http playback support;
• Video only recording;
• A corresponding metadata retriever, not fully implemented yet.

Some thoughts on the enhancement:

• No abstraction layer for media sink, and lack of a pluggable mechanism to pick decoder dynamically based on the source's format and sink's capability. The overall architecture doesn't seem to be flexible enough to date;
• It's good to re-use the same MediaExtractor from player engine for metadata retrieving to avoid duplication. However, MediaExtractor itself might not get all required keys reliably, e.g. it might be more accurate to get some keys (e.g, video height/width) from decoder's output;
• To support audio only and audio+video recording;
• Http support is not complete, no BUFFERING status update, not asynchronous message for data insufficient/available;
• No RTSP support yet.

Anyway, it's still under active development and it's from Google, so it wouldn't be surprising that it evolves quickly as a replacement of OpenCORE in the near future;)

An overview of Stagefright player

There is a new playback engine implemented by Google comes with Android 2.0 (i.e, Stagefright), which seems to be quite simple and straightforward compared with the OpenCORE solution.

• MediaExtractor is responsible for retrieving track data and the corresponding meta data from the underlying file system or http stream;
• Leveraging OMX for decoding: there are two OMX plugins currently, adapting to PV's software codec and vendor's hardware implementation respectively. And there is a local implementation of software codecs which encapsulates PV's decoder APIs directly;
• AudioPlayer is responsible for rendering audio, it also provides the timebase for timing and A/V synchronization whenever audio track is present;
• Depending on which codec is picked, a local or remote render will be created for video rendering; and system clock is used as the timebase for video only playback;
• AwesomePlayer works as the engine to coordinate the above modules, and is finally connected into android media framework through the adapter of StagefrightPlayer.