Wednesday, February 16, 2022

Google Translate

 I observed an interesting behavior while playing with Google Translate today:


Not sure if it's a result of some incorrectly tagged samples or it's the underlying neural translation model has accumulated a small sense of humor:) If somehow we can shape the personality of a large neural network with the training dataset, can we say that the trained model is sort of conscious, maybe to the slightest extent?

Wednesday, December 30, 2015

ExoPlayer Architecture

ExoPlayer is an open source media player from Google. It provides an example implementation for DASH and Smooth Streaming playback with Common Encryption, so that 3rd-party applications can extend it to build rich media experience which isn't directly available from the built-in MediaPlayer.

[1] https://google.github.io/ExoPlayer/guide.html
[2] https://github.com/google/ExoPlayer
[3] http://www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=65274
[4] https://www.iso.org/obp/ui/#iso:std:iso-iec:23001:-7:ed-1:v1:en

Tuesday, June 3, 2014

Introduction of audio pipeline in ChromiumOS

1. System architecture

Chromium OS is an open source project, based on Linux kernel with Chrome browser as the main application interface.
The communication mechanism between Chromium OS and its browser currently is via D-bus.

2. Chrome OS audio server (CRAS)

2.1 Introduction

  • CRAS sits between Chrome and ALSA 
    • Output: Chrome connects to this audio server, sends audio data to it, then audio server renders this to the selected ALSA device through alsa-lib. 
    • Input: the server will get audio samples from alsa and forward them to Chrome. 
  • Three main requirements: 
    • Low Latency: 20ms latency and waking up every 10ms 
    • Low CPU usage: Less than two percent of the CPU of a CR48 is used while playing back audio 
    • Dynamic audio routing

2.2 Code structure

  • server - the source for the sound server 
  • libcras - client library for interacting with cras 
  • common - files common to both the server and library 
  • tests - tests for cras and libcras
  • source code location: chromiumos/src/third_party/adhd/cras

2.3 Block diagram

2.4 Run-time behavior

2.4.1 Initialization

2.4.2 Audio IO thread

Audio IO thread is responsible for reading data from each stream, mixing and sending to output device. Note: Digital signal processing(DSP) is applied post-mix, i.e. global effects.

2.4.3 DSP thread

There is a dedicated thread to handle DSP related requests.

2.4.4 server main thread

2.4.5 Audio routing

2.4.5.1 Jack plugged in

2.4.5.2 Switch to Bluetooth

3. Chromium Media Pipeline

3.1 Introduction

Pull-based pipeline, consists of data source, demuxers, decoders and renders.

3.2 Pipeline integration into Webkit

3.3 Pipeline state machine

3.4 Pipeline decomposition

3.4.1 class diagram

3.4.2 Audio Sink to CRAS

3.4.3 Example of initiating a new playback stream

3.5 Multi-channel processing

Currently IO devices enumerated with CRAS server only support up to stereo output. If a client requests to add a stream which isn’t compatible with iodev’s audio format, then CRAS client will perform format conversion automatically.

4. Interactions between UI and CRAS

Because all applications run in browser tabs, Chromium has defined an extension API currently for audio to control CRAS parameters, such as setting volume, etc.

References

Thursday, October 24, 2013

DASH tools from GPAC


1. Content creation
  • DashCast
    $./DashCast -conf dash.conf -av Fantastic_Four_Trailer.mp4 -seg-dur 2000 -frag-dur 200
  • MP4Box
    $./MP4Box -dash 2000 -frag 200 -rap -frag-rap Fantastic_Four_Trailer.mp4

2. Playback

Sunday, September 15, 2013

Open source media frameworks


  • VLC LGPL, C
    git clone git://git.videolan.org/vlc.git
  • MPlayer GPLv2, C
    svn checkout svn://svn.mplayerhq.hu/mplayer/trunk mplayer
  • GStreamer LGPL, C
     git clone git://anongit.freedesktop.org/gstreamer/
  • GPAC LGPL, C
    svn co svn://svn.code.sf.net/p/gpac/code/trunk/gpac gpac
  • XBMC GPL, C++
    git clone git://github.com/xbmc/xbmc.git
  • Xine GPLv2, C
    http://sourceforge.net/projects/xine/
  • FFMpeg GPL/LGPL, C
    git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

Tuesday, May 21, 2013

Next phase of my life

My daughter was born a week ago. If you know me or my wife, please share our happiness! :-D

Saturday, May 11, 2013

Casual thoughts on Stagefright

For the past 4+ years, I've mainly focused on Android multimedia area in my daily work. While the maintenance of Stagefright is much easier compared with OpenCORE,, there are still places which I think could be improved in future. Although I will not have the privilege of closely working on it any more after moving to my next occupation :)

Main issue
One of the biggest pains while working with Stagefright is various ANRs and tombstones related to mediaserver during stability test or user trials.
On one hand, Stagefright isn't mature and robust enough, there are some inherent issues limited by its structure, which makes it very difficult to fix all of them thoroughly. For example, you can easily reproduce an ANR while seeking and play/pause quickly during HTTP streaming due to the blocking read() call to network data and no way to cancel it gracefully, and the existing buffering mechanism doesn't work well because it can't predict which position to be read after seeking; and the potential deadlock between mediaserver and mediaplayer due to the subtle difference between calling mediaplayer from another application and from mediaserver itself (e.g. CameraService) as the binder transaction is skipped from intra-process call, etc.
On the other hand, even if you find the root cause and a potential solution, it's difficult to validate the fix because of the problem is highly impacted by network condition, race condition among threads, etc. Although we can simulate network traffic conditions with bandwidth control tools like netem, it still requires fine tuning and many tries if we don't hack into the code to force it to enter the situation. For lock related issues, I used to write simple CppUTest cases in many loops with different sleep time between function calls and then wish myself luck.

As an advocate of TDD and test automation, I think it would be ideal to have native media engine code written with testability in mind, such as incorporating gtest or other C++ test frameworks, so we can easily add unit test cases for newly reported issues. It would also help to facilitate the up-merge process of each release. Based on my experience, CTS and mediaframeworktest is far from sufficient to catch most of the issues we encountered before shipping the device.

In addition, with more and more features added into AOSP, it would help to unify different pieces of multimedia module into a more flexible architecture, to provide a generic pipeline for playback, recording, and transcoding etc, as well as an elegant way to support customized audio features, like LPA, tunnel mode, 5.1 channels, etc.

I guess it sounds greedy given that most of its code was written by only a couple of Google engineers in a short time with more and more functionality been added to each release. Keep moving, Stagefright!

Tuesday, April 23, 2013

The Seven Year Itch

It's the seventh year since I married my wife and it's also the seventh year of my working at Motorola Mobility:) While my wife is pregnant and we are looking forward to our first baby in the upcoming May, my journey at Motorola is going to an end at the same time. Although the company was kind enough to offer me a relocation option along with the pink slip, I am not able to accept it at this moment for the obvious reason.

Hello Moto, Goodbye Moto~

Friday, November 16, 2012

Multimedia on JellyBean 4.2

Besides wireless display and some new camera hardware interface, it seems there isn't much change on Multimedia framework from the new released JellyBean 4.2 source code. Most of the patches are for bug-fixes and small code refactoring.
And same as 4.1, we can enable NuPlayer as the default media engine for local & http playback with a system property "media.stagefright.use-nuplayer".
However, compared to Awesomeplayer, there are still some functional gaps to fully switch to NuPlayer:

  • Timed text support;
  • BUFFERING related status update in GenericSource;
  • Secure input buffer support for content protection, e.g. Widevine;
  • Other minor improvements, such as preparing decode pipeline before PREPARED event, setLooping, changing surface texture while playing,etc.
Google has also added some code for fragmented MP4 parser, perhaps for MPEG-DASH but it's said to be experimental.

Friday, May 20, 2011

NuPlayer for HTTP live streaming

HTTP Live Streaming is separated from Stagefright on the recent release, which is basically another light-weighted playback engine, except it only supports the fixed container and codecs format currently.
It seems that the author really prefers rewriting than refactoring:)

Unlike Awesomeplayer, NuPlayer is built upon Stagefright's foundation classes, and leverages the Looper/Handlers mechanism to handle requests asynchronously by queuing them in a message loop, so there are less mutex/lock in place.


  • NuPlayer::Source is the parser module. Actually its interface looks like a combination of MediaExtractor and MediaSource, and it also makes seekTo as an explicit API now.
  • NuPlayer::Decoder connects to ACodec for AVC decoding, and to DecoderWrapper for AAC decoding, which in turn wrapps AAC software decoder in the OpenMAX style. ACodec is functionally similar as OMXCodec in Stagefright, besides the application of State pattern and passing MediaBuffers around with messages.
  • NuPlayer::Render is responsible for rendering audio and also controls when to post video buffers back to NativeWindow for A/V sync.

Wednesday, May 18, 2011

Travel

I've been on travel to Libertyville for nearly two weeks. While it's a bit boring to live at a hotel, the good thing is I have a lot of spare time now, and I can freely surf the internet, so it's probably a good time for blogging:)

Sunday, November 14, 2010

DRM integration in Stagefright

Google has pushed a change into AOSP which integrates DRM support into Stagefright recently:
http://android.git.kernel.org/?p=platform/frameworks/base.git;a=commitdiff;h=d5770917a50a828cb4337c2a392b3e4a375624b9#patch12

Before we delve into the change, there are basically two types of DRM schemes:
1. All of the data was stored under a uniform encryption layer (which is defined as DecryptApiType::CONTAINER_BASED in drm framework currently);
2. Encrypted data is embedded within a plain-text container format, so it can be decrypted packet by packet, thus also applicable for progressive download and streaming.

The entire commit is mainly composed of the following parts:
1)Extension in DataSource interface:
+    // for DRM
+    virtual DecryptHandle* DrmInitialization(DrmManagerClient *client) {
+        return NULL;
+    }
+    virtual void getDrmInfo(DecryptHandle **handle, DrmManagerClient **client) {};

FileSource implements those APIs, where it communicates with DRM service to open a decryption session. For CONTAINER based protection (e.g. OMA DRM v1), FileSource intercepts readAt() function and return the decrypted data transparently to its client -- file parser, which is thereby ignorant of the underlying protection mechanism.

2)Extension in MediaExtractor interface:
+    // for DRM
+    virtual void setDrmFlag(bool flag) {};
+    virtual char* getDrmTrackInfo(size_t trackID, int *len) {
+        return NULL;
+    }
The above APIs are used to retrieve context data, such as Protection Scheme Information Box in mp4/3gp files, to properly initialize the corresponding DRM plugin.

3)DRMExtractor and the DRM format sniffer:
SniffDRM is registered to identify DRM-protected files and its original container format. As mentioned, for CONTAINER_BASED encryption, FileSource handles data decryption transparently for parsers. For the other scheme, which are encrypted with sample/NAL as the basic unit, DRMExtractor is created to wrap the original MediaExtractor, to decrypt data after reading each sample/NAL from the original extractor. In this way, DRM related stuff is separated from actual file parsers.
However, as DRM is usually an extension based on the underlying container format, so it may not be as easily decoupled from file parser when it comes to other protection schemes. For example, Microsoft's PIFF extension to ISO base media format requires IV for each sample, and details of sub-sample encryption info if applicable, etc. Besides, it also imposes duplicate logic in DRM service to recognize the original container format for non-CONTAINER_BASED encryption.

4)Misc:
-changes in AwesomePlayer for rights management;
-changes in MPEG4Extractor to retrieve "sinf";
-etc.

Thursday, September 30, 2010

RTSP in Stagefright (2)

A basic sequence diagram of setting up RTSP connection in Stagefright:

Tuesday, September 28, 2010

RTSP in Stagefright (1)

RTSP support was added into Stagefright on GingerBread release:
  • A preliminary foundation module, provides a generic way to asynchronously handle commands and events/messages sequentially in the schedule thread. Each message has a target id, indicating its corresponding handler, which is then registered in a looper. A looper spawns a thread, and schedules registered handlers to process any messages posted to it.
  • ARTSPController acts as MediaExtractor for RTSP playback, which then delegates RTSP related stuff to ARTSPConnection, and payload parsing related to ARTPConnection/ARTPSource. ARTPSource leverages ARTPAssembler to re-assemble RTP packets to frames. AAMRAssembler, AH263Assembler, etc. all inherit from ARTPAssember according to the corresponding protocols. Buffers parsed are sent back via AMessage and queued in APacketSource, which acts as the MediaSource for downstream components.
  • There is also ARTPSession and ARTPWriter which re-uses the existing RTP protocol for VOIP(Gtalk?) solution.




Sunday, July 4, 2010

Native mediatest updated for FroYo

I had almost had to update the small program for every major release, so I've uploaded it to github for better version control:
git clone http://github.com/freepine/MediaTest.git

No wonder Android engineers keep warning us from accessing unpublished code:)

Wednesday, June 2, 2010

Media framework change in Froyo

Here is a Google I/O session by Dave Sparks, who talked about new APIs of media framework introduced in Froyo, and also some light on the future of Stagefright, OpenCORE, etc in the Q&A section.

Wednesday, February 3, 2010

Analyze memory leak of Android native process

Android libc_debug.so has a built-in function to dump all heap allocations with its backtrace, which is very useful to debug memory leaks of native processes. Below are the steps summarized during my investigation of mediaserver process:
  1. apply the patch in ./frameworks/base, which registers a memory dumper service in mediaserver process, then rebuild
  2. (*)flash new system.img, replace libc.so with libc_debug.so, then reboot
    • $ adb remount
    • $ adb shell mv /system/lib/libc_debug.so /system/lib/libc.so
    • $ adb reboot
  3. run memorydumper to get the initial heap allocations of mediaserver process
    • $ adb shell /system/bin/memorydumper
  4. play several files, save the process maps during playback, then get the memory dump again
    • $ adb pull /proc/<mediaserver_pid>/maps .
    • $ adb shell /system/bin/memorydumper
  5. get the diff file of memory allocations
    • $ adb pull /data/memstatus_1136.0 .
    • $ adb pull /data/memstatus_1136.1 .
    • $ diff memstatus_1136.0 memstatus_1136.1 >diff_0_1
  6. run the script to resolve symbols from the backtrace addresses in the diff file
    • $ ./addr2func.py --root-dir=../ --maps-file=./maps diff_0_1
[Update: 07/05/2010]
In Froyo release, there is no need to replace libc.so with libc_debug.so. Try below steps instead of original step 2:
  • adb shell setprop libc.debug.malloc 1
  • adb shell ps mediaserver
  • adb shell kill <mediaserver_pid>
And I also updated the script to skip libc_debug.so while parsing symbols.

[Update: 06/22/2013]
Actually Android has a built-in service which could dump mediaserver's memory directly, so you can replace memorydumper related steps with below command. Sorry that I wasn't aware of this tool when I wrote this article :)
 #dumpsys "media.player" -m

Tuesday, January 5, 2010

Current status of Stagefright

  • Local file playback (AMR/MP3/MP4/...) and basic http playback support;
  • Video only recording;
  • A corresponding metadata retriever, not fully implemented yet.
Some thoughts on the enhancement:
  • No abstraction layer for media sink, and lack of a pluggable mechanism to pick decoder dynamically based on the source's format and sink's capability. The overall architecture doesn't seem to be flexible enough to date;
  • It's good to re-use the same MediaExtractor from player engine for metadata retrieving to avoid duplication. However, MediaExtractor itself might not get all required keys reliably, e.g. it might be more accurate to get some keys (e.g, video height/width) from decoder's output;
  • To support audio only and audio+video recording;
  • Http support is not complete, no BUFFERING status update, not asynchronous message for data insufficient/available;
  • No RTSP support yet.
Anyway, it's still under active development and it's from Google, so it wouldn't be surprising that it evolves quickly as a replacement of OpenCORE in the near future;)

Monday, January 4, 2010

An overview of Stagefright player

There is a new playback engine implemented by Google comes with Android 2.0 (i.e, Stagefright), which seems to be quite simple and straightforward compared with the OpenCORE solution.
  • MediaExtractor is responsible for retrieving track data and the corresponding meta data from the underlying file system or http stream;
  • Leveraging OMX for decoding: there are two OMX plugins currently, adapting to PV's software codec and vendor's hardware implementation respectively. And there is a local implementation of software codecs which encapsulates PV's decoder APIs directly;
  • AudioPlayer is responsible for rendering audio, it also provides the timebase for timing and A/V synchronization whenever audio track is present;
  • Depending on which codec is picked, a local or remote render will be created for video rendering; and system clock is used as the timebase for video only playback;
  • AwesomePlayer works as the engine to coordinate the above modules, and is finally connected into android media framework through the adapter of StagefrightPlayer.