Wednesday, February 3, 2010

Analyze memory leak of Android native process

Android libc_debug.so has a built-in function to dump all heap allocations with its backtrace, which is very useful to debug memory leaks of native processes. Below are the steps summarized during my investigation of mediaserver process:
  1. apply the patch in ./frameworks/base, which registers a memory dumper service in mediaserver process, then rebuild
  2. (*)flash new system.img, replace libc.so with libc_debug.so, then reboot
    • $ adb remount
    • $ adb shell mv /system/lib/libc_debug.so /system/lib/libc.so
    • $ adb reboot
  3. run memorydumper to get the initial heap allocations of mediaserver process
    • $ adb shell /system/bin/memorydumper
  4. play several files, save the process maps during playback, then get the memory dump again
    • $ adb pull /proc/<mediaserver_pid>/maps .
    • $ adb shell /system/bin/memorydumper
  5. get the diff file of memory allocations
    • $ adb pull /data/memstatus_1136.0 .
    • $ adb pull /data/memstatus_1136.1 .
    • $ diff memstatus_1136.0 memstatus_1136.1 >diff_0_1
  6. run the script to resolve symbols from the backtrace addresses in the diff file
    • $ ./addr2func.py --root-dir=../ --maps-file=./maps diff_0_1
[Update: 07/05/2010]
In Froyo release, there is no need to replace libc.so with libc_debug.so. Try below steps instead of original step 2:
  • adb shell setprop libc.debug.malloc 1
  • adb shell ps mediaserver
  • adb shell kill <mediaserver_pid>
And I also updated the script to skip libc_debug.so while parsing symbols.

[Update: 06/22/2013]
Actually Android has a built-in service which could dump mediaserver's memory directly, so you can replace memorydumper related steps with below command. Sorry that I wasn't aware of this tool when I wrote this article :)
 #dumpsys "media.player" -m

24 comments:

Anonymous said...

Thank you for the information. You've done a lot work here. Can you please state the Android version along with your findings?

Li Chen said...

The patch was based on Eclair, and also applicable on the master branch.

Anonymous said...

Hello,
I tried your steps to analyse memory leak. But I have a few doubts.
I was able to get initial heap allocations of mediaserver process. Then i tried playing a file and again tried to run memorydumper as you have mentioned in step 4. But I couldn't get any dump since the execution of memorydump didnt stop ( even after i closed the application). Could you please tell if there is anything else i need to do?
My other doubt is in the python script, is the mapfile you are talking of is the one which will be in /proc/pid/maps or the memstatus_pid.maps in /data folder?. Also could you please upload a sample output file? I am trying to check memory leaks in gstreamer ported on android.
Thanks in advance.

Li Chen said...

Hi, the memory dump takes some time to finish (usually in seconds). It dosn't help with closing the application as it's trying to dump the native mediaserver process' memory:)
If it lasts too long, then you might need to turn on the logs in frameworks/base/memorydumper/MemoryDumper.cpp to see what's going on there...

And yes, the map file is /proc/pid/maps, which I forgot to replace the symbols of "<" and ">" in the original post, and I just corrected it:)

Anonymous said...

Hello,
Thanks a lot for replying. The execution of memorydumper was taking a very long time because my count value was very large ( was around 9000). Now after getting the dumps, and after running the python script I am not able to figure out how to proceed in finding out the memory leaks. It would be of of great help if you could upload a sample output file and tell how to proceed in finding out where and how much memory is leaking. Thanks a lot again.

Li Chen said...

Each line of the dump output consists of 3 parts:
1) size of the allocated memory chunk
2) dup: how many times the malloc has been executed with the same call stack.
3) the backtrace when malloc is invoked.

e.g. below line indicates that it has allocated 4 memory chunks of 1104 bytes through the same call stack:
size 1104, dup 4, 0x8000b094, 0x8000b374, 0x8000aa60, 0xa7bb2890, 0xa7663f84, 0xa76643a4, 0xa7ba7698, 0xa7ba957c, 0xa7baa6f4, 0xa7bae2a8, 0xa602b0be, 0xa602b328, 0xa602b6e0, 0xa7bad000, 0xa7bad1d0, 0x80011c44, 0x80011710

The script just translates the hex addresses of the call stacks into file&function names.

So if you got a diff file where the dup number keeps increasing for some stack traces after testing, then it's very likely the allocated memory hasn't been freed appropriately, which needs to be investigated case by case.

Unknown said...
This comment has been removed by the author.
Unknown said...

A typo needs correction here.

$ adb shell pull /proc//maps .

should be

$ adb pull /proc//maps .

Unknown said...

Getting a crash when I run memorydumper.

To reproduce, open and close the camcorder app. Then run the memorydump.

This is on froyo build

Li Chen said...

Thanks for pointing out the typo:)

I haven't experienced the crash with memorydumper on FroYo. Perhaps you can use the built-in dumpsys as an alternative:

dumpsys media.player -m

Anonymous said...

thanks for nice information. please clarify my doubts.

after following your steps upto 4, step 5 is executed fine. but memstatus_.0 and memstatus_.1 files size are zero. but when i do adb pull /proc/pid/maps i can some size will printed. is that OK to have zero sized files (memstatus_.0 & .1)
for step 6 i am not able to run python file. please provide more info. as i am testing on device, to run python file should i have to install python and then run. or where should i place that pyton file. is it on /system/bin/ or /data.

thanx in advance.
Raghu

Li Chen said...

Size is 0 indicates no memory info dumped:)
The original steps were for Eclair. On Froyo, you'll need to follow the updated steps to set the property of libc.debug.malloc to 1, then restart mediaserver process before testing.

And run the python scripted on host machine instead of the device to resolve symbols:)

raghu said...

thx for your help.
--> on froyo device. i am running memorydumper service attched my own native process.

able to get initial files.
memstatus_pid.0 and memstatus_pid.0.maps, smaps, statm files with valid content.

while getting memstatus_pid.1 file i am getting crash. i could able to see log as "enter dumpHeap".

any reasons ???

thanks,
Raghu

Santhosh said...

Hello :)

I am not able to find memorydumper tool in /system/bin/ folder. I could not find the frameworks/base/memorydumper/MemoryDumper.cpp
Can you please help me get this?

Santhosh said...

It is part of the patch :)

Anonymous said...

Hi freepine,

I tried the steps mentioned above to setup memorydump and also got the memory dump files.

In order to check how it works, I deliberately added a memory leak of 200 bytes in Mpeg4Extractor for everytime a read is called. I expected the memory allocation of 200 bytes to be shown in the memory dump output file when I run the command addr2func.py. But I dont find the allocation at all. Any pointers on what can be the possible cause for this ?

Thanks in advance.

Anonymous said...

Hi,
Will this patch work for Gingerbread.

Anonymous said...

Hi,

Thanks for very useful info!!

I have tried with the steps you have provided & able to get memory dump, which has size, dup & call backtrace info.

However I want to understand more to find out memory leak.
Does it mean all the chucks of data which are tracked are having mem leak?
For my usecase it has been generated huge file with lot many
size, dup & back trace info. How to find where exactly is the possibility of mem leak.

Please reply
Thanks..

Sac Bee said...

Can this run on Jellybean? I tried getting this built, but I see fopen errors :(

I changed from /data/ to /sdcard/ but didn't help :(

Li Chen said...

Sachin, yes, the patch is a bit out-dated. And you can skip the memorydumper patch related steps here and use the built-in dumpsys tool directly, then resolve the symbols with the python script.

WonHee said...

Hi,
First, Thank you for the information.
And I have a question

I do below process
adb shell setprop libc.debug.malloc 1
adb shell ps mediaserver
adb shell kill
#dumpsys "media.player" -m

And I get below information.

>> Allocation count 563
Total memory 2310277
ize 337272, dup 1, 0xb59f90e2, 0xb6f24d5e, 0xb5650f14, 0xb564fdea, 0xb564a7ac, 0xb5649f30, 0xb565c6
8, 0xb56bd450, 0xb56b4f08, 0xb5a6b390, 0xb5a6b41a, 0xb5a6bc18, 0xb6fe4bde, 0xb6f25552
.......

How analyze this information ?
If you know, Please inform me.
Thank you.

Unknown said...

Hi Freepine,

It was very helpful for investigating the mediaserver memory leak issue that I had. Thank you very much.

But the script was too old for latest version of Android and when it cannot find the symbolic so, it just stops the script.

I just modified your script very lightly to use it for my case. I uploaded modified script into my github. I mentioned the origin in Readme file but If you don't like me to upload it there, Please let me know.

https://github.com/leehack/addr2func

Regards,
Jhin

Li long said...

Hi Freepine

Thanks for your share, it's very helpful for us. It works fine on Android L, but something wrong on Android M. So do you have tried this method on Android M ?

Thanks!

Anonymous said...

Hi Freepine,

We are getting some memory leaks in mediaserver in Android5.1.1 version.To debug i, we followed your steps as you mentioned.We got the output file mtrace.But we didn't understad the mtrace file.
Could you give suggestions to understand mtrace file ,so that it is healpful to usto debug.