linux - Upstart init is leaking memory, how do you debug it? -

i've got memory leak in upstart init process (pid 1), options have on debugging it?

edit: suggest me real tools this, manually putting printfs or calculating memory allocations hand isn't gonna cut it. dumping init core , poking around not option.

upd1: valgrind doesn't work. replacing /sbin/init on kernel command line proper valgrind + init magic doesn't seem option tries access /proc self smaps, isn't available before init ran.

upd2: dmalloc doesn't work either (doesn't compile on arm).

a poor man's solution log every call malloc , free, comb through logs , pattern.

ld provides amazing feature here.

--wrap=symbol

use wrapper function symbol. undefined reference symbol resolved "__wrap_symbol". undefined reference "__real_symbol" resolved symbol.

this can used provide wrapper system function. wrapper function should called "__wrap_symbol". if wishes call system function, should call "__real_symbol".

here trivial example:

void * __wrap_malloc (size_t c) {    printf ("malloc called %zu\n", c);    return __real_malloc (c); } 

if link other code file using --wrap malloc, calls "malloc" call function "__wrap_malloc" instead. call "__real_malloc" in "__wrap_malloc" call real "malloc" function.

you may wish provide "__real_malloc" function well, links without --wrap option succeed. if this, should not put definition of "__real_malloc" in same file "__wrap_malloc"; if do, assembler may resolve call before linker has chance wrap "malloc".

---

update

just clear on how useful.

• add custom file upstart's build.

like this:

void*__wrap_malloc( size_t c ) {    void *malloced = __real_malloc(c);    /* log malloced associated backtrace*/    /* like: <malloced>: <bt-symbol-1>, <bt-symbol-2>, .. */    return malloced }  void __wrap_free( void* addr ) {    /* log addr associated backtrace*/    /* like: <addr>: <bt-symbol-1>, <bt-symbol-2>, .. */    __real_free(addr); } 

• recompile upstart debug symbols (-g) can nice backtraces. can still optimize (-o2/-o3) code if wish.

• link upstart ld_flags --wrap=malloc, --wrap=free.
  anywhere upstart calls malloc symbol magically resolved new symbol __wrap_malloc. beautifully transparent compiled code happens @ link time.
  it's shimming or instrumenting out of mess.

• run recompiled upstart usual until you're sure leak has occured.

• look through logs mismatch malloceds , addrs.

a couple of notes:

• the --wrap=symbol feature not work function names macros. watch out #define malloc nih_malloc. libnih you'd need use --wrap=nih_malloc , __wrap_nih_malloc instead.
• use gcc's builtin backtracing features.
• all of these changes affect recompiled upstart executable.
• you dump logs sqlite db instead may make easier find mismatch mallocs , frees.
• you can make log format sql insert statement insert them database post-mortem further analysis.