System analysis

strace

Available on all GNU/Linux systems, Can be built by your cross-compiling toolchain generator.
Even easier: drop a ready-made static binary for your architecture, just when you need it. See http://git.free-electrons.com/users/michael-opdenacker/static-binaries/tree/strace
Allows to see what any of your processes is doing: accessing files, allocating memory... Often sufficient to find simple bugs.
Usage: strace (starting a new process) strace -p (tracing an existing process) See man strace for details.

A tool to trace library calls used by a program and all the signals it receives

Very useful complement to strace, which shows only system calls.
Of course, works even if you don't have the sources
Allows to filter library calls with regular expressions, or just by a list of function names.
Manual page: http://linux.die.net/man/1/ltrace

A system-wide profiling tool
Can collect statistics like the top users of the CPU.
Works without having the sources.
Requires a kernel patch to access all features, but is already available in a standard kernel.
Requires more investigation to see how it works.
Ubuntu/Debian packages: oprofile, oprofile-gui