This is a simple example to debug binary mex files compiled by Matlab.
I am going to use Gdb, and Matlab without UI.
You can apply this trick with LLDB on MacOS X, and also with GDB on any Linux platform.
The only thing you need to do before running a debugger is to compile the target binary with -g option.
As most developers knew, '-g' option means to add symbols' information and their location in the source code into the output binary in compile time.
Start Matlab with GDB enabled.
With -Dgdb flag, matlab will be run only after you enter 'run' command in GDB prompt. '-nojvm' option would let you debug your source faster.
Run the target command and find out what was going wrong
In general, setting a breakpoint at the symbol, 'mexFunction', implies that debugger is going to stop at the very beginning of the routine.
Then enjoy debugging!
[ref1] Instruction page on Mathwork homepage.