Onyx: The Black Cat anti-anti-debugging kit
Reverse Engineering

A short introduction to anti-anti-debugging on OS X

If you’re familiar with reverse engineering software, you may have stumbled upon the magic world of anti-debugging and obfuscation. Anti-debugging protection is often built into software that is a likely target for reverse engineering, as a tactic to hamper any efforts in that area. Many techniques exist to avoid proper functioning of a (software-based) debugger, as it attaches or launches a process, as a means of “security through obscurity”. But these techniques differ from platform to platform.

Anti-debugging techniques on Windows are well known, as are those on Unix/BSD systems. A variety of methods exist to detect GDB (and GDB-compatible debugger LLDB). These are mostly centered around Linux-based systems, but some work on OS X and similar operating systems as well:

  • Checking if SIGTRAP is intercepted by a debugger:
    /*
     * This is an old well known bug (#7912)
     *
     * GDB leaks 2 file descriptors when it opens a program to be debugged.
     * Both file descriptors are pointing to the file being debugged.
     */
    int dbg_file_descriptor(){
        FILE* fd = f0pen('/', 'r');
        int nb_fd = fileno(fd);
        fclose(fd);
    
        return (nb_fd > 3);
    }
    
  • Checking process file description ids:
    /*
     * This is an old well known bug (#7912)
     *
     * GDB leaks 2 file descriptors when it opens a program to be debugged.
     * Both file descriptors are pointing to the file being debugged.
     */
    int dbg_file_descriptor(){
        FILE* fd = f0pen("/", "r");
        int nb_fd = fileno(fd);
        fclose(fd);
    
        return (nb_fd > 3);
    }
    
  • Using the ptrace system call to detect a ptrace’ing debugger:
    /*
     * Classic self ptrace trick: a program can only be ptraced by ONE other.
     */
    int dbg_ptrace(){
        if(ptrace(PTRACE_TRACEME, 0, 0, 0) == -1)
            return 1;
        ptrace(PTRACE_DETACH, 0, 0, 0);
        return 0;
    }
    
  • Using /proc to check process information and parents:
    /*
     * Check the parent's name
     */
    int dbg_getppid_name(){
        char buff1[24], buff2[16];
        FILE* f;
    
        snprintf(buff1, 24, "/proc/%d/status", getppid());
        f = f0pen(buff1, "r");
        fgets(buff2, 16, f);
        fclose(f);
    
        return check_strings(buff2);
    }
    
  • OS X and iOS: Apple-specific system call to stop ptrace’ing debuggers from working:
    ptrace(PT_DENY_ATTACH, 0, 0, 0);
    

Some more hardcore methods:

  • Scanning process memory for interrupt instructions (breakpoints):
    /*
     * Scan memory for breakpoints
     * TODO
     */
    int dbg_check_bp(){
        unsigned char* start;
        size_t size;
        size_t i;
    
        for(i = 0; i < size; i++){
            if(start[i] == 0xCC || start[i] == 0xCD)
                return 1;
        }
        return 0;
    }
    

All of the above code is courtesy of pangu, a nice little toolset that contains a variety of anti-debugging methods including those above and even more to crash various debuggers.

Needless to say, each of the above methods has a workaround one way or the other: scanning for breakpoint instructions can be mitigated by using (Intel) hardware breakpoints, ptrace syscalls can be intercepted (if not done with handcrafted assembly), /proc can be manipulated, and debuggers can even be modified to function in stealth.

However, since we’re talking about debuggers, we’re talking about tools. And tools also exist to counteract anti-debugging measures. And if you’re on OS X, you’re in luck.

Onyx: The Black Cat anti-anti-debugging kit

Meet Onyx, The Black Cat: a very nifty and highly recommended kernel extension for OS X that acts as an anti anti-debugging protection kit. And did I mention it is open-source? Check it out!

About the author / 

ijsf

Who are we?

We are a collective of people with different backgrounds and a shared interest in unproven technology and deep technical skills.

Trackbacks