Debugging

Overview

iSH includes several tools and facilities for debugging both the emulator itself and the code running inside it. This guide covers debugger scripts, crash handlers, state inspection, and step-debugging tools.

Debugger Scripts

iSH provides helper scripts for GDB and LLDB that enhance the debugging experience.

GDB Script (ish-gdb.gdb)

Location: ish-gdb.gdb (symlinked to build directory)

handle SIGUSR1 noprint pass
handle SIGTTIN noprint pass
handle SIGPIPE noprint pass
set print thread-events off

define hook-run
    python
import subprocess
if subprocess.call('ninja') != 0:
    raise gdb.CommandError('compilation failed')
    end
end

define hook-stop
    python
try:
    symtab = gdb.selected_frame().find_sal().symtab
except:
    pass
else:
    if symtab is not None and symtab.filename.endswith('.S'):
        gdb.execute('set disassemble-next-line on')
    else:
        gdb.execute('set disassemble-next-line auto')
    end
end

Features:

Signal Handling: Ignores common signals (SIGUSR1, SIGTTIN, SIGPIPE) that iSH uses internally
Auto-rebuild: Automatically runs ninja before each run
Smart Disassembly: Enables disassembly view when stepping through assembly files (.S)

Usage:

cd build
gdb ./ish
source ish-gdb.gdb
run -f alpine /bin/sh

LLDB Script (ish-lldb.lldb)

Location: ish-lldb.lldb

process handle -n 0 -p 1 -s 0 SIGUSR1 SIGTTIN SIGPIPE

Features:

Configures LLDB to pass through common signals without stopping

Usage:

cd build
lldb ./ish
command source ish-lldb.lldb
run -f alpine /bin/sh

Crash Handler

iSH includes a built-in crash handler that captures useful debugging information when the emulator crashes.

Implementation

From main.c:

static void crash_handler(int sig) {
    void *array[50];
    int size;
    fprintf(stderr, "\n=== Signal %d caught ===\n", sig);
    size = backtrace(array, 50);
    fprintf(stderr, "Backtrace (%d frames):\n", size);
    backtrace_symbols_fd(array, size, 2);
    if (current) {
        struct cpu_state *cpu = &current->cpu;
#ifdef ISH_GUEST_64BIT
        fprintf(stderr, "x86_64 CPU state at crash:\n");
        fprintf(stderr, "  RIP=0x%llx RSP=0x%llx RBP=0x%llx\n",
                (unsigned long long)cpu->rip, (unsigned long long)cpu->rsp,
                (unsigned long long)cpu->rbp);
        fprintf(stderr, "  RAX=0x%llx RBX=0x%llx RCX=0x%llx RDX=0x%llx\n",
                (unsigned long long)cpu->rax, (unsigned long long)cpu->rbx,
                (unsigned long long)cpu->rcx, (unsigned long long)cpu->rdx);
        fprintf(stderr, "  RSI=0x%llx RDI=0x%llx\n",
                (unsigned long long)cpu->rsi, (unsigned long long)cpu->rdi);
#else
        fprintf(stderr, "x86 CPU state at crash:\n");
        fprintf(stderr, "  EIP=0x%x ESP=0x%x EBP=0x%x\n",
                cpu->eip, cpu->esp, cpu->ebp);
        fprintf(stderr, "  EAX=0x%x EBX=0x%x ECX=0x%x EDX=0x%x\n",
                cpu->eax, cpu->ebx, cpu->ecx, cpu->edx);
        fprintf(stderr, "  ESI=0x%x EDI=0x%x\n",
                cpu->esi, cpu->edi);
#endif
    }
    fprintf(stderr, "======================\n");
    _exit(128 + sig);
}

int main(int argc, char *const argv[]) {
    signal(SIGSEGV, crash_handler);
    signal(SIGBUS, crash_handler);
    signal(SIGABRT, crash_handler);
    signal(SIGILL, crash_handler);
    signal(SIGFPE, crash_handler);
    // ...
}

Example Crash Output

=== Signal 11 caught ===
Backtrace (15 frames):
./ish[0x4023a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x3c0f0)[0x7f8b3c0f0]
./ish[0x401234]
./ish[0x405678]
x86 CPU state at crash:
  EIP=0x8048a34 ESP=0xbffff7a0 EBP=0xbffff7b8
  EAX=0x00000000 EBX=0x08048000 ECX=0x00000001 EDX=0x00000000
  ESI=0xbffff800 EDI=0x00000000
======================

Information Provided:

Signal Number: Type of crash (11 = SIGSEGV, 6 = SIGABRT, etc.)
Native Backtrace: Stack trace of the emulator process
Guest CPU State: x86 register state at the time of crash
- Instruction pointer (EIP/RIP)
- Stack pointer (ESP/RSP)
- General-purpose registers

CPU State Inspection

In GDB

Access the emulated CPU state:

# Break on a specific instruction address
break cpu_run_to_interrupt
run

# Inspect CPU state
p *cpu
p/x cpu->eip
p/x cpu->regs[0]  # EAX
p/x cpu->eflags

# Pretty-print all registers
define print_cpu
  printf "EIP=0x%08x\n", cpu->eip
  printf "EAX=0x%08x EBX=0x%08x ECX=0x%08x EDX=0x%08x\n", \
    cpu->eax, cpu->ebx, cpu->ecx, cpu->edx
  printf "ESP=0x%08x EBP=0x%08x ESI=0x%08x EDI=0x%08x\n", \
    cpu->esp, cpu->ebp, cpu->esi, cpu->edi
end

In LLDB

# Break and inspect
breakpoint set -n cpu_run_to_interrupt
run

# View CPU state
frame variable cpu
print cpu->eip
print cpu->regs
memory read -s4 -c8 &cpu->regs  # Read all 8 registers

CPU State Structure

From emu/cpu.h:

struct cpu_state {
    struct mmu *mmu;
    long cycle;

    // General registers (32-bit mode)
    union {
        struct {
            _REGX(a);  // EAX/AX/AL/AH
            _REGX(c);  // ECX/CX/CL/CH
            _REGX(d);  // EDX/DX/DL/DH
            _REGX(b);  // EBX/BX/BL/BH
            _REG(sp);  // ESP/SP
            _REG(bp);  // EBP/BP
            _REG(si);  // ESI/SI
            _REG(di);  // EDI/DI
        };
        dword_t regs[8];
    };

    dword_t eip;  // Instruction pointer

    // Flags
    dword_t eflags;
    // ... (flag bits and lazy evaluation fields)

    // FPU state
    union mm_reg mm[8];
    union xmm_reg xmm[8];
    float80 fp[8];
    word_t fsw, fcw;

    // TLS
    word_t gs;
    addr_t tls_ptr;
};

ptraceomatic - Step Debugging Tool

Location: tools/ptraceomatic.c Purpose: Runs a program natively using ptrace and simultaneously in iSH, comparing register state at each instruction.

Requirements

64-bit Linux 4.11 or later
ptrace support

How It Works

From the source:

// Fun little utility that single-steps a program using ptrace and
// simultaneously runs the program in ish, and asserts that everything's
// working the same.

// returns 1 for a signal stop
static inline int step(int pid) {
    trycall(ptrace(PTRACE_SINGLESTEP, pid, NULL, 0), "ptrace step");
    int status;
    trycall(waitpid(pid, &status, 0), "wait step");
    if (WIFSTOPPED(status) && WSTOPSIG(status) != SIGTRAP) {
        int signal = WSTOPSIG(status);
        printk("child received signal %d\n", signal);
        // a signal arrived, we now have to actually deliver it
        trycall(ptrace(PTRACE_SINGLESTEP, pid, NULL, signal), "ptrace step");
        trycall(waitpid(pid, &status, 0), "wait step");
        return 1;
    }
    return 0;
}

Building

cd build
ninja tools/ptraceomatic

Usage

# Run a simple program and compare execution
./tools/ptraceomatic ./ish -f alpine /bin/true

# The tool will:
# 1. Run /bin/true natively with ptrace
# 2. Single-step through each instruction
# 3. Run the same in iSH
# 4. Compare register state after each instruction
# 5. Report any mismatches

Use Cases

Validating emulator accuracy
Finding instruction implementation bugs
Debugging divergent execution paths
Regression testing

Exception Handling (iOS)

For iOS builds, iSH includes exception handling:

void iSHExceptionHandler(NSException *exception);

This handler catches Objective-C exceptions and provides debugging information on iOS where traditional crash handlers behave differently.

Debugging Techniques

Debugging Gadget Execution

Set breakpoints on gadget functions:

# Break when a specific gadget is executed
break gadget_mov
break gadget_add_reg_reg

# Conditional breakpoint
break gadget_interrupt if cpu->trapno == INT_UNDEFINED

Debugging System Calls

Trace system call execution:

# Break on syscall entry
break do_syscall

# View syscall number and arguments
commands
  printf "syscall %d\n", syscall_num
  continue
end

Memory Debugging

Inspect guest memory:

# Read guest memory at address 0x8048000
break cpu_run_to_interrupt
run
set $addr = 0x8048000
set $mmu = cpu->mmu
# Use TLB functions to read guest memory
call mem_ptr(cpu->mmu, $addr, MEM_READ)

Logging + Debugging Combo

Combine logging with breakpoints:

# Build with strace logging
meson configure -Dlog="strace"
ninja

# Run in debugger
gdb ./ish
break do_syscall if syscall_num == 120  # clone
run -f alpine /bin/sh

Debugging JIT/Gadget Generation

Break during code generation:

break gen_step
break gen_end

# Inspect generated gadget chain
commands
  printf "Block: %p, size: %d\n", state->block, state->size
  x/10gx state->block->code
  continue
end

Common Debugging Scenarios

Scenario 1: Undefined Instruction

break gadget_interrupt if trapno == INT_UNDEFINED
commands
  printf "Undefined instruction at EIP=0x%08x\n", cpu->eip
  # Dump instruction bytes
  x/10xb cpu->eip
  # Check what instruction this is
  disassemble cpu->eip,cpu->eip+10
end

Scenario 2: Segfault

break handle_read_miss
break handle_write_miss
commands
  printf "Memory access fault at 0x%08x\n", addr
  backtrace 5
end

Scenario 3: Infinite Loop Detection

# Set a counter and break after N iterations
set $count = 0
break fiber_ret_chain
commands
  set $count = $count + 1
  if $count > 10000
    printf "Possible infinite loop at EIP=0x%08x\n", cpu->eip
    # Stop the program
    signal SIGINT
  end
  continue
end

Debugging Build Issues

Assembly Issues

When gadget assembly fails to compile:

# Get preprocessed assembly
cd build
ninja -v 2>&1 | grep "entry.S"
# Copy the command and add -E to see preprocessed output

# Examine generated offsets
cat cpu-offsets.h

Linking Issues

# Verbose linker output
cd build
ninja -v

# Check symbol visibility
nm libish_emu.a | grep gadget_

Get Started

Architecture

Building

Advanced

Debugging

Overview

Debugger Scripts

GDB Script (ish-gdb.gdb)

LLDB Script (ish-lldb.lldb)

Crash Handler

Implementation

Example Crash Output

CPU State Inspection

In GDB

In LLDB

CPU State Structure

ptraceomatic - Step Debugging Tool

Requirements

How It Works

Building

Usage

Use Cases

Exception Handling (iOS)

Debugging Techniques

Debugging Gadget Execution

Debugging System Calls

Memory Debugging

Logging + Debugging Combo

Debugging JIT/Gadget Generation

Common Debugging Scenarios

Scenario 1: Undefined Instruction

Scenario 2: Segfault

Scenario 3: Infinite Loop Detection

Debugging Build Issues

Assembly Issues

Linking Issues

See Also

Get Started

Architecture

Building

Advanced

Documentation Index

​Overview

​Debugger Scripts

​GDB Script (ish-gdb.gdb)

​LLDB Script (ish-lldb.lldb)

​Crash Handler

​Implementation

​Example Crash Output

​CPU State Inspection

​In GDB

​In LLDB

​CPU State Structure

​ptraceomatic - Step Debugging Tool

​Requirements

​How It Works

​Building

​Usage

​Use Cases

​Exception Handling (iOS)

​Debugging Techniques

​Debugging Gadget Execution

​Debugging System Calls

​Memory Debugging

​Logging + Debugging Combo

​Debugging JIT/Gadget Generation

​Common Debugging Scenarios

​Scenario 1: Undefined Instruction

​Scenario 2: Segfault

​Scenario 3: Infinite Loop Detection

​Debugging Build Issues

​Assembly Issues

​Linking Issues

​See Also

Overview

Debugger Scripts

GDB Script (ish-gdb.gdb)

LLDB Script (ish-lldb.lldb)

Crash Handler

Implementation

Example Crash Output

CPU State Inspection

In GDB

In LLDB

CPU State Structure

ptraceomatic - Step Debugging Tool

Requirements

How It Works

Building

Usage

Use Cases

Exception Handling (iOS)

Debugging Techniques

Debugging Gadget Execution

Debugging System Calls

Memory Debugging

Logging + Debugging Combo

Debugging JIT/Gadget Generation

Common Debugging Scenarios

Scenario 1: Undefined Instruction

Scenario 2: Segfault

Scenario 3: Infinite Loop Detection

Debugging Build Issues

Assembly Issues

Linking Issues

See Also