The python debugger provides a great way to "peer" inside your running python processes. If you just want to start a program, add breakpoints to it and do your debugging you can use pdb just like you would use gdb:

usage: pdb.py scriptfile [arg] ...

Drop to PDB on Signal

Running a program within pdb provides a lot of ability to see what’s happening in your program. However, sometimes you may want to see what a program is doing even though you haven’t run it with pdb. Or perhaps you have an intermittent problem and you want the program to run like normal until you encounter the problem.

If the program that you want to interrupt and debug is run as a foreground process (attached to a ptty) then this recipe should do the trick.

import code, traceback, signal

def debug(sig, frame):
    """Interrupt running process, and provide a python prompt for
    interactive debugging."""
    d={'_frame':frame}         # Allow access to frame object.
    d.update(frame.f_globals)  # Unless shadowed by global
    d.update(frame.f_locals)

    i = code.InteractiveConsole(d)
    message  = "Signal recieved : entering python shell.\nTraceback:\n"
    message += ''.join(traceback.format_stack(frame))
    i.interact(message)

def listen():
    signal.signal(signal.SIGUSR1, debug)  # Register handler

Note

The above code was provided by Brian McErlean at stackoverflow.com

To use this, all your program has to do is call listen() to set up the signal handler. Whenever you need to debug the process run the bash command:

kill -s USR1 <pid>   # where pid is that of the program to debug

After running this command the ptty that the program is attached to will display a >>> python interpreter. Using dir() you can see the environment and poke around the stack.

Piped PDB on Signal

In order for the above solution to work the program to debug needs to be a foreground process. If the process you need to debug is running in the background then you need some additional code to allow the process to communicate via a unix pipe to your ptty. The person that wrote the above solution also provide a solution for this as well.

try: import readline  # For readline input support
except: pass

import sys, os, traceback, signal, codeop, cStringIO, cPickle, tempfile

def pipename(pid):
    """Return name of pipe to use"""
    return os.path.join(tempfile.gettempdir(), 'debug-%d' % pid)

class NamedPipe(object):
    def __init__(self, name, end=0, mode=0666):
        """Open a pair of pipes, name.in and name.out for communication
        with another process.  One process should pass 1 for end, and the
        other 0.  Data is marshalled with pickle."""
        self.in_name, self.out_name = name +'.in',  name +'.out',
        try: os.mkfifo(self.in_name,mode)
        except OSError: pass
        try: os.mkfifo(self.out_name,mode)
        except OSError: pass

        # NOTE: The order the ends are opened in is important - both ends
        # of pipe 1 must be opened before the second pipe can be opened.
        if end:
            self.inp = open(self.out_name,'r')
            self.out = open(self.in_name,'w')
        else:
            self.out = open(self.out_name,'w')
            self.inp = open(self.in_name,'r')
        self._open = True

    def is_open(self):
        return not (self.inp.closed or self.out.closed)

    def put(self,msg):
        if self.is_open():
            data = cPickle.dumps(msg,1)
            self.out.write("%d\n" % len(data))
            self.out.write(data)
            self.out.flush()
        else:
            raise Exception("Pipe closed")

    def get(self):
        txt=self.inp.readline()
        if not txt:
            self.inp.close()
        else:
            l = int(txt)
            data=self.inp.read(l)
            if len(data) < l: self.inp.close()
            return cPickle.loads(data)  # Convert back to python object.

    def close(self):
        self.inp.close()
        self.out.close()
        try: os.remove(self.in_name)
        except OSError: pass
        try: os.remove(self.out_name)
        except OSError: pass

    def __del__(self):
        self.close()

def remote_debug(sig,frame):
    """Handler to allow process to be remotely debugged."""
    def _raiseEx(ex):
        """Raise specified exception in the remote process"""
        _raiseEx.ex = ex
    _raiseEx.ex = None

    try:
        # Provide some useful functions.
        locs = {'_raiseEx' : _raiseEx}
        locs.update(frame.f_locals)  # Unless shadowed.
        globs = frame.f_globals

        pid = os.getpid()  # Use pipe name based on pid
        pipe = NamedPipe(pipename(pid))

        old_stdout, old_stderr = sys.stdout, sys.stderr
        txt = ''
        pipe.put("Interrupting process at following point:\n" +
               ''.join(traceback.format_stack(frame)) + ">>> ")

        try:
            while pipe.is_open() and _raiseEx.ex is None:
                line = pipe.get()
                if line is None: continue # EOF
                txt += line
                try:
                    code = codeop.compile_command(txt)
                    if code:
                        sys.stdout = cStringIO.StringIO()
                        sys.stderr = sys.stdout
                        exec code in globs,locs
                        txt = ''
                        pipe.put(sys.stdout.getvalue() + '>>> ')
                    else:
                        pipe.put('... ')
                except:
                    txt='' # May be syntax err.
                    sys.stdout = cStringIO.StringIO()
                    sys.stderr = sys.stdout
                    traceback.print_exc()
                    pipe.put(sys.stdout.getvalue() + '>>> ')
        finally:
            sys.stdout = old_stdout # Restore redirected output.
            sys.stderr = old_stderr
            pipe.close()

    except Exception:  # Don't allow debug exceptions to propogate to real program.
        traceback.print_exc()

    if _raiseEx.ex is not None: raise _raiseEx.ex

def debug_process(pid):
    """Interrupt a running process and debug it."""
    os.kill(pid, signal.SIGUSR1)  # Signal process.
    pipe = NamedPipe(pipename(pid), 1)
    try:
        while pipe.is_open():
            txt=raw_input(pipe.get()) + '\n'
            pipe.put(txt)
    except EOFError:
        pass # Exit.
    pipe.close()

def listen():
    signal.signal(signal.SIGUSR1, remote_debug) # Register for remote debugging.

if __name__=='__main__':
    if len(sys.argv) != 2:
        print "Error: Must provide process id to debug"
    else:
        pid = int(sys.argv[1])
        debug_process(pid)

Provided by Brian McErlean: code.activestate.com

Its implemented by first sending the process to debug a signal, and then opening a pair of pipes with the name /tmp/debug-pid.in and /tmp/debug-pid.out. The remote process, on receiving the signal, opens the other end of this pipe and these are used to pass code to be executed from the debugging process, and read responses from the debugee.

There are a few warnings to make:

  • There is absolutely no security here - pretty much anyone who can write to the pipe can gain full control of any process using this. Use only for developer environments, not live systems!
  • Sending a signal can interrupt whatever I/O or activity the process is currently doing, so you won’t always just be able to detach again and let it run unchanged.
  • It uses signals to wake the process, so currently only works on unix-like systems that support this.
  • Untested with threads
" "