Debugger won't hit a re-tried activity

This is a problem I've had on and off for months but never really figured out what the issue is.

Debugging on processes seems to work fine on the first run through, or even sometimes on the first retry (if an activity errors), but after that the debugger will refuse to pick up / hook into anything when you push the 'retry' button.

I have tried breakpoints and debuggers all over the place, in ways that normally work, but I nothing is letting me in.

I hit 'retry' on the activity in Service Centre, and I see the error appear in the log (so I know it executed) but I can't seem to actually debug it. It seems if I want to debug an activity I need to make an entirely new process flow through to this point, which is incredibly frustrating and time consuming when I'm deep in a complex business process.

Are there any hints or suggestions for actually making this debugger work on re-tried activities?