Recurring problem with swipe to answer

Hi,

I've had this on my old iPhones as well. Occasionally when I receive a call it will not let me swipe to answer so I just have to wait then call them back later.

This has to be my single biggest gripe with the iPhone. Does anyone else have this issue or know of a way to stop it from happening again. I must stress this is a rare occurrence but one that is a problem and has been for sometime for me.