PM says 'scheduling constraints' caused election cycle's second short-notice cancellation, fueling rumors he's preparing for a different high-profile visit