Families held for hours after crossing back from Canada amid security warnings that Iran might retaliate for killing of top general; US customs denies targeting them