Diaspora affairs minister with history of contentious remarks made comment at meeting in US earlier this month; says he wasn't trying to be confrontational