Nearly one-fifth of potential defectors cited anti-Semitism as the reason they'd leave, but the rejected party leader has yet to address the subject