Video-sharing app under fire for failing to curb hate; official says it might be 'insurmountable challenge,' but company committed to trying