Officials say comments by William Latson had become a distraction for the community