The startup's software mimics human decision-making to spot anomalies and detect money laundering