Merrill relied on third-party automated systems that were insufficient to monitor for manipulative trades, FINRA says.