Yeah, that should help. You would still need statistical tests to tell the difference between manipulation and latency issues. A vote might legitimately be included late simply because it reached the witness late.
But yeah, that would be a way to look for it.
Exactly, you are right!