Also, witnesses should be extra vigilant when doing maintenance, migrating/adding servers, generating new key pairs. Although they may not be malicious, the common mistake is to use the same key pair on two servers, which results in double producing.
So, always disable your witness when doing maintenance by broadcasting this key STM1111111111111111111111111111111114T1Anm, and triple check that the keys are different on every node. Re-enable the witness with the main server node's key when all is good and keep an eye on the logs until the next produced block is OK.
Here's an example of a double production (collision):
1560475ms th_a database.cpp:538 _maybe_warn_multiple ] Encountered block num collision at block 22296845 due to a fork, witnesses are: [["steem-bounty","2018-05-10T03:26:00"],["steem-bounty","2018-05-10T03:26:00"]]
Note that if the collision happens between two witnesses, it's often due to a temporary lag, for example:
28220ms th_a database.cpp:538 _maybe_warn_multiple ] Encountered block num collision at block 22214744 due to a fork, witnesses are: [["netuoso","2018-05-07T07:00:24"],["xeldal","2018-05-07T07:00:27"]]
Notice the timestamps with exactly 3 seconds difference.