For the exact reason you mentioned... we went there in the morning. It was quite pleasant and we enjoyed many of the paintings that were there. So many Dutch artists in one place :)
However, with time passing the museum got more and more packed, so that we did not stay very long in the afternoon. Also we didn't want to miss the closing dinner ;)