Indeed, the plant can only put so many resources toward seed production. When too many seeds are started on the plant, the size of the seeds can be extremely small.
I shoot for pollinating when the plants are bursting with white pistils, and still have at least 5-6 weeks of flowering time left. :)