Reducing block_log size came at a price, in this case it was Storage/CPU tradeoff, we reduced space needed, at the cost of extra CPU used.
Was it worth it? Yes, I think so, it's because for most of the time (while the node is at the head block) CPU usage is negligible, and even if there's a lot of work to be done during reindex, there's not that much work that could be split between the cores to run in parallel so CPU is underutilized.
Re: those 4 ideas for block_log you've mention. Answer is: yes. Especially making it independent of storage implementation (whether it's a huge monolith (un)comporessed file, or a 1M blocks file, or one dir every 1M files with each block in separate file, or a sophisticated database structure within HAF). That's actually what we've already discussed years ago, it's just matter of priorities.
One important thing to keep in mind is that as the complexity of the implementation increases, we need proper tools with the power to manipulate it.
For example, until recently we could easily check consistency of the file at given block height with md5sum, currently even such simple task is not so easy.
For the pruning, we would have to make sure that there are proper incentives still in place in order not to wake up in the world that there's not enough seed nodes in existance to effectively obtain the full block_log.