You are viewing a single comment's thread from:

RE: Speeding up PostgreSQL data loading

I saw how you're using ZFS for compression and therefore prioritizing ARC, that all seemed quite sensible. That Open Street Map loader application will gobble up to 90GB of RAM for its cache, and on 64GB servers I was surprised how well PostgreSQL continued to work even with shared_buffers and the working memory settings all dropped to 1GB. I hope you're able to find a useful trick or too in the talk, things like turning WAL off can be helpful when you're cycling on local development and just want the load time down. I was doing NVMe RAID0 for a while, I have a 4TB Firecuda 530 I'm planning to use for testing your code out. Keep up the good work and we're really rooting for you here. Migrating apps from SQL Server to OSS has been a good sized chunk of my life.