After a bug was reported about the bitcoin sync stalling, I found a couple of system issues and fixed them. I also made a couple of shell scripts that loop on iguana, so even if it crashes, it just resumes. The change from 2000 blocks to 100 increased the overhead, but I managed to get the mythical 1 hour sync!
The following is the bandwidth chart, it is in kilobytes per second, so multiply by about 10 to get the megabits per second. It used to peak at over 120MB/sec as it is a 1 gigabit/sec connection for the server. But since most people dont have any connection like that, we dont lose much by slowing down the transfer rate. now a 250 megabits/sec connection should be able to replicate this performance.
I used the ./fastgenbtc script on an SSD server with 64GB of RAM. Still it ran out partway through, but it did automatically restart. Can you find in the bandwidth data where it crashed/restarted?
Now this is 1 hour to get all the data into bundle files. Still another hour or so is needed to process it and get to realtime blocks, but with the frontloading of the download, that means even if connection is lost it can continue crunching away and also that a 100 megabit per second connection should come close to the 2 hour time to realtime processing.
I notice at these settings the tmp dir only gets about 2GB, so it uses up RAM like crazy, but saves on HDD space during the sync.
James
it took 75 minutes total to get all the stragglers. At the end you just end up waiting a long time for the last blocks to come in, but at least this only happens at the end and not all the time like a serial sync ends up doing.
01:20:54 3793.74 327.89
01:21:54 2880.15 326.28
01:22:54 21622.71 386.93
01:23:54 31129.05 317.10
01:24:54 30578.95 315.52
01:25:54 11181.91 207.80
01:26:55 2724.29 162.14
01:27:55 5569.86 144.24
01:28:55 22005.42 217.63
01:29:55 28805.29 244.96
01:30:55 33015.54 295.16
01:31:55 31859.43 297.41
01:32:55 31507.55 302.09
01:33:55 33022.51 312.74
Time eth0
HH:MM:SS KB/s in KB/s out
01:34:55 30681.66 312.11
01:35:55 40202.57 273.54
01:36:55 17690.79 186.20
01:37:55 33492.91 277.29
01:38:55 29564.62 313.06
01:39:55 36833.52 331.44
01:40:55 30255.90 279.72
01:41:55 33410.82 263.10
01:42:56 29287.15 253.20
01:43:56 32937.38 262.86
01:44:56 23001.50 239.44
01:45:56 39026.84 384.33
01:46:56 40690.34 405.71
01:47:56 24515.06 324.35
01:48:56 27608.76 313.32
01:49:56 25815.32 274.26
01:50:56 28547.83 298.87
Time eth0
HH:MM:SS KB/s in KB/s out
01:51:56 35297.50 325.84
01:52:56 37797.67 360.16
01:53:56 39582.91 362.75
01:54:56 35919.24 361.97
01:55:56 36561.06 343.31
01:56:56 39791.59 357.29
01:57:56 46759.03 405.40
01:58:56 30049.32 372.31
01:59:57 24275.48 177.98
02:00:57 20154.10 160.89
02:01:57 21601.22 256.47
02:02:57 29025.60 269.12
02:03:57 34627.89 227.31
02:04:57 28082.29 195.24
02:05:57 29393.04 225.25
02:06:57 26977.68 216.65
02:07:57 41273.89 344.71
Time eth0
HH:MM:SS KB/s in KB/s out
02:08:57 24803.07 283.87
02:09:57 22699.78 222.74
02:10:57 839.88 68.85
02:11:57 32025.31 243.54
02:12:57 37059.55 245.45
02:13:57 19033.64 189.71
02:14:57 1012.05 72.38
02:15:57 1308.39 68.14
02:16:58 1437.04 70.02
02:17:58 893.00 65.82
02:18:58 1933.68 81.08
02:19:58 2435.06 86.25
02:20:58 3717.50 150.69
02:21:58 1833.38 138.06
02:22:58 1591.88 54.67
02:23:58 2126.64 95.32
02:24:58 282.25 86.92
Time eth0
HH:MM:SS KB/s in KB/s out
02:25:58 24856.70 285.44
02:26:58 34006.65 286.93
02:27:58 25836.59 218.01
02:28:58 11556.06 155.70
02:29:58 33898.82 289.15
02:30:58 42396.82 362.45
02:31:58 46498.98 414.79
02:32:58 44278.07 434.15
02:33:59 25908.16 346.61
Great updates. I like to see all the iguana progress.
great stuff James, is this running on a linux box right?
Any preparations taken ahead of segregated witness?
i develop on mac laptop running osx, but the tests are on unix boxes, ubuntu and i think debian, doesnt matter as i am not using any external libs other than pthread for the core. i only use curl for getting external website prices and that requires openssl. recently our config devs got iguana working on windows too, so it should work there too, but i dont think anybody has tried a btc sync on windows yet
i dont think segregated witness will activate anytime soon and in any case it is backwards compatible. I dont know why they dont just increase the blocksize to increase the blocksize... Implemeting a parallel blockchain to increase blocksize seems very contrived and a lot of work, so not sure when I will directly support it.
I'm not clearly understand what is iguana indeed .
This post has been linked to from another place on Steem.
Learn more about linkback bot v0.3
Upvote if you want the bot to continue posting linkbacks for your posts. Flag if otherwise. Built by @ontofractal