After going live with the gridcoin.science website in support of the research blockchain, I decided to take another step to support my favourite things: Steemit, Gridcoin and science. Today, I added a new witness node to the network named gridcoin.science, embodying all three in a single endeavor to the mutual benefit of our communities.
A Little About Me
I am Tim, a biomedical engineer living in Christchurch, New Zealand. I have a Bachelor's Degree in Mechanical Engineering, and am currently 6 months from completing a PhD. This doctorate is focused on building computer models of the brain to better understand neurodegenerative diseases like Alzheimer's and Parkinson's. If you would like to learn more about me, I invite you to read my introductory post.
Some of what I get up to on Steemit:
I am an active moderator for the Utopian.io platform, created by @elear. In this capacity I curate Steemit content to reward high quality contributions to open-source communities. The project has already released over USD$227,506 in contributor rewards, and can be explored further on the Utopian.io homepage.
I am an active Steemit community member. My blog aims to represent, inform and progress the distributed compute communities surrounding Gridcoin and BOINC. These topics are close to my heart as a medical researcher, where my progress is often slowed by compute limitations. I aim to facilitate a cross-over between the Steemit community and the scientific community to the benefit of both.
I have been involved with the ongoing debate and analysis around vote and flagging weights, linear vs
n^2rewards, and vote buying. As the platform moves to address these issues for the betterment of the whole, I'd like to be involved in both the search and the implementation of solutions to these challenges.
Meet Nick
Running this witness with me is my close friend Nick (@deltik), a software and systems engineer living around the world – usually in Austin, Texas, or Vienna, Austria. The time difference between him and myself ensures round-the-clock support if anything were to happen to the witness server.
Nick's professional experience began with web hosting back in 2008. Since then he has branched out into managed support, custom software solutions, and cloud virtualization. He is currently working as a converged infrastructure engineer, while contributing significant resources to open-science BOINC projects like SRBase and Cosmology@home.
Despite the ability to copy conventional digital data bit for bit, that data is easily corrupted and lost.
Nick is fascinated by blockchains because in contrast, they are ideal for preserving data integrity. One such blockchain implementation is Steem, and he believes in the potential of this backend for the social aspect rather than it merely being a transaction ledger.
For this reason, Nick has joined forces with me to support the Steem blockchain with a new witness!
Witness Tech Specs
We've implemented the witness daemon a bit differently. We wanted the foundation to be rock-solid, so we started with dedicated hardware:
CPU: Intel® Xeon® Processor D-1521 @ 2.40GHz
Memory: 64GB DDR4-2666 ECC
Disk: 2×480GB SSD and 2×2TB HDD
Bandwidth: 250Mbps
Operating System: Ubuntu 16.04 LTS
Storage
On the disks, we did a custom installation of Ubuntu 16.04 LTS on a ZFS root. There are two pools that mirror each type of disk (fastpool made from the SSDs and slowpool made from the HDDs). We chose ZFS because of its unparalleled data reliability features.
What does ZFS do for us?
- The Steem blockchain is quite compressible. Since ZFS supports native compression, we were able to achieve a 37% reduction in disk usage.
- ZFS lets us take regular snapshots, which means we can roll back the witness in case of a screw-up. Hopefully there won't be any incidents, but it's best to be prepared.
- Snapshots also allow us to send backups as a stream (
zfs send/zfs receive) into flexible backup destinations. We have a 500GiB NFS backup datastore right now, which is more than enough for the witness, and we're exploring more resilient disaster recovery solutions for the future. - The ZFS Adjustable Replacement Cache (ARC) boosts performance of the ZFS datasets, which make input-output (I/O) operations on our witness faster.
- Unlike standard RAID, ZFS is aware of the data it stores and checksums all of it. Combined with the disk mirror (analogous to RAID 1), we are even protected from silent data corruption, and regular scrubs heal any inconsistent data automatically.
Virtualization
As for the witness itself, we put it in a KVM virtual machine on this dedicated server. Here are the specs of that virtual machine:
CPU: 4 logical cores of Intel Xeon D-1521
Memory: 16GiB
Disk: 400GiB SSD-backed
Bandwidth: Unmetered
Operating System: Ubuntu 16.04 LTS
Virtualization has several advantages:
- Isolation: The witness runs in its own environment so that it can't interfere with the operational infrastructure below.
- Security: If the witness is compromised by some unforeseen vulnerability, we can stop it in its tracks, go back in time, and rectify the vulnerability before it's exploited again.
- Overhead: Virtual machines don't have the hardware overhead of dedicated servers, so starting up is much faster.
- Portability: In combination with ZFS snapshots, we've opened the possibility of migrating the witness to another physical server if we ever need to upgrade.
Why not containerization?
Although steemd can make use of operating-system-level virtualization through the use of Docker, we chose full virtualization for greater isolation and to explore how far we could go with zram.
Memory
There's a problem. As of January 2018, steemd loads over 20GiB into its memory-mapped file, shared_memory.bin, but we only gave 16GiB of RAM to the virtual machine. Doesn't that mean we run out of memory or have to do unacceptably slow paging to swap?
Actually, no. We discovered that this witness runs comfortably with the help of zram. steemd's entire memory-mapped file, no matter how large the config.ini shared-file-size is set, compresses to around 8.2GiB (as of January 2018) in zram.
Thanks to the speed of zram, there is a negligible loss in Steem witness performance.
The Steem blockchain is going to keep growing, though, which means that someday, the shared memory file will be too large to fit in the 16GiB allocated to the witness virtual machine. Fortunately, the dedicated server is fully under our control and has 64GB of RAM, so we will have room to scale vertically for a while.
Room for Improvement
If the community supports this witness, we will match the commitment by developing a high availability infrastructure.
Our goal will be to minimize downtime. There are existing ways to make witnesses more robust, of which we will make use, but we would like to go above and beyond. We are considering:
- The possibility of an active/active configuration, allowing two or more witnesses to run at the same time without stepping on each other
- Shared storage, enabling live migrations so that physical servers can be taken down while the virtual machines stay up
- Faster ways to redeploy
steemd, maybe obviating the need for--replay-blockchain
Closing Remarks
If you have made it this far - thank you for reading!
To vote, visit https://steemit.com/~witnesses and add gridcoin.science to the box at the bottom of the page, click vote, and authorise using your Active Key.
Wrapping things up, I'd like to invite you to ask @deltik or me anything in the comments below. Let's get to know each other better!
Nice!
Welcome in, I'm sure you'll perform a great witness system for the Steem blockchain. You've got my vote.
I'm always around if you've got any witness related issues or questions.
Thanks Startail - I'm sure there will be questions down the line. I really want this witness to be the best it can be, so if you pick up anywhere it is lacking during your scans I would love to know! Constructive criticism will always be appreciated - how else does one improve?
You have my continued support too, for what it's worth. :)
Hi @dutch,
very nice!
Iam absolutely supporting this, resteemed, 100% upvoted and the next thing i will do after finishing this comment is voting for witness also ;)
Thanks alot for doing such a great job in all those fields of work that you mentioned.
We are happy to have you here !
Kind regards
Jan
Thanks Jan, and more importantly thank you for all you do to help drive Utopian.io forward. That platform has a lot of potential in it yet, and people like you are the ones who can help get it there.
Looking forward to working more with you in the future.
Hoorah! As you already know - you certainly have my vote! Do let me know if I can provide support in any way :)
Great to have a representative of the Gridcoin Team in the board of Steemit Witnesses. You have my vote!
Why you need high availability? Can't you just make two witnesses and rely on the mesing/p2p capability of blockchain?
Also the fact that the daemon needs 20 Gibibytes of RAM makes me wonder if it is capable of scaling much over to what it is now.
See, I am not very positive. But Have an up-vote.
No worries; your skepticism is valid.
High availability is necessary because of a few things we noticed:
If we can come up with an active/active setup, witness failover would not be necessary, and perhaps it could almost eliminate missed blocks.
Yes, the daemon needs many gibibytes of RAM, and that requirement is going to keep growing unless there's an overhaul of what gets loaded into RAM. For now, we must scale vertically.
The use of zram is a mitigation that buys us quite a bit more time before we eventually run out of RAM. Hopefully a solution comes up before we need to upgrade physical RAM.
voted. good luck on the witness run! :)
Hey - thanks man! Good to see you outside Discord. :)
Amazing work and as always a very clear and engaging peace to read, even by someone who doesn't have all the skills and information you do. Thanks for making it understandable, and thanks for your hard work!
Thank you for the kind words @minersean. I see you have very recently started contributing Dutch translations to Utopian.io - great work! I hope to see even more content from you on the platform in future.
I did! I'm just waiting for the work load for you moderators to go down a bit before I start more work, I can see that there's a lot of contributions you still have to look at haha!
Outstanding initiative, @dutch! (by the way: nice to Steem you! :-)). What you offered us here actually is a Witness setup tutorial, no doubt it'll be a great resource for future (and current) Witnesses!
I'm gonna send you my Witness vote right now. Regarding that voting procedure, do you know how I could support the community adoption of @drako's databoard (where ALL existing Witnesses are listed, alongside their relevant activity parameters, here:
https://steemian.info/witnesses), to replace Steemit's default interface, where the 50 first Witnesses only are available)?
Thank you for the kind words, although it falls somewhat short of a set-up tutorial. There are some very high quality set-up guides out there though. In this post I hoped to highlight the points of difference this witness brings to the table, and how they can be implemented to help the network scale more efficiently.
The best way to promote any development is to ensure the pull request is made on GitHub and an open dialogue is established there, as well as on Steemchat. Are you sure that is the correct profile? The user does not appear to have any development content.
For now, you can already view a complete witness list here.
Many thanks for your precise and thorough answer - it brings me more confidence in Steem's future fate to see this network may rely on the work and support of that dedicated members like @deltik and yourself (and it makes me hope that someday the Steemit community will be able to convert it to the free software regime).
And yes, the Witness list I was refering to is the one you mentioned (made by @drakos, with an -s at the end :-)).
Just to clarify:
Do you mean to use Gridcoin.science as a proxy for voting or as a vote for @gridcoin.science
The second - @gridcoin.science is the witness that @deltik and myself established.
Why do you need to create another account for steemit for voting when there is actually no mutual benefit for anybody except for the ones who control the rewards/votes? This is just more manufactured consensus and manipulation of steemit based upon an unofficial website? This whole thing is exactly what we talked about in the last controversial Gridcoin hangout. Meanwhile, @dutch you still haven't replied to me in my own thread. All I see is more of the same behavior to back up my talking points around why I think steemit/bitshares is parasitic towards Gridcoin at this point. Especially when there is no difference between funding the hangouts than there is funding marketing initiatives. People should be funding this stuff by themselves when we say this platform is what we supposedly use to fund it. The statements are extremely contradictory when you decide to act like it's important and act two-faced when (foundation) funds should only be meant for developers on github.
The reason for creating another account is to separate my active profile from the witness profile, which will be used for other automated activities in future. This also generates separation between my personal funds and any funds the witness generates through the minting of blocks. I want this witness to follow the example set by @lukestokes, delegating any SP it may generate back into projects that build up the Steemit community.
With regard to your claim of parasitism, neither of the platforms I believe in and pour myself into are 'parasitic':
Steemit is a platform that supports an exponentially growing community, able to govern and pay for itself. This platform enables untold things covering science, the arts, sharing of quality news, blogging, and even just unwinding and having a good laugh together. The content creators are rewarded, and the maintainers of the blockchain are too.
Gridcoin is a blockchain not dissimilar to the Steem blockchain, except it rewards contributions to open-source science made by donating compute power. The Gridcoin compute network has outgrown the most powerful supercomputers in the world, and is advancing many fields of research at a phenominal rate. Contributors work on literally everything from mapping the Milky Way to curing the Zika virus - with success! The researchers (classically referred to as 'miners') are rewarded, and the nodes securing the blockchain are too.
Clearly, on their own these platforms are beneficial to their communities, and humanity as a whole. Combining the two yields even further benefits. The Gridcoin community already has a strong presence on Steemit, and I witness daily crossover. Steemians become researchers in the Gridcoin network, and Gridcoin researchers become active members of Steemit. Both communities are extremely altruistic in their aims, and they are building each other up.
Steemit Inc. is a company (among many, like Bitshares) and comparing their tokens to Gridcoin is not the same and they are extremely unlike each other. I don't think that's how the blockchain should be used because they are completely separate things. Meanwhile, the work done here on steemit actually has caused a problem lately that you still haven't addressed. Such as the drama and the personal divides I see happening between the users who tout this site to be so beneficial. As soon as things get personal it is no longer the same narrative. The hypocrisy is astounding to me and when I see you no longer acknowledging (and are misdirecting about) these things you are clearly nonobjective towards anything I have brought you. I still await for a reply from you in my thread about this, too.
Then what are you doing here on Steemit? Log off and go to any social media you find more suitable. This is not your malodorous IRC hangout where you are obviously used to such nonsense, you'll get flagged very soon here. Don't come crying afterwards you are a victim.
You are going to silence me based upon your disagreement with me? That's just fueling more of exactly what I'm talking about here. I'm saying it is an opinion of mine based upon what I think. There is clearly a group of people at large who would like to control a narrative here on steemit. I don't need you steemit money, I would rather use it for the immutability it provides. Even then, I don't get to have it because the vote manipulation is also coinciding with the manipulation of information. Hence the threaten to flag, I don't see the point. You guys are really acting like you have done nothing wrong when it's really just everybody being too dependent on steemit. I'm not going to follow the narrative and you can try to control it with empty threats.
Well, that's easily settled, don't worry.
The votes/tokens/money is what gives the exposure to every post. While I am sure you are aware, which is the point. I don't know why neither you nor @dutch have continued to ignore all this here. The both of you are trying to avoid and misdirect from the valid confrontation I have here. I am not interested in dragging it out in the Gridcoin Slack where I know dutch has tried to do with me by initially bringing it up in the general chat.
Good post, you have very interesting scientific background!
Voted.
I have 25 votes left, so if somebody else from Gridcoin community is taking part - let's add some votes!
@deltik - I would advise you to make at least introduction post, so people can ask you questions there.
By the way, I'm planning to visit Vienna this year, is there any Gridcoin or Steemit meetups going on?
As mentioned by @dutch, I've been a steem witness for a while. You can read my latest project update at https://steemit.com/witness/@sc-steemit/my-bitshares-and-steem-witness-project-update-and-future-plans
At the bottom of that post you find my initial witness deceleration for both steem and bitshares.
Actually I already voted for you before, I think several months ago
I hope other readers of this post will vote too!
There is another member from the Gridcoin community who has been running a witness for quite some time now - @sc-steemit. They also generate reports on the witness list, the latest of which can be found here.
Thanks, actually I already voted for him before, I think several months ago
I
You'll get mine too!
This is great news!
Sometimes I wonder if you do sleep from time to time at all...?
Interesting. I'm new in steemit but I can see that this steem thing seems full of potential for a never ending big spectrum of things. :)
Awesome!
Awesome! I'm voting for you!
Great involvement and dedication in making the STEEMIT platform a great place to be. The witness concept and setup is as good as it can be and comes just to support the entire blockchain and make it stable enough in moments of hard times. Saw many involvements, wish i could do the same...
Yes!!! Thanks for all your work for the communities.
Super excited to see this!! You have my vote!
For my point of view the technical implementation sounds reasonable and the hardware should be more then enough.
You got my vote.
You're amazing...I think you need no sleep so you have the time to so much...
You got my vote :)
Thanks @dutch. Here is a vote for science!
@dutch nice post i really liked that i am agree with your post
Incredible work @dutch and @deltik! You have my vote!
upvoted... and voted for!
You have my vote :)
Awesome! You have my vote!
hey @dutch. I've voted for gridcoin.science as witness. Look forward to seeing how things turn out for gridcoin!
nice posts.I will always follow up and vote. ☺️ please see my profile and vote
Hi Dutch, very interesting read. You do so much with Gridcoin and now Steemit too. You explained this very nicely, thanks! Here's to a long and distinguished career after your phd :)
Hi, you seem to know what you're doing, so you got my vote for witness.
Great work here.
Thanks for upvoting my introductory post.
I just followed you, I'm now officially your fan, pls you can also follow me so we meet more often. Thanks
Thanks for sharing this nice post. Would you please look at my post?
https://steemit.com/music/@sepehrkia/the-best-songs-ever-2-camel-rajaz