Hivemind Easy Deployment With hivemind-docker

[drakos]

Project Information

Repository: https://github.com/Jolly-Pirate/hivemind-docker
Project Name: hivemind-docker

Introduction

Hive (https://github.com/steemit/hivemind) is a "consensus interpretation" layer for the Steem blockchain, maintaining the state of social features such as post feeds, follows, and communities. Written in Python, it synchronizes an SQL database with chain state, providing developers with a more flexible/extensible alternative to the raw steemd API.

I'm using docker-compose for this project because of its flexibility in managing multiple containers, and to minimize the use of long complicated docker cli commands. With dependency checks in the scripts, I covered many possibilities to make a hivemind deployment easy. The tricky and frustrating part was to get postgres to use a mapped local database outside of docker volumes. The reason for that is to have better control of the database and to simplify redeployment to other servers, instead of having to redownload/reimport a dump which takes hours. Postgres is picky about permissions, plus its native command initdb was causing errors when synchronizing an imported hivemind dump. After solving the problems with some workarounds and thoroughly testing, hivemind-docker was finally ready.

---

An elephant and a whale can be friends

---

Features

• High flexibility with simple commands
• Local storage of the database for easier redeployment in an infrastructure
• Cli commands to control the containers

Requirements

• Docker Engine 18.06.0+
• PostgreSQL 10+
• 2.5GB of memory for hivemind synch process
• 250GB storage for the database

Git Installation

git clone https://github.com/Jolly-Pirate/hivemind-docker.git
cd hivemind-docker
chmod +x run.sh

Preparation

Do the following steps sequentially:

1. Create a .env file from the example and secure it
   cp .env.example .env
chmod 700 .env
2. Carefully edit all the variables in the .env file and save it
   nano .env
3. Preinstall tools and NTP synchronization
   ./run.sh preinstall
4. Install docker and docker-compose
   ./run.sh installdocker
5. Build the needed containers
   ./run.sh build

Running Postgres

Hivemind requires a postgres backend. Start postgres with these two commands consecutively:

./run.sh initdb (will initialize a fresh database cluster)
./run.sh start postgres (will start postgres using the credentials from .env)

Importing database dump

For an efficient way to get hivemind going with a short DB synchronization, download a database dump (from a daily postgres snapshot), and import it. The dump was done with PostgreSQL 10.8. ETA depends on your internet speed, storage and CPU.

NOTE: The download and import will run in screen sessions and automatically exit when complete. After the dump file is fully read, the import may seem stalled, but it's actually creating the indexes. Do not stop the process with ctrl-c or close the session . You can detach from the session with ctrl-a-d.

./run.sh importdb

ETA ~3h

Running Hivemind

After the DB import, start hivemind to synchronize the missing blocks.

./run.sh start hive

ETA ~1h

Running Hivemind and Postgres simultaneously

If you didn't do the 3 steps above, hivemind can be ran from scratch, the script will create a new postgres database, then synchronize it with an endpoint RPC server. However be advised that this approach is lenghthy, depending on your machine specs. For this plug-n-play solution, run the following command:

./run.sh start all

ETA few days

To stop the hivemind and postgres containers

./run.sh stop all

If you already imported the database, doing ./run.sh start all will resume synchronization from the last processed block.

Testing Hivemind

Once hivemind is fully synchronized, you can test it by querying it on the port you defined in .env, for example HIVEMIND_PORT=8080:

./run.sh testhive

which runs this command:

curl -s --data '[{"jsonrpc":"2.0", "method":"condenser_api.get_follow_count", "params":{"account":"initminer"}, "id":1}]' http://localhost:8080 | jq -r

and gives this result:

[
  {
    "jsonrpc": "2.0",
    "result": {
      "account": "initminer",
      "following_count": 0,
      "follower_count": 15
    },
    "id": 1
  }
]

Running Jussi

Jussi is an optional reverse proxy, its configuration won't be covered in this guide. You can check it out at https://github.com/steemit/jussi

Edit and place the DEV_config.json file in the hivemind-docker folder, then start it with:

./run.sh start jussi

You can test it with

./run.sh testjussi

Checking the logs

At any time, you can check the logs with:

./run.sh logs

Press ctrl-c to stop following the logs.

Project command options

The commands for managing the docker project are listed by typing:

./run.sh

Here's a summary of the available commands:

 preinstall    - preinstall tools and NTP synchronization
 installdocker - install docker and docker-compose
 build         - stop the running containers and (re)build all the images

 initdb        - initialize database cluster (e.g. postgresql database)
 importdb      - download and import the database dump

 start|stop|restart (e.g. start all)
           all - initdb+postgresql+hivemind
      postgres - postgresql container (with initdb dependency)
          hive - hivemind container (with postgresql dependency)
         jussi - jussi reverse proxy
 enter         - enter a container with bash shell; e.g. enter hive
 logs          - live logs of the running containers
 status        - check the containers status

 testhive      - test a hive API call to hivemind
 testjussi     - test a steemd API call to jussi

 dbsize        - check the database size
 dbactivity    - check the database activity

Proof of work done

Github account: https://github.com/Jolly-Pirate

Acknowledgment

Thanks to @emrebeyler. His guide on setting up hivemind was an inspiration for this project.

• https://steemit.com/hivemind/@emrebeyler/steps-to-restore-a-hivemind-database-snapshot
• https://steemit.com/hivemind/@emrebeyler/update-to-daily-hivemind-snapshots

Also, thanks for providing the community with a daily hivemind database snapshot.

_{Posted via https://www.palnet.io}

---

Available & Reliable. I am your Witness. I want to represent You.

🗳 If you like what I do, consider voting for me 🗳

Check my guide about Voting for Witnesses.

Go to https://steemit.com/~witnesses. Next to my name, click the once.

Alternatively you can use SteemConnect to vote for me or set me as proxy

Comments

[justyy]

Thank you very much for your contribution! I especially like your quote "An elephant and a whale can be friends" - LOL

1. It might be useful to have a wizard during installation so that we can answer questions and be prepared.
2. Can the risks of exposing Postgre password e.g. $POSTGRES_PASSWORD in the environment variable be mitigated?
3. I supsect that these scripts require sudo permissions anyway, so probably you don't need to manually add sudo before some of the commands.

+1 great work for using Docker (containerisation) !

Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.

To view those questions and the relevant answers related to your post, click here.

---

Need help? Chat with us on Discord.

[utopian-moderator]

[drakos]

1. Maybe in a future update.
2. You can chmod 700 .env for added security (I'll add that to the readme). I implemented a $POSTGRES_PASSWORD variable to avoid being prompted for the password every time you want to query the database.
3. sudo has a timeout of 15min, so it's good to have it where needed, especially in the entrypoint.sh script, which addresses the postgres permission problems when trying to make it work in docker.

[utopian-io]

Thank you for your review, @justyy! Keep up the good work!

[furion]

Have you considered running hivemind as a service, similar to SteemSQL?

[drakos]

I'm considering it, and anyone could, since it's relatively easy to set up (thanks to hivemind-docker) and not very resource hungry. But unlike SteemSQL, hive is limited to the social transactions, like comments, posts, votes.

[apshamilton]

Thanks for making this easier. I tried running Hivemind before but had all sorts of problems with Postgres and permissions.

[drakos]

Glad this will help.

[bronevik]

What problems?

Installing postgres from scratch is like 2 config edits and 2 sql queries.

[apshamilton]

I had issues installing it on a Mac.

Posted using Partiko iOS

[bronevik]

Docker on Mac has his issues too )

You may go on aws or vultr or any other vps provider, rent 1 vps for db and 1 for docker and get things done in few hours. Configure all these stuff on local machine is a good way to save $40-50 and lose 2 or 5 spare evenings.

Good luck in any case.

[apshamilton]

Thanks, but I have lots of local machines and using cloud services is the antithesis of decentralization.

Posted using Partiko iOS

[bronevik]

using cloud services is the antithesis of decentralization

I can't agree.

Modern apps and DevOps practices are completely datacenter and provider agnostic. For example any large-scale analytic cluster has redundancy and datacenter-level failover much like blockchains have.

Decentralized? Yes. Distributed? No.

[chekohler]

Well done, I read up about this a few months ago and people talking about the new possibilities hivemind will bring. It's so exciting to see it finally happening and all the other projects around scot. This place has really started to come together! When I see updates like this it really renews my faith in the project and the community

[barbara-orenya]

Je ne vais pas prétendre avoir compris le moindre mot 😄 ..mais après tout, cela n'a pas d'importance que moi je ne comprenne pas 😉

[drakos]

It's for the nerds lol

[jasonbu]

Oh, man... thanks @drakos. Was just talking about this and how it would be great if this was available!! And boom! There it is. Thanks for putting this together.

[drakos]

Your wish became true. Can you wish me to win the lottery? 😀

[teamhumble]

wow, awesome. thanks for doing this!

[steem-ua]

Hi @drakos!

Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your post is eligible for our upvote, thanks to our collaboration with @utopian-io!
Feel free to join our @steem-ua Discord server

[utopian-io]

Hey, @drakos!

Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!

Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).

Want to chat? Join us on Discord https://discord.gg/h52nFrV.

Vote for Utopian Witness!

[mahdiyari]

Good job ;)

[adityajainxds]

@mahdiyari Can I ask you something . Are you interested in small steemit project , this will improve your witness position for sure.without any investment , but definitely you will be benefited with this project as it will highlight you in steemit. You can find me on discord adityajainxds#3203. Just listen to the project if you like you can join else no problem.i can explain you here also in the comment. Thanks have a nice day.

[pibara]

Hmm, thinking if I upgraded my 2GB QNAP to 4GB, I might be able to run a tiny setup good enough to revive the daily flagwar visualization that I used to run from @pibarabot untill Steemit's rate limiting killed it. What do you think? Could this run on a 4G QNAP? And if it would, does it give access to the weighted per post vote history needed for my specific purpose?

[foxkoit]

I must go look more in this ... this sounds good :)

[oadissin]

Thank you for informing the community about the progress on this important project.
Peace

[arcange]

Congratulations @drakos!
Your post was mentioned in the Steem Hit Parade in the following category:

• Pending payout - Ranked 3 with $ 166,55

[prameshtyagi]

so do we see communities coming to life now??

[fuadsm]

What does "communities" mean in Steem?

[treeplanter]

You just planted 0.10 tree(s)!

Thanks to @ucukertz

We have planted already
8112.62 trees
out of 1,000,000

Let's save and restore Abongphen Highland Forest
in Cameroonian village Kedjom-Keku!
Plant trees with @treeplanter and get paid for it!
My Steem Power = 21540.42
Thanks a lot!
@martin.mikes coordinator of @kedjom-keku

[sweettais]

Great! Thank you for your work!

[yanes94]

Hey Pirata! It's incredible to have you back!
;)

[improv]

Scuttlebutt has it that you oppose EIP. That true? I'm looking for witnesses to vote for who will keep EIP out of HF21.

[drakos]

It is true.

[bronevik]

Docker is great but it's still STONGLY advised to configure separate Postgres instance outside Docker for any production environment.

[ahmadzareen]

I covered many possibilities to make a hivemind deployment easy.