Update5 - [Nearly There]
The brave among you can try the new server at: http://52.15.210.158/home
Not this is not ssl certified yet and may need to get reset but it'll scratch the itch haha. Once the DNS updates I can get us back to happy htps and steemit dot chat domain.
UPDATE4 [9:34pm EST]
Database is being restored on the new server. Still waiting for the DNS to update (if it's not one thing it's another, eh?) but it should all come together nicely. I'll keep on till it's running again.
UPDATE3 [5pm EST]
New server is up and running. Just waiting for the DNS to redirect so I can renew certs.
UPDATE2 [3pm EST]
I have box access again and am doing a DB backup before bringing things back on line. I have been in communication with @gandalf about how to prevent an outage like this in the future. We have a plan and with the funds from this post we can make it happen. We should be back up in an hour or so and performance will be addressed shortly there after. It may require some restarts but I'd like to get things smooth again.
UPDATE:
Looks like a network card failed in or near the server. For those curious when it went down we were processing 45 Gigabytes of messages per day for about 12k users. When we migrated to this server it was around 500 Megabytes a day for around 4000 users. While we've tripled in size since then I suspect the real growth is in the active users versus those that have made accounts and never came back.
To that end once I get box access again I'll be migrating the back end database to enterprise grade hardware (AWS EBS) that can scale with some real steem!
Hey folks,
The server has been straining under the influx of new users and yesterday gave up. I am working with the service provider to get it back online enough for me to migrate it to a more powerful server that can better handle the load and scale for future load.
My apologies for the down time. We will work to restore service ASAP.
Kind Regards,
James aka Riverhead.
Thank you @riverhead for all the help and hardwork you put into running the platform! Steemit couldn't have gotten this far without your service to the community!
Damn, that's a hefty comment reward ! He he, thank you both for keeping us updated.
45 Gigabytes of messages per day!! This is super impressive! I haven't been brave enough to try the not ssl certified yet and I'll be waiting for it.
Take care James! Have fun all the while!
Been a bit crazy for us but oh well :D
It is actually a great problem to have. It means that everyone is knocking on our door! Thank you for all your help!
Here is the LIVE feed at all the people trying to get Steemit accounts.
Thanks for the notes folks and the votes. Helps pay for our new upgraded hardware :D. I've added some numbers to the update.
I have no problem contributing 100% upvote to this post, and your comment @riverhead - most of us depend on steemit.chat and I'm glad it went down temporarily for you to generate some funds to get it back up even stronger. :)
thanks for sharing this info....
I love how Steemit staff engage with the community, especially in times like this. It is great to see that they view steemers not only as users, but also as members of this warm community. It is not hard to see that they work hard to make Steemit a better place! :) Thank you @riverhead
Whoa thats huge amount of data per day
Thank you for the update. As a newcomer, a minnow only 7 days old, I am trying to learn the ropes. I was afraid that I was doing something wrong due to which I was unable to get on the chat.
It is but natural to face such growth issues when the popularity rises fast and outpaces infrastructure. The exponentially growing popularity shows how great a job all of you at Steemit are doing! Kudos. Following you.
Am more into wildlife and nature photography as well as travel. I would be honored if you see some of my work and provide your valuable comments.
Regards,
A minnow (7 days and counting)
WOWW... Its the indication of ..... how fast steemit is growing Due to your service Steemit chat..... Thanks for it @riverhead
yeah, it's better to fix it as soon as possible. because when it takes time to load, it make people frustration and simply start to ignore it.
Thank you for informing us. We're now eagerly waiting for the system back
Thanks for the update 100% upvoted from @chanthasam
You do great work @riverhead :) upvote 100% well deserved
Looking forward to it @riverhead, missing steemit.chat!
cant wait :)
Thanks James - sure you are going to solve this issue soon - the chat just is loved by so many steemians in the meantime that we miss it so much.
Great news! Also if you guys could do something to make it load faster on a phone, that would be awesome!
Phew! Thanks @riverhead, I was just about to collapse to the floor, hug myself and stick my thumb in my mouth! I was seriously distressed; glad to know you're on it like a car bonnet :-)
Cg
Thanks!
Lots of people use it, and it's great to have. The minnows are eager for it to come back.
How many more people have been trying to use the servers than before?
Thanks for the update and all your work behind the scenes.
Thank you for update its great bro thank you @rastalikelove
Thanks @riverhead, its a great service that most of us use everyday, no wonder it collapsed! lol
thank you for sharing
Thanks for the heads up :)
@riverhead @powerfire Why not rent a server and install a sql server?
i was wondering the same. Now, this make sense now... I miss the chat, though it has its annoyances haha Hope to get it back soon.
Thanks for this update @riverhead. I was wondering what happened to the chat. Thank you for all of the work you are putting in to getting it back. I very much enjoy the chat, and am truly appreciative of your work.
Good to know that. Been looking for what happen.
It's good work the chat is doing to bring the community close in some ways though there are pros and cons. One of the con is spamming links. :-)
Thats a great piece of information. Thanks or sharing it
What I love about Steem is that you can randomly run into the people that make it all keep running. Thank you for looking after Steemit Chat! I wish I had any constructive suggestion, but I think the app is in general pretty solid.
I'm glad to see Steemit Chat is growing just as rapidly as our platform!
Thanks for the hard work! A link to your post was included in the Steem.center wiki page about Steemit Chat. Thanks and good luck again!
Thank you for your support for steemitchat. We all appreciate. Kindly follow me @bewajijohnson
The unsecured version is so much more quiet :D Waiting for it to go back live again as usual :)
Can you people upvote and resteem my posts?
Thank you very much for this. I was just coming over to ask, but you already have answered.
There are a few addresses that I need that were sent to me on steemit.chat for shipping out some free steemit T-shirts, so thank you for all of your hard work and dedication.
Thank you @riverhead. Your efforts are well appreciated. Followed and Upvoted. Please do give me a favor and do the same as well.
Also, if you want to know about superstitions. I have started a new series. Here's the link
https://steemit.com/superstition/@archerized/most-common-superstitions-friday-the-13th-the-truth-and-logics-part-iii
been wondering about whats wrong with steemit.chat then i found this. thanks for posting.
@riverhead Thank you i was talking in a foreign language for the last 2 days ;-) but i am back in now he he he !!!
i am new to steemit lets follow and upvote each other, please do visit my blog once @santoshbhatta . Thanks for the updte my friend :)
aaaaaaah that's what is going on! this unavailability to register in steemchat (since I am newer and it could really help me iykwim) just drove me crazy last 3 days!
thank you for article! I am happy I found out what is going on =)
wish you alot of patience and good luck in fixing this =)
I miss you Steem Chat...
It's back! :)
Awesome, yay, back in! Thanks for your hard work @riverhead!
So who had fun with the chat. Thank you for promptly solved the problem.
Didn't even think that the chat is 12,000 people, surprisingly. It seems faster now and began to work.
There's 12k accounts. I'd bet less than half are real active users.
Then it is clear, then probably half of that -
most exchanging links :)))
steemit.chat forever :)
Amazing!
Congratulations @riverhead
You took 15 place in my Top 100 of posts
Thank you for your efforts and everything you do!
Thanks riverhead for providing hardworking service to this community,awesome bro ,....
:)
so nice and useful information thanks my friend if is possible for you please check my posts i wish the best for you and your family
Thanks for the information @riverhead
very help me
follow me @hattaarshavin