I want to inform you that due to a recent cyber attack, the HiveSQL infrastructure was taken down yesterday around noon (UTC).
The (not) funny thing is that I was doing a presentation about Hive and the services I created on top of Hive when it happened. I was embarrassed to see some hiccups, but I couldn't decently drop everything immediately to see what was wrong. I cut this one short though...you know...experience and foreboding.
Since then I have worked tirelessly to fix the problem and restore full functionality, which required restoring the system from a backup.
I apologize for any inconvenience this may have caused. We are taking steps to prevent similar incidents in the future. Thank you for your patience and understanding as we work to get everything back up and running.
UPDATE:
As of this afternoon around 5pm UTC, HiveSQL is fully recovered and working normally. All applications that rely on this service are therefore also operational.
Thank you to everyone who supported me during these stressful times.
Attack Description
Since HiveSQL is a community-funded service, I find it important to be as transparent as possible and to share my experience with others, hoping this can help them not to fall victim to a similar issue.
A wave of attacks is currently targeting ESXi servers. These attacks are detected globally and especially in Europe. According to experts from the ecosystem as well as authorities, they might be related to Nevada ransomware and use CVE-2021-21974 as a compromise vector.
Despite the protection systems in place, the attackers succeeded to access one of the infrastructure's servers, probably using another internal host under their control, although investigations are still underway to confirm these assumptions.
The attacker took the opportunity to encrypt the disks of virtual machines, making them unusable unless I paid a ransom of around $50,000. Bad luck for them, the HiveSQL infrastructure is regularly backed up on a remote and secure site.
However, before restoring the backup, I took care with our service provider to identify the operating mode of the attack so that it could not be executed again once the infrastructure was restored.
Once the identified security holes were closed, we were able to start the restoration process. Given the volume of data handled by HiveSQL (over 4TB), this is still ongoing at the time of writing.
The attack was what I consider a surface attack, meaning that it targeted the infrastructure itself but did not result in the exfiltration of any data. An analysis is underway to validate this presumption. But even if this turns out to be correct, we are currently performing an audit of sensitive information and will reset any of them.
Lesson learned
As the manager of HiveSQL, I have been closely monitoring its performance and security for the past 6 years. Despite constant attempts of attack and intrusion, the infrastructure has remained strong and reliable. That is, until now.
I can say that I was taken off guard by this issue. However, I am determined to learn from this experience and take the necessary steps to prevent similar incidents from happening in the future. I can assure you that I'm taking every measure to ensure its security and stability.
1. Backups, Backups, ... and Backups
The backup system put in place saved the day!
I have always placed a strong emphasis on the importance of regular backups. Backups provide a crucial safety net for any infrastructure and ensure that valuable data and information are protected in the event of an unexpected outage.
This incident highlights the significance of regular backups. By having a reliable backup system in place, we were able to quickly recover from the attack and minimize its impact. This serves as a powerful reminder of the importance of having a robust backup strategy in place, as well as the need for constant monitoring and updating of security systems.
2. Insider Threats - The Possibility of an Attack From Within
Our protection against outside attacks has proven to be highly effective, providing a secure barrier for the past 6 years.
However, it is important to consider the possibility that the attack could have originated from within the data center. We realized that our servers rented in the data center were not completely isolated in a network silo, thus offering a potential attack surface to other machines physically connected to the same network.
It is crucial to consider the possibility of an insider threat and investigations are also carried out in this direction. As a precaution, I am conducting a thorough analysis and implementing additional security measures to ensure the protection of our systems and data.
3. Technology watch
Keeping software up to date is a critical component of maintaining the security and stability of any infrastructure. Technology watch and patching software are essential tools for staying ahead of potential threats and fixing vulnerabilities as soon as they are discovered.
Despite my diligent efforts in that, I recently missed an important patch that left one of our servers vulnerable to attack. This serves as a reminder that no security measures are foolproof, and even the most proactive approach to security can sometimes fall short. I am taking steps to ensure that all critical patches are promptly installed in the future to minimize the risk of similar incidents happening again.
Conclusion
I understand that this downtime may have caused some inconvenience to users and apps relying on HiveSQL, and I would like to extend my sincerest apologies. I can assure you that I'm constantly working to improve and provide the best service possible.
Thank you for your continued support and understanding. I look forward to the next 6 years of providing a secure and reliable HiveSQL infrastructure
I will let you know as soon as this recovery process is complete. All communication and support related to this issue will be done in the HiveSQL Discord Channel
Check out my apps and services
Thanks for the update and for all you do for the chain!
Thank you @thekittygirl
I was working abroad when I saw HiveSQL was down and at first, I thought maybe it has to do with the WiFi I was connecting as they could have some restrictions from the firewall, but now I see it was a real problem. Good that the issues were identified and thus packed up and we'll wait patiently for the process to finish.
Technically speaking, you cannot exclude a potential problem with the WiFi or the firewall without having duly validated it was not.
Humanly speaking, that would have been a goddamn coincidence.
I gifted $PIZZA slices here:
@pixresteemer(4/5) tipped @arcange (x1)
Please vote for pizza.witness!
Looks like you’re prepared for anything! Thank you for the update 💙
I wouldn't use the quantifier "anything", because the scope is too wide and attackers are always one step ahead. But yes... I'm prepared.
They’re sure keeping you on your toes, which is also of benefit to you, as you learn and grow each day 💙
Esxi, makes total sense. There has been a lot of recent issues.
Do you have it closed off to the Internet so the only way would be internal network (like you said the data center)?
Have you considered migrating to qemu/kvm?
You running MS SQL on Windows or Linux?
Yes, hence the reason for suspecting a side attack.
Yes, but no thanks. HiveSQL is an infrastructure, not just one host. In that case, it is not only the hypervisor software that matters but also all the related components, including the sysops skills and knowledge. This extends to the disaster recovery plan as a whole.
Much appreciated and thank you for all the details.
Security is always an ongoing process, and such breaches are expected to happen some times.
Out of curiosity, how much harder would it have been to pull all the data again from the blockchain if there were no backups?
Again, thank you man!
I wouldn't be here replying to your comment 😓
It's not the question if, but when you het a ransomware attack nowadays.
Good that you have your backups in place.
Although I have read stories where hackers try to get in your network to delete the backups and then release the ransomeare.
I'm looking into an extra immutable read-only backup in an S3 cloud storage solution for these cases.
HiveSQL backups are indeed stored on a separate and well-secured site/infra.
My concerns were more about whether the restore would work and how long it would take. It had been a long time since I had done a Disaster Recovery test given the work and hardware resources required. So it was kind of Proof-of-Fire 😅
It was a good test then 😄👍
Thank you for your witness vote!
Have a !BEER on me!
To Opt-Out of my witness beer program just comment STOP below
Sorry to hear that. This must be a constant concern for anyone with online systems. You just have to be prepared for the worst as you cannot anticipate every kind of attack. I use HiveSQL, but do not rely on it for anything vital. I hope you can resolve this soon anyway.
When I made HiveSQL public, a lot of people told me I was crazy. I can tell you that the attempts of intrusions are not lacking.
So far I've done well to fend them off...
Anyway, the additional experience acquired thanks to this unfortunate event will only complicate their task.
I'm definitely not ready to host a public service. I know little about how to defend it. Those involving financial risk have to be prime targets. I assume with HiveSQL it's more about getting ransom money if they can take it down as you are just serving data that is public anyway. Good luck with it.
Damn, I'm glad everything's fine and no harm has been done. The timing is extremely suspicious though. Maybe it's not a coincidence. 🤔
I don't see why the timing would be suspicious and what it would coincide with. Could you enlighten me?
I find it strange that it happened while you were doing a presentation about Hive. It feels like sabotage.
Don't get me wrong wrong but i didn't know there were people more paranoid than me (coming from a security minded person, that's a compliment).
The audience of my presentation was relatively small and I doubt that the attackers were aware of it 😉
Thanks for the update and taking care of things!
Communication is important in times of crisis. The most difficult thing is to insert it in the long list of things that need to be done and seem to have a higher priority. 😅
Thanks for all the work behind the scenes!
Thank you @relf87
Your contributions will always be valuable, dear @arcange we always win when we receive your proper guidance, in particular I am pleased to have your important and correct opinions and comments about Hive and this great world of cryptocurrencies, which has been of great support and complement to live, with the cures we receive.
Thank you @omarrojas
Dear @arcange I will always be grateful for your very positive writings and the evaluations you give to my publications. You are a great support, to solve the low economic conditions of those of us who are in hive and live in Venezuela. I will always wish you much success in your publications and cures.
Thank you for your kind words @omarrojas
We will always be grateful for your excellent curation work @arcange , which motivates us to continue developing important publications. successes.
Merci pour les infos détaillées. Je ne savais pas que HiveSQL était sur ESXi, cela ne te génère pas de problème de perf car je suppose qu'HiveSQL est extremement sollicité? J'espère ne pas avoir de pb avec le ESXi que j'ai pour un client.
Avec une bonne configuration de l'ESXi, on ne constate pas de différence de performance notoire. Par contre, les avantages procurés par ce type de déploiement sont indéniables.
Merci pour l'info qui tombe bien, j'ai un deploiement d'une architecture un peu complexe à faire et qui sera beaucoup sollicité, je refléchissais donc à l'utiliser, mais j'étais inquiet d'une grosse différence de performance n'ayant pas l'info via l'ESXi que j'utilise actuellement car peu sollicité du fait de la taille du client.
Man... shit happens @arcange
I can see by your detailed reporting that you're kinda feeling some responsibility for the vulnerability due to the patching issue, but all I can say is - we're all human.
You're one of the most diligent, hard-working and talented devs I've met here on hive. I remember chatting to you about hive SQL back in Krakow and in a 15-minute conversation I learned more about the intricacies of the blockchain querying functions I'd been using to narrow down potential Curie-worthy posts for submission than I'd learned in the previous year of self-teaching/experimenting. Although I've got to shout out @carlgnash for being super helpful tutoring in the beginning. The technical stuff is very difficult for me, and between his tutorage and my convo with you at SF3 I became very confident in querying the blockchain with SQL.
Anyway, all I'm really saying is don't beat yourself up too much about this attack. It takes someone of integrity to admit if they made a mistake, but as I said above we're all prone to them now and then. In the great debate about AI, perhaps this is one positive thing AI will bring to the table, the ability to manage complex systems such as SQL and other complex database systems, without the human fallibility that comes with stress, the way our brain works and the inevitable decline in efficiency that comes with age. I know I'm feeling that last one these days.
Good on you for noticing the attack quickly and working hard to identify the extent of the attack, close the vulnerabilities, and restore the integrity of the data with backups.
All the best 👍🙂
Thank you for your comment @raj808
I remember meeting you in Kraków and I am happy to read that our conversation about HiveSQL and writing queries was fruitful for you.
Honestly, I don't beat myself about this attack, quite the contrary. Knowing how the infrastructure is built and how it intertwines with other services, I'm quite proud of myself.
But In terms of security, you have to be constantly vigilant and what happened tickled me at a time when this vigilance might have slowly died down. On the other hand, it allowed me to validate once more, and in real stress conditions the Disaster Recovery plan that I had put in place. In ~24 hours, the services were restored, which is a good score I could have done better, but since this was a disaster with prior intrusion, it was necessary to allocate time for an infrastructure security assessment.
Your thinking on protecting systems with AI is interesting. I'm not sure that solves all the problems because it is surprising to see the creativity that humans can show, especially bad people when they are motivated. But I am certain that AI will contribute to better security and in any case improve the level of vigilance.
Have a good week-end
Assholes. :/
Sorry to hear of it. Must be really stressful.
Thanks for everything you do around here. Really. This can't have been an easy time for you. <3
Yeah... backup, backup, backup! (learned the hard way... also... backup!)
You've got this. 💥
Yeah, backups... but a validated recovery plan too 😓
Painful :(
People can be such a-holes. Sorry about your week. You deserve better, my friend <3
From the CVE mentioned.
The usual problems I am guessing, like bridged networks and/or access to other hosts with logins. Usually, the best is to isolate networks and only expose host ports that result in services. Examples:
INFO: Public VMware infrastructure should always have different Layer 2 networks between VMs and the infrastructure services, hence not making the port available from a normal user traffic perspective (which might be where the exploits might come from). Cloud is terrible for this problem because they don't often offer this kind of infrastructure.
Many don't realize that the bare metal hosts they rent in a data center aren't isolated from the internal network and other hosts that are on the same switch. Hosting providers should draw their client's attention to this.
Yeah... I deal with infrastructure (and virtual solutions) at public level for many users, so therefore my heads up for many trying to use cloud or other service providers.
Excellent post @arcange, thank you for explaining.
The necessity of backups reminds me of the former https://MySpace.com, which I only recently discovered had a major data breach way back when, ultimately losing their users data.
I also see that you have a Recovery and AutoClaim app.. are both of them open source?
I'm glad to know you just a little bit more now, and I'm interested in attending https://HiveFe.st with @nathansenn and other @dbuzz team members. Thank you for all that you do.
Amazing work . . yes, I only found out this year what happened to #MySpace
Posted via D.Buzz