02/07/2019 incident Report

[likwid]

The bug took place the 2nd of July 2019 at around 18:00PM.

We apologize to all affected customers. We are making sure it will not happen again.

Service should be back to normal within 4h after this post publish time.

@samotonakatoshi was the first person to give notice about the incident.

Reason

First of all, likwid was not hacked.

The reason was a bad coding practice. To be specific, a bad error handling.

All started when https://dev.steemit.com rpc node gave a random 502 bad request error. Likwid was set to re-invoke itself in such cases. That ended up with multiple Likwid instances running in parallel, which created an exponential endless loop of self-invoking likwid instances. Having that many asynchronous instances made impossible for the database and fail-safe features to prevent multiple redundant payouts.

Final result and lost balance

We found out about the issue quick enough and the server was brought down within the first 20min after the issue-trigger.

The following table contains the different likwid customer accounts who got multiple transfers.

author	STEEM	SBD
@cryptotokeneur	0.66	0
@snuff1	0.52	0
@taifkhansent back	54.339	19.452
@threespeakwallet sent back	15.725	5.628
@dongkong sent back	1.494	4.176
@kabasakal sent back	27.618	9.886
@ura-soul paid back	70.457	25.222
@elsiekjay	0.607	0.217
@mahdier sent back	153.472	54.938
@wehmoen sent back	394.068	141.059

For all affected accounts, we would appreciate if you can send back the lost funds. Please let us know when funds are sent back. Thanks a lot in advance.

Methodology

For the sake of transparency and compliance, please find below the foresincs snippet code that has been used (nodeJS + dsteem library) to identify the lost founds:

async function forensics () {
    var transfers = []
    let history = await client.database.call('get_account_history', ['likwid', -1, 1000])
    var duplicates = []
    transfers = history.filter((x) => {return  (x[1].op[0] == 'transfer' && x[1].op[1].from == 'likwid')}).map((x) => x[1].op[1])
    const unique = [...new Set(transfers.map(item => item.to))]

    for (let i = 0; i < transfers.length; i++) {
        let el = transfers[i]
        let _duplicates = transfers.filter((x) => { return x.to == el.to && x.amount == el.amount})
        if (_duplicates.length > 1) {
            duplicates.push(...new Set(_duplicates))
        }
    }
    let final_duplicates = []
    final_duplicates.push(...new Set(duplicates))
    for (let i = 0; i < unique.length; i++) {
        let author = unique[i]
        let _transfers = final_duplicates.filter((x) => x.to == author)
        if (_transfers.length > 1) {
            console.log(author)
            sbd_transfers = _transfers.filter((x) => getCurrency(x.amount) == 'SBD')
            if (sbd_transfers.length > 1) {
                sbd_transfers.shift()
                total_sbd_debt = sbd_transfers.reduce((a, b) => { return { amount: parseFloat(a.amount) + parseFloat(b.amount) } })
                console.log(parseFloat(total_sbd_debt.amount) + ' SBD')
            }
            steem_transfers = _transfers.filter((x) => getCurrency(x.amount) == 'STEEM')
            if (steem_transfers.length > 1) {
                steem_transfers.shift()
                total_steem_debt = steem_transfers.reduce((a, b) => { return { amount: parseFloat(a.amount) + parseFloat(b.amount) } })
                console.log(parseFloat(total_steem_debt.amount) + ' STEEM')
            }
        }
    }
}

Comments

[ura-soul]

From what I can see I was paid twice for one post but then not paid for a later post.

This one was paid twice: https://steemit.com/@ura-soul/pkoshmbj
(25.222 SBD + 70.457 STEEM)

This one has not been paid: https://steempeak.com/@ura-soul/nkuppeoy
(23.575 SBD + 68.140 SP)

So I make that an outstanding balance that I need to pay of: 1.647SBD + 2.317 STEEM

It would be easier for me to just pay the final amount in SBD as I Powered up the liquid STEEM already. Let me know what you want to do.

[likwid]

Hello, could you please join our discord likwid channel? https://discord.gg/YzAb4dS

[ura-soul]

Sure, I am there.

[likwid]

I just sent you direct message in discord.

[likwid]

Payout has been released 3e2da86ee60621ce69a63aa371b402f1a89e61de

[taifkhan]

@likwid will you please check my payout i have not get my payout from last 3 days ?

[likwid]

Been trying to contact you in discord. Would you please join? Thanks a lot in advance.

[likwid]

All pending payouts have been now released.

[dongkong]

I have returned the amount !

[likwid]

Accounted.

[dongkong]

Tnq 🙂

[likwid]

This post earned a total payout of 0.325$ and 0.244$ worth of author reward which was liquified using @likwid. To learn more.

[wehmoen]

[frankvvv]

I think you got another bug, since you keep paying people who got flagged/downvoted, and your payout does not seem to take that into account.

[mozer]

I have typed to discord channed. Still waiting for may response.

[mozer]

Still waiting for my payment

[likwid]

Rewards have been released.

[gravitcaper]

1,5%+1,5%=3%
bot is mistaken

[joythewanderer]

Hey there, I noticed you might have missed a post payout from my threespeak about 10 hours, could you check it for me? Thanks ^^

this one

screenshot of my wallet, seems to be missed.

[honusurf]

Every time I try and use steemd.com to look up my comments and stuff, they never seem to appear anymore, & I get this message below instead:

And then I always get this message too!

I also always run into this message below!

#Time for me to stop voting for them for witness, I think they are corrupt! Must be the global blacklist or something else going on!

I only vote for 2 witnesses, now this new removal will take me down to one! Un-Voting @steemd in the morning if my stuff is still broken! I will add @parriko for witness to replace @steemd! This is getting ridiculous, there is not going to be any way for me to access my account any more if everybody keeps blocking me like this! #onmyown! Seriously this is all @themarkymark’s fault! He’s the one doing all this to me, slandering me calling me the troll. Truth is he’s way worse than me, and he is the real troll! Just go look at the vote history and see how irresponsible he is flagging all my posts for months. Pretty soon I’m just going to concentrate on downvoting everybody who votes for @themarkymark for witness, no excuses! I’m going to start making my list!

Also anybody I see voting for him will be eligible for my downvote too! I don’t know what else to do but try to get rid of him as a witness! So if anybody out there wants to help, just make sure you don’t vote for @themarkymark for witness! That would help me a lot! Here is an example below from today.

Someone send me a list of everyone voting for markymark! I’m going to have to go after them one by one! Heres another:

And another:
I downvote people, but not like this, this is different. Here’s another:
This is why I downvote anyone talking to him! Here’s another one:
He’s the problem troll here, not me! I’m just defending myself! Get rid if him whatever it takes! Somebody do something about that maniac! I’m not giving in. I could go on all night with these screenshots, but I’m tired and have to fo to bed now! Good night!
Ok I guess I got time for one more, he even downvotes my sunset photography:

In the morning I’m just going to add more and just keep posting this over and over every single day as the list grows until somebody helps me out and listens! Goodnight!

Follow @coininstant for more!