The cluster storage is another temporary solution, but yes we plan to host your videos for much longer than before, but we can't promise that we will store it forever.
Our end goal for storage is to have every file available on IPFS / WebTorrent / DAT for a pretty long amount of time on our servers (which mostly will depend on our budget for infrastructure, something that is hard to foresee today), but we will need our users to use either a desktop or a mobile application running at least one of those 3 techs to keep old and inactive videos online.
Another solution would be to incentivize Steem witnesses to run IPFS nodes and pay them proportionally to their used storage, but I don't see how that'd work.
Hi dude. @exnihilo.witness is ready to run a IPFS nodes if needed.
By the way, good job Auvergne, as usual.
Why not use a system similar to STORJ for distributed hosting of video files? This would distribute the load while also solving any redundancy issues. Uploaders could contract with storage nodes on chain for the space and bandwidth usage. Give uploaders control over the level of redundancy as well, which would reflect in the cost of the contract. You could have both short term contracts as well as long term archive type contracts or whatever you want to call them. I think it would be a bad idea to limit something like that to steem witnesses though... it would be a bottleneck for one, but it also wouldn't be very pro-decentralization if you catch my meaning. I've got a large file server on a fiber line just sitting idle waiting for the right decentralized storage network to come along. Storage is cheap... if you create the right inventive people will support it.
Great ideas.
Hey,
talking about running an IPFS node: currently it looks like my local DHT hardly contains any hash from the DTube peer. Is it possible that you could make the peer ID of the D.Tube Peer (or several if you have it) public? So far I have not found a way to get them and so it is often the case that I could not find a single peer via
ipfs dht findprovshere in Germany and therefore the operation of an own node has hardly any sense. The peer ID can be entered as bootstrap peer, so this would certainly be a good step to get more people to run a local peer.Thxalot,
JanSe
Did you ever get an answer to this?
I'm trying to setup a small ipfs node to "auto pin" videos and related assets that I upload to d.tube.
Hi @magali,
currently, D.Tube no longer uses its own IPFS node (as you can read here, they now have their own solution). Back then, they all re-initialized the nodes regularly (so there is no unique peer ID). If you now want to pin videos again (but they are no longer streamed via D.Tube), then download the video first using
wgetand then add it normally with your node.Big f_cking Edit: Since when do they use IPFS again?!! 😮 Can they make up their minds? Seems to be running IPFS again now, but I think the problem will still exist, but I will stay on it.
Thxalot,
JanSe
I know you guys have been banging on this problem for a while now, so I'm pretty sure you considered decentralized storage solutions with their own internal currency solution for hosting like Sia or Filecoin.
It would be interesting to hear your reasons for going with Ceph over a more decentralized solution. Are the blockchain-based storage solutions just not up to the amount of traffic that DTube can/will produce or…?
This bit in particular is why I inquire about coin-supported storage solutions, since they are effectively doing exactly this, albeit on a different blockchain but with people who are specifically engaged and interested in providing distributed storage. It would be really cool if some steem witnesses wanted to provide some extra support for storage in general and used their server capabilities to help do that, but as I understand it they're pretty strapped just providing enough computational and storage power for the steem blockchain as it is. That's why we have multiple kinds of provider for people who are interested in providing different services for pay.
Because decentralized storage doesn't mean as a IPFS node hoster you don't want to use an efficient way to store IPFS content.
Except that doesn't answer the question at all.
One of the things that the DTube folks have set on a regular basis is that they want to be censorship and access control minimalized, using IPFS as a deliberate choice for moving toward that goal. However, obviously, IPFS isn't up to being that performant in this case, or they wouldn't need to go to an enterprise cluster storage solution.
Since they have stated their desires and that cluster storage is a temporary solution, I assumed that they had looked at other blockchain-based solutions which literally involve incentivizing other people to provide decentralized storage and compared that to the cluster storage option.
Talking about that sort of thing and the trade-offs involved is what I was interested in hearing about, from people who had actually been involved in being elbows deep in solving the problem.
If you need a secondary storage solution for hosting IPFS content, you might as well use that storage solution instead of IPFS if it solves the problem. If it doesn't solve the problem, and there are other systems which purport to solve the problem, and you don't use them – the reasoning around that should be interesting.
Sia and Filecoin (and even Storj) purport to literally solve the problem that DTube has said that they have and I was curious about their analysis of why going with a more traditional cluster storage solution won out. Is it truly a temporary measure until they finish assessing more decentralized solutions? Is it considered a midpoint solution, acting as a storage buffer between front end users and more decentralized, less attackable storage architectures on the backend? This is good stuff to know for other people who might be interested in building digital applications which interact with the steem blockchain and require their own storage capacity.
It'd be nice to know what someone who knows something about it has to say.
You people went waaaay to deep, i just want the answer about Decentralised solution vs Ceph too @heimindanger
Brings into question whether or not they truly want to decentralize it. Think about it... how do you decentralize information while retaining centralized like control in order to abide by DMCA's? Or blanket ban Alex Jones? Now that doesn't mean this content can't be distributed. If I understand correctly, anyone can already download any video and re-host it themselves out of the kindness of their heart. There's no monetary incentive for this however, and it has to be done manually. So it would appear that all that's really needed would be a platform to monetize and automate this while retaining the ability to remove content to satisfy DMCA's, or whatever else someone doesn't like. You wouldn't actually need to incorporate any of the existing decentralized applications, and there wouldn't really be a need for encryption since the files are publicly available anyway. In the end though, none of these types of platforms will truly be decentralized because well... people just simply aren't ready to accept the consequences of decentralization yet. You know, I find it ironic that we've practically banned freedom of speech because of our intolerance, yet we continue to do things like pay taxes and suck down fluoride without question. Really makes you wonder who's actually in control.
I'm not quite enough obsessed with conspiracies to ask that question. Especially since given a fair amount of experience with large projects and the necessity of making certain concessions in order to get a project working at all, much less how you want it to work, I've been there.
There are certainly ways to have decentralized file storage and be compatible with the DMCA and have some degree of control of content visibility – and the secret is just that, visibility. Whether things are visible to a user is an entirely different thing to whether or not it's being stored in a distributed manner. In fact, these are orthogonal things; they have nothing to do with one another.
I think people are perfectly ready to deal with the results of decentralization and in fact are eager for the effects of decentralization, and in fact really want the results of decentralization, especially when it comes to distribution of file storage on the backend of a database, which is what we're talking about.
But, like I said, I'm not about to take up conspiracy when a perfectly reasonable, logical, and sensible explanation which doesn't require hidden motives is clearly available.
You can continue to see secret lizards under every face, but that doesn't change the mechanics of the world.
I'm not claiming conspiracy. Or are you just reacting to my mention of Alex Jones (who I do not support by the way) which I was simply using as a recent and well known example of censorship which in itself might very well be a conspiracy, but that's irrelevant to this conversation.
There's a big difference between decentralized and distributed. Decentralized meaning there is no central authority or control, and distributed meaning the data itself is not centralized however control over it still is. You cannot truly decentralize while retaining control to censor without the consent of the majority. So the real question is how to distribute the data in a way that takes advantage of some aspects of decentralization while retaining the control to censor, because let's face it... even though many of us are willing to accept the consequences of decentralization in the interest of freedom, the vast majority are not.
I may be wrong, but it sounds to me like your focusing mainly on the benefits of decentralization from a technical standpoint and what that means for computing and I completely agree with that viewpoint. On the other hand, most people outside of the tech world associate "decentralization" solely with eliminating central authority which, aside from greed, is the driving force behind decentralized currencies in the first place.
What I mean by saying people are not ready for the consequences of true decentralization is that people have to be tolerant of material they don't like or agree with since the very nature of decentralization means giving up the ability to censor all the ugliness that comes with it. Until people wake up and accept the personal responsibility that's required for our freedom we will never have true decentralization (or freedom for that matter). People not wanting that responsibility is what got us into the mess were in in the first place.
I support Alex Jones. Why not support him or Tommy Robinson or Veritas or Mark Dice? Why not?
Having witnesses to host IPFS servers is brilliant!
So far I have not seen a witness offering that (I have not been too much involved in steem it lately, though)
Other important thing is that DTube should inform creators on how to host their own content in their own IPFS nodes. That way, they have their own backup(s) and ensure their content is more available. Perhaps writing a guide that helps them set up a IPFS node and pin their content.