Host DevCon Videos on Swarm & Etherna

Swarm is a decentralized storage and communication system that was initially incubated by the Ethereum Foundation.

Etherna is a transparent video platform running on Swarm.

It was suggested in a comment in another DIP to host DevCon videos on Swarm, this Dip is to make that happen :slight_smile:

If this DIP gets implemented, the DevCon Archive will have a publisher account on Etherna, where it will be able to upload videos and embed them into pages on https://archive.devcon.org/ website, offering a third alternative in addition to Youtube and IPFS.

Videos will be accessible for free by users, and the bandwidth will be charged on the account owner / can be subsidized.

Please note that this is a very high level proposal, and any questions regarding the implementability, the architecture, and the building blocks will be addressed here.

Disclaimer: the author of this DIP (Costanza Gallo) works for the Swarm Association.

15 Likes

Great idea. We need a YouTube alternative for the web3 community. And it makes sense for DevCon to upload videos to a web3 project.

4 Likes

pretty cooool, let’s make video platform more decentralized!

4 Likes

Hi @costgallo

Thanks for the great proposal!
We’d definitely help publicize this. What kind of help do you need from our side?

Best,
Rose

3 Likes

Hi @Rose,
thank you for the answer! Shall I make a formal DIP?
Best,
Costanza

2 Likes

Yes, that’d be great :slight_smile: thanks!

1 Like

Hi @costgallo this is a great idea!
As you mentioned, our video archive already offers to watch from YouTube or IPFS. I think it would be great to have Swarm as a third option.

I’m curious to read more about how Etherna works.
How does that relate to Swarm? How does publishing work? Or what would a potential integration look like?

Happy to discuss or if you want to include any of that in the formal DIP, that would be great.

Thanks!

3 Likes

Hi @wslyvh, I’m Mirko Da Corte, CEO of Etherna.
Thank you for your interest!
We are currently working on Etherna’s Whitepaper, I hope we will be ready to publish shortly. Anyway I will try to do a quick resume of Swarm integration.
Etherna uses Swarm for storage video’s data, like video stream encodings, name, author and thumbnail. They are directly accessible with any Bee node directly on the Swarm network, and our dapp permits to select different gateways for independent use.
Anyway Etherna implements a default Swarm gateway as a service, that make easy to access contents also without personal nodes, and implements our economics to reward the creators’ work. In future this will be built as a decentralized Swarm L2.
We also keeps personal channels on Swarm (video lists), published as Swarm Feeds. This will permit to everyone to subscribe to any channel, and consume videos in a full decentralized way, without any request to us. Personal settings, as channel subscriptions, will be also kept on Swarm.
We planned to open our index databases publishing and keep synced data on Swarm, using FairOs-dfs. This will permit to everyone to verify, analyze and consume data directly from decentralized network.
We have also other R&D projects on Swarm, for example to decentralize indexes, but these will come later.
Hope to have replied, I’m available to answer any question :slightly_smiling_face:

5 Likes

Hello @wslyvh,

Thank you for your reply! Mirko from Etherna answered the technical questions, and here’s the formal DIP..

How do you want to proceed?

Best,

Costanza

Thanks for joining here and sharing more info! I’d be interested to read the whitepaper or other documentation, once ready.

For now, I’d add that we’d love to keep our video archive up at archive.devcon.org/
I think Swarm would be another option (next to Youtube and IPFS) to watch or download the video.
E.g. https://archive.devcon.org/archive/watch/4/ethereum-foundation-values/

This proposal seems to suggest that Etherna will be the UI/player to watch the videos. Although, once the data is available on Swarm I guess we can use it across different websites/experience?

Some other thoughts/questions based on the formal DIP

  1. The timelines seems quite challenging (we’re already running short on time). I think it would be best to minimize the amount of integrations to avoid unneeded complexity and risk for delivering on time.

  2. Can you explain a bit more on how postage stamps work? Should we expect to pay for any services/storage/pinning? Or should we run our own Bee node to pin our own content instead?

  3. What are your thoughts on distribution/CDN? From our experience, decentralized storage just isn’t ready yet to serve (large) video files.

Also see: “But Chris, video is too heavy”

DIP-19 also suggest that there’s a 500MB limit for video files? This might be an issues as we have files ranging from a few 100 MB’s up to 2GB.

Because of this, we’ll likely keep a centralized service as the main source to watch videos and use decentralized storage solutions to download mp4 files in different encodings from. Curious to hear your thoughts…

1 Like

WP is currently under internal review, and waiting for pagination. I can notify here when ready.

We absolutely can embed videos on external pages, so also on archive.devcon.org without any problem. You can see for example here https://info.etherna.io/, where we already embedded our presentation video on the website (scroll a bit down). Our idea was exactly this, to show swarm embedded video aside of youtube and ipfs on devcon archive site.

Yes, Etherna is a platform built on top of Swarm (as described before), and offer a simple access gateway. Anyway any video data is on Swarm, and you can use Etherna’s protocol without our implementation. Application running on https://etherna.io/ is a dapp, full client side, and configurable for use with alternative gateway endpoints. In future also the index will be switchable. Like any dapp, any other application compatible with Etherna’s protocol can handle its contents, directly on Swarm.

Answering to pointed questions:

  1. we already have developed almost all required features for enable this use case, and we are really near to be ready to a public release of the platform. We have a working gateway, load balanced, with global CDN enabled. What is still missing is pinning implementation, but it’s not a difficult task, and is our first priority after the release. “Update Postage Stamps management” is already in progress, and near to be ready. I think it will come before September. 500MB limit is more a first scalability precaution rather than a real limit. We already are investigating optimizations to be implemented on our gateway that can remove this limit.

  2. I can answer this from different point of views, from Swarm and from Etherna. For Swarm, Postage Stamps are paid at front of any data upload, and they are needed as an antispam cost and must be periodically renewed. Moreover will come an incentive system built on them. Pinning instead is only a local space cost, because it doesn’t have effects on the net. Content could be republished on the net from pinning node, but the Global Pinning feature that should have automatized this must be refactored and it is not currently available. From the point of view of Etherna: postage stamps are managed by us on our gateway. They will have a cost for the user when created and, we decided to offer the first year of stamps for any published videos on our index. Anyway, currently we are full offering them, until we implement platform payments with next milestone release. Also, pinning will have a cost if managed by us, but only from next milestone. We will integrate swarm native storage incentives when available. It’s not necessary that you run your own node to maintain devcon videos.

  3. we already have it. We are thinking on how develop a full decentralized version of Etherna Swarm nodes (implementing our economics, on L2), anyway we still are not there. Our gateway is currently centralized, and leveraging on centralization advantages for serving data. Under the architecture we have Swarm nodes, but in front of them we have load balancer, cdn and will arrive specific routing system for reach contents directly on the right internal pinning node. This solves distribution problems, also for heavy contents.

Limit of 500MB as I said is a precaution limit, but we will remove it. If it is a necessity, we can also give more priority.

Etherna is this, a service/protocol oriented to web3, but that can also offer advantages from web2 when necessary for best UX, until new techs will be ready.

2 Likes

Thanks again for the info and extensive answers.

What kind of help do you need from our side? Or what would a potential integration look like?

I can imagine it might be easiest to use devcon.archive.org as the main source/base.
devcon-website/src/content/archive/videos at archive · efdevcon/devcon-website · GitHub

This is all on Github and our main source of truth. This also covers videos from previous events and we’ll continue to push all new/latest videos pushed there as well. It also means we don’t end up creating different/multiple integrations into our pipeline.

Another benefit might be that we’re less strict RE timelines. We can do that after Devcon. Although it would be great of course to have a Swarm option asap.

Also happy to sync over a call some time if that helps…

@wslyvh Our first idea was that some one, from your side, will upload videos on our platform by hand with a Devcon account.

Anyway we expose APIs, so this could be automatized in some way, but a custom solution needs to be implemented. Better if integrated with your current workflow.
Eventually we can cooperate on build of this, to keep synced videos from your github as a source.

Moreover, we still doesn’t offer video encoding, so every video has to be encoded before of upload, but I think for IPFS is the same problem, and we could use the same encoded source.

Absolutely, we can have a call for organize next steps. You can reach me at mirko(at)etherna.io.

Contextually, I also announce that we have published the whitepaper on our website https://info.etherna.io/ :slight_smile:
We will make a public announcement at the same time as registration opens on the platform on the 19th of this month.

1 Like

I’m not sure about the exact amount, but Devcon 6 will be our biggest event yet with around 600-800 of recorded sessions. An additional +700 videos will be added from previous editions. I would really prefer to automate or integrate this into our workflow instead of manually uploading.

We’re still working out the details of our live-streaming/AV pipeline, but we’re definitely pushing to our repo as our main source. The suggestion for an integration/sync with our Github repo might indeed be a good solution. I think we’ll likely facilitate different encodings. If not, using a single source (as we have right now) would be fine.

Will connect over email to schedule a quick call

1 Like