https://github.com/joystream/atlas/issues/1577
The purpose of this document is to attempt to summarise the current status of work on Atlas, in a broad sense, for the benefit of all recent incoming and near term incoming team members, and in particular a new product manager to be onboarded very soon.
The Atlas product size and team has grown considerably. The product as matured significantly feature wise, and is now being used by a small but passionate community of project contributors, who primarily use it as an information sharing and community building platform for the Joystream DAO. The product has, until now, focused quite heavily on reaching a certain baseline level of familiar video platform functionality, which has served as some level of insulation from the errors that inevitable come from not being more closely connected to the user base we are building for, however, going forward, we expect to build more and more totally new and unfamiliar product features. These factors have raised the need to make sure
Atlas is the web based consumer product exposing the content consumption, publishing, monetisation experience made possible by the Joystream system. A production version of this application is available here
Reflecting the version of our system as reflected by the Sumer network, described here
https://www.joystream.org/sumer
The product is built entirely in the open, so all of our assets are public and transparent, and all assets exists under permissive licenses.
not yet on masterWhen the name Atlas is used in this document without additional qualification, it refers to the core digital product artefacts, including source code and design assets, which are open source. It does not refer to a particular instantiation, such as the version running on Jsgenesis infrastructure and distributed through the domain https://play.joystream.org/.
Atlas is primarily meant to show current and prospective community members what Joystream can enable as a platform. It, and in particular the deployment of it on the joystream.org domain, is not intended to be a value capturing asset for Jsgenesis the company, unlike the role of youtube.com or the YouTube mobile app for Alphabet. This is a very important, and perhaps unfamiliar, distinction between products in the Joystream project, and traditional startups.
This purpose is served in two distinct ways at two distinct time scales.
Now, and in the immediate future, Atlas shows community members and creators what the system can enable in terms of familiar and unique Web3 features. The Jsgenesis instantiation also shows the state of the current creator community and content selection.
Over the somewhat longer term (mid 2022+), it supposed to serve as a base template for others to build their own product offerings on top of Joystream. This innovation is uniquely unlocked by the open permissionless infrastructure of a blockchain system, and a forthcoming subsystem called the gateway system which provides a built in business-model for developers and entrepreneurs to build their offerings on top of the Joystream system. The innovation will initially likely be in terms of
but then our bet is that people will start introducing deeply differentiated video product experiences that starts to work very differently to the initial baseline Atlas.
The Atlas web app itself has a relatively conventional web stack for purpose, perhaps with the exception of relying on an external extension for key management, as is customary for web applications in Web3. The much more important aspect worth understanding is all the different infrastructure pieces and roles interact to make Atlas a working product. The architecture diagram below gives a succinct summary of this, and subsequent sections dive into some further detail on each of the pieces outlined in the diagram.
Maintainers
The Joystream blockchain is the technical centrepiece, serving as both the data, social and economic infrastructure layer for the overall system. It is a standalone Proof-of-Stake (PoS) blockchain (hence not a set of smart contracts, as is often the case for other systems). The native token, or currency, of the system is called JOY, is often denoted by the currency symbol $JOY. A distinguishing feature of the system is that it is upgradable, meaning that it has a built in capability for the community to upgrade the rules that govern the system. It has many interconnected subsystems, but the most important parts w.r.t. Atlas are
The Joystream blockchain is written on top of a blockchain framework called Substrate, which is a Rust based SDK for developer to build their own blockchains. This means we do not need to deal with very low level concepts, such as peer-to-peer networking, the consensus algorithm or state database. We only need to define the business logic of our specific use case, and this is done in Rust.
Maintainers
When applications need to read the state of the blockchain in some way, for example to see all video channels that exist, this mostly does not work well if you attempt to do so my directly connecting to a validator node. Validator nodes, who are involved in the business of validating the integrity of the system as it evolves, do not have any search indexes that allow for efficient queries of the current state of the system. For this reason, the query node is an intermediate query node layer which maintains various explicit query state that can be accessed through a GraphQL API. Our particular query node is built on top of a framework we built, called Hydra, described here
https://www.joystream.org/hydra
A critical issue with the query node infrastructure is that any changes to the blockchain require that infrastructure developers have to define the APIs, through schemas called input schemas, and processing code that takes blockchain ledger events and transactions, and updates the underlying query state of the node, these are called mappings. Writing such input schemas requires understanding both the underlying business logic of the part of the blockchain you want to expose to applications, and also what the requirements are of the particular application(s) you have in mind. This capability currently does not exist within the Atlas team itself, and so far they have relied on infrastructure developers to support them.
Currently, there is only a single query node deployed in every test or production deployment for Joystream networks, and it is operated by Jsgenesis in those cases. The likely future scenario is that running a query node will get bundled into the activities of a Gateway, as what particular schemas and queries are relevant to a given product instance may not apply to another. An alternative could be to make it a paid DAO role, or to try to rely on a third party middleware protocol like Subsquid eventually will become.
Maintainers
Orion is a service for maintaining viewing statistics for content in Joystream, it is still in its very early stages, and likely will become the future Gateway node. Its also highly likely that it stops becoming a primary data store for the this viewing data, as this really needs to live in a shared public data layer, rather than a trusted server. Read more about this in The Missing Shared Data Layer.
Maintainers
Storage nodes are responsible for long term archiving of user assets, such as videos, avatars, etc. The reference implementation for the Joystream storage node protocol is is called Colossus. Importantly, these assets are not stored on-chain, that is by validating nodes. A given asset is stored by a small subset of storage nodes, for redundancy, and each storage node only stores a small share of the overall set of assets. They are operate by community members through on-chain roles in the storage working groups, and are policed and directed by on-chain authorities, i.e. the lead and the council. These nodes also accept initial uploads from users when they publish these assets.
Maintainers
Bandwidth nodes are responsible for distributing user assets to end-users at scale, similar to how a CDN works. The reference implementation for the Joystream bandwidth node protocol is called Argus. Bandwidth nodes replicate data as needed from storage nodes, and maintain a local cache based on local request statistics. Currently there is no authentication layer, hence anyone can download anything, without payment or credentials. They are also operated by community members through on-chain roles in the distributor working groups.
Note: These nodes are actually only going into production in Giza release, which is imminently going to be release.
Maintainers
A key design requirement of Atlas was that the user had full custody of all digital assets, meaning that no third-party (even the future Gateway) should have access to any of the keys that control the $JOY, NFTs, Social Tokens, channels or memberships of the user. This is not the only way to build apps on top of Joystream, someone could - and likely will, build a fully custodied and friction minimised version.
With this constraint in mind, the only robust way to allow users to manage keys, while also using the app in the browser, is to rely on a browser extension for key management. For the Ethereum ecosystem, Metamask plays this role. Since Joystream is built on Substrate, we use the extension Polkadotjs extension, which is the equivalent for this ecosystem. It is not a full wallet, for example it does not have the ability to speak to a validator node, so ti cannot send transactions or display balances, it only allows for holding keys and signing transactions.
The main product features that currently work in production are
The core tech infrastructure that powers the storage of all media assets, such as images and video media, as well as distribution of that media to end-users, happen through systems that are operated purely by community members who are coordinated by the testnet blockchain and some incentive schemes from Jsgenesis. Those operators run custom written node software that performs the required tasks, and they select what infrastructure to run those nodes on (self-hosted, AWS, Google, their house!). This means that this infrastructure regularly experiences various performance issues, both because the technology is immature, and also because it is today operated by enthusiasts with limited time, experience and incentives. Right now are we in the process of releasing a new network, the Giza network, which will be the last major overhaul of the tech side of this infrastructure before the mainnet, described here
https://www.joystream.org/giza
Gateways are a future infrastructure service, operated by what will be called gateway operators. They are responsible for the last mile between the blockchain economics and infrastructure and a mainstream consumer audience. This is not a part of the system which is currently fully designed or built. Briefly put, the role of a gateway is to achieve the following goals
Currently, the Orion node, which holds viewership statistics and content featuring policy information, is expected to become the precursor for the gateway node. A good introduction to some of these topics can be found in this video
https://play.joystream.org/video/1266
An open question which remains is what the open shared data layer for traffic information should look like. Currently, a very naive approach is taken to this with the Orion node, but it has obvious problems around verifiability, permissioning, fault tolerance and availability.
Here we attempt to summarise features which have been considered, or are, possibly within scope to be built by Jsgenesis, but without any prioritisation or timeline.
It is yet to be determined what is an absolute must feature set for mainnet launch, which is expected Spring 2022.
These features are considered baseline, in the sense that they are familiar from existing offerings, such as YouTube.
These are features uniquely unlocked by Web3 infrastructure.