Video hosting for a public authority
High reliability, service stability, unlimited scalability
In our practice, we work both directly with customers and subcontracting. In this case, a large system integrator turned to us with the task of building a video portal of the State Duma, which is the main Russian legislative body. The State Duma holds meetings, all these meetings are recorded and stored on digital media with the possibility of subsequent viewing.
The task was to receive and transcode information from cameras into standard formats and resolutions and organize a repository of video content of unlimited volume, which would allow viewing archived records for any period of time without any restrictions.
The portal is called video.duma.gov.ru and as part of the project, we have deployed Media Services for this portal, including video transcoding, storage, distribution across servers, replication, etc.
The volume of content on the portal is constantly growing, while all content should be available. To implement these requirements, the service is designed in such a way that it can be scaled indefinitely, i.e. one can add new machines to the cluster.
At the same time, it is important that the load on the servers is distributed evenly, for which the file distribution algorithm across the servers cannot be "straightforward". New and therefore the most popular video clips should be evenly distributed among all servers, and not "fall" onto a latest "free" server.
In addition, it is necessary to ensure reliable storage of information: when the server or hard drive fails, the availability of content should still be preserved. This is ensured by duplicating content on two or more media, and in the event of a disk or server failure, the content is automatically replicated.
As soon as a new server appears in the pool, it begins to participate in the backup scheme. The internal component of the platform, which we call Household, ensures that the replicas are evenly distributed across the servers. To achieve this, when a new server appears, the replicas are partially transferred to it.
Adding a new server does not require programming. First the operating system and the necessary packages are installed on the server. Then the server registers in the platform configuration, after which other components of the system begin to see it and work with it.
In 2014, when the system was put into operation, this process has not yet been automated, but now the Mediagram platform is able to monitor the availability of space and the load on the servers and deploy the servers in the cluster automatically and as needed. For example, as soon as the load exceeds a certain threshold, a new cloud server is added to the cluster. The same thing happens with monitoring and adding disk space.
Setting up, finalizing and integrating the platform with the front-end took about two months. Five years later the service is stable, no problems requiring our intervention during this time have arisen.