Netflix and Spinnaker

The Development of an Efficient CD setup

Introduction

The fact that an engineer, perhaps, pushed out an errant deployment or made a change to a property that they should not have, and the entire system went down, does not want Netflix to interrupt its show streaming. This was the verdict of Dianne Marsh, director of engineering tools for Netflix.

For a complex architecture, a completely open-source, multi-cloud, continuous distribution framework that Netflix built internally to help production teams release product updates with the assurance that nothing will fail, Spinnaker was key to ensuring constant uptime in this regard. Spinnaker has two key features: cluster management and deployment management

Another Netflix open-source tool called Asgard, a cloud delivery mechanism built to optimize the availability of Netflix services to Amazon Web Services (AWS), was succeeded by Spinnaker.

Problems with Asgard

Asgard appreciated the community’s vast scale acceptance, but it also met specific challenges. Asgard was built to assist only AWS, but users wanted to use it in their own environments. Also, the developers didn’t need the authorization to be able to do installations as an internal Netflix tool, but potential investors needed such capabilities. To use it for their own requirements, people had to fork Asgard. Netflix saw a problem as people forked it out.

It contributed to a breakdown of the source code of Asgard so getting the technology back was not very easy for them. Netflix lost the ingenuity that those other businesses did on those projects. So they needed to think about how they could make their platform an extensible forum that didn’t have to be forked. Instead of putting anything out there, they should collaborate with the public to accept the whole situation they were facing.

Origination of Spinnaker

Netflix began to look at Asgard’s successor not only as a distribution mechanism but also as a continuous network for execution. So, the work on a prototype that later became known as Spinnaker began. Another project management initiative inside Netflix, headed by Sangeeta Narayanan, has been the head of Netflix’s edge developer experience. In contrast, the core team at Netflix was focusing on Spinnaker. Since they did not wait for the centralized team to complete the product they were working on, they developed features and capabilities for their own needs.

Marsh, the Engineering Tools Team director, clarified that there were very urgent needs for the Edge Center API team; they needed to execute rapid installations in a secure, stable, and repeatable manner. They decided to use the project because other departments saw the work carried out by the Edge Center API team. Since it was not included in the unified team, it placed undue pressure on the Edge team, and the tool was not intended to help Netflix overall.

Open Sourcing Spinnaker, 2014

In 2014, Netflix began accessing Spinnaker independently, and the next year it was open-sourced. Netflix assured that Spinnaker solved much of the Asgard culture’s challenges based on the company’s familiarity with Asgard. In designing the multi-cloud approach and creating the plug-gable infrastructure, Netflix collaborated with several collaborators.

Google, Microsoft, and several other firms collaborated alongside the production teams. Consequently, in their realms, Google, Microsoft, Amazon, Pivotal, and even Oracle could use Spinnaker. There was no fork required. The loop has been completed. These businesses took advantage of Netflix’s creativity and, in addition, Netflix benefited from the invention that the communities brought to it.

The verdict of Dianne Marsh, Director of Engineering Tools for Netflix

To put it in perspective, Team Netflix wanted railings, not barriers, to be provided. Mash believes that the teams didn’t want to discourage the masses from doing anything they want on the platform; however, they tried to explain why they feel it would be a wrong decision.

Conclusion

On the whole account, Spinnaker has quite a lot of influence from Netflix. In this regard, the most fundamental aspect to be kept in mind by the engineers was that they had to be perfectly aligned but loosely bound at the same time throughout the design process. Hence the sole responsibility of mutual communication between the teams lied totally on the individual teams themselves. To endure confidence in all those teams, Spinnaker played a vital role by enforcing their decision and checking their mutual compatibility, and ensuring that those deployments and modifications are the desirable ones.

Technology and culture have a substantial impact on each other in an organization, and so does the tools used to develop and deploy that technology. The tools used in any development program essentially portray the culture and moral values of the organization. Otherwise, the situation can get perplexed. If these instruments do not represent the organizational culture, then the production team may be questioning the culture, vision, and mission of the organization with its methods. These are two very distinct approaches, and they need to be addressed very differently.