Twitch Engineering: An Introduction and Overview – Twitch Blog

Twitch Engineering: An Introduction and Overview – Twitch Blog

Web APIs and Data

Besides our real-time services (video and chat) we also have a substantial number of services including, but not limited to:

• Web APIs that allow users to manage and customize their profiles and subscriptions
• Search and Discovery services that help you to find the streams that you want.
• Revenue systems — these are systems that allow us to manage advertising and subscriptions, and make sure that our partners get their income.
• Administrative tools that help our support staff resolve your problems.

This is all built on a mix of Rails, Go, and various different open-source applications that are used for routing, caching, and data storage. We have also been doing a lot of vertical and horizontal partitioning of our data and data APIs in order to make our systems scale better — both at a technical and organizational level.

Web and Client Applications

All of these services are lovely, but useless without easy to use client applications that create a good user experience. We have a multitude of different client applications that make it easy to access Twitch.

• Our desktop web application, which started off as a vanilla Rails application, but has evolved into an Ember app.
• Native apps for the major mobile and tablet platforms (iOS and Android).
• Various console applications, including XBox One, XBox 360 and Playstation 4.

Data Science Infrastructure

Science Engineering builds data collection systems and analysis tools to study how our users interact with our products. These systems include:

• The data pipeline, which collects, cleans, and loads over a billion events a day into our data warehouse
• Streaming aggregators to summarize metrics in near real-time for broadcasters.

Tools and Operational Infrastructure

The above services directly support the application and product. However, the operational infrastructure that we need to manage these services is also extremely important. We have and maintain tools for:

• Testing all of these services by manual QA and automated testing — we currently use Jenkins for automated build and test.
• Tools for rapid deployment and rollback of changes — we have a couple of home-grown tools here.
• Monitoring and alerting systems that let us know when things are going wrong, and help us to diagnose and fix them. Currently a mix of tried and tested tools like ganglia, graphite, and Nagios.
• Distributed trace. This is something new that we've been working on, to help us understand and debug our systems as we move further and further into the realm of distributed systems and microservices.

Read full article from Twitch Engineering: An Introduction and Overview – Twitch Blog