Scaling live trivia game to half million concurrent users

  • Sudden incoming traffic at the start of the game
  • Showing accurate, live game stats
  • Handling and processing huge number of concurrent requests

What is live Trivia?

Display question and Display statistics screens

How does live Trivia work?

  • Host asks a question and user is presented with question and multiple options(typically three) to choose right answer from.
  • User gets 10 seconds to choose right answer from presented options.
  • Post 10 seconds, the user is eliminated or moves to the next question by choosing the correct answer.
  • For every question/answer, user can see stats related to the question immediately on the screen post 10 seconds. Stats include % of submitted answers to each of the presented options + correct answer
  • This continues till all the questions are finished. Typically 10 questions per show
  • Prize money is distributed among the users who answered all 10 questions correctly.

Components

  • Display Question. User is shown a question + answer options
  • Display Statistics. Stats to be shown for each answer

Game flow

High level architecture

  1. Game operator changes the game state(display question/display statistics). Let’s go with game state “display question1”.
  2. Gaming service accepts requests from game operator and updates the game state to Redis. In this case, game state in Redis is changed to “display question1”
  3. Client polls polling service continuously to check current state. If there is state change then client takes action based on new state.
  4. Polling service returns new state “display question1” and client displays question on the screen with options to select right answer. Polling service is built on Kong (which is Lua application over nginx). Nginx support Redis out of box. When ever there is state change, a Redis pub-sub event is triggered. We wrote simple Lua cache loader script which reads event and updates global cache of game state for all the workers to consume. Each worker has cache is of size 8KB and Global cache of all workers is of size 64MB.
  5. Clients submits answer to the question by selecting one of the options and answer submission request is sent to gaming service
  6. Gaming service receives answer submission request and adds event to Kafka topic
  7. Kafka brokers are setup to receive answer submission events from all gaming service hosts.
  8. Flink cluster consumes event batch from Kafka for processing and updates multiple sources. The current system at Swoo doesn’t have Flink setup as we have seen max 100K concurrent users. Kafka Streams serves well if your live concurrent users are below 100K.
  9. SycllaDB receives user answer submission data and persists it.
  10. Flink also updates Redis with answer count for each option after processing events.

Infra:

  • Kong (Load balancer): 5 instances of c5.xlarge, 4 cores, 8gb ram.
  • Polling Service: 5 instances of c4.xlarge, 4 cores, 8gb RAM
  • Gaming Service: 14 instances of c5.large, 2 cores, 4gb RAM
  • Kafka: 4 instances of t2.medium, 2 cores, 4gb RAM
  • Flink: 4 instances of m4.xlarge, 4 cores, 16gb RAM
  • Redis: 2 cache.m3.2xlarge machines(primary and secondary), 8 cores, 33.8 gb RAM
  • SycllaDB: 3 instancies of i2.xlarge, 4 cores, 30.5gb RAM

--

--

--

https://www.linkedin.com/in/khaleel-pasha/

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Connecting Azure AD B2C to Auth0 via the B2C custom identity provider

How to write null safe code with Kotlin?

Objects are for interactions, functions are for data

Get Free giveaway Holikme Mop Broom Holder Wall Mount Metal Pantry Organization and Storage Garden…

broom holder

Problem while building Docker images on Windows 10

Force MFA for AWS IAM users

KEY AREAS FOR SUSTAINABLE PRODUCT DEVELOPMENT AND SUPPORT LIFECYCLE

Notes, That will get you Terraform Certification!!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Khaleel Pasha

Khaleel Pasha

https://www.linkedin.com/in/khaleel-pasha/

More from Medium

How to Design A Real-time CryptoCurrency Price Alert Backend

Concurrency in Go ( Beginners )

Talk about Go: Network programming about grouping and unpacking in TCP flow

Mongodb query sorting