"Riot Messaging Service - Message Players Like a Pro" by Michal Ptaszek

League of legend:

• 100+ millions users
• 7.5+ millions concurrent connections

Using a 2 tiers architecture based on RMS (Riot Messaging Service) written in Erlang.

Warning with DDoS introducted by load-balancer when players want to connect at the same time.

Visibility is a vital key. You should view a lot of things and monitor everything. The detailed metrics are really cool for developers but customers don't give a shit about it. A good dashboard is only green and red in case of issues.

Dynamic configuration can save you. All the configuration is stored in a git repository. RMS pull this server every minutes and notify when a change appears. All change are applying in 30s around the world.

Self-service capabilities with a real time dashboard to all data routed all traffic on the platform. If something goes wrong, everything is notified in this board.

It is important to know your capacity by testing your product. Be prepared to massive failure and find a solution to recover the service in any case. exemple: 2x more connections is easy to manage if the platform was tested to 10x more connections.

Questions

• Data is managed by mnesia in distributed mode.

• Load-balancer is a classic ELB on AWS for websocket reasons.

• each edge server is aware of all connections established to the server and notify the routing tiers when a connection appears or disappears. The routing tiers gets maintain also a table in case of failure, it can clear the mess.

• the protocol used between edge/routing is pure http rest. the protocol for the clusters is distributed erlang.

• the cost is calculated with CPU utilizations and connection counts metrics. A daemon on cloudwatch notify all connections. Don't want to save too many dollars.

• Every single containers have its own IP address, run the containers in the host network.

Resources

• https://www.youtube.com/watch?v=DnjXM-Z9uho