CASE STUDY: Moving at the Speed of Cryptocurrency with Infrastructure as Code

Blue Sentry works with many of the world’s largest financial institutions.  LukkaTech is one of those customers and provides the world with institutional-grade solutions that deliver transparency, clarity, and insights into financial data.  Included services provide clients with cryptocurrency accounting tools in multiple sectors, from crypto-native businesses to traditional financial institutions. As a trusted partner, Blue Sentry Cloud utilizes a team of engineers to design, build and deploy complex and challenging solutions that allow them to scale, modernize and gain a competitive edge in a fast-paced and dynamic industry.

THE CHALLENGE

As part of the complex and dynamic infrastructure stack, LukkaTech runs InfluxDB clusters on AWS. While setting up an EC2 Cluster to run InfluxDB can be very easy, it can become very complex to manage pretty quickly since once you have the cluster up you will start configuring data ingestion, monitoring, replication, and logging. To have all of those items in alignment you rely on several configuration files and that’s where Lukka had the biggest challenge. Since an enterprise InfluxDB Cluster counts on at least 6 EC2 nodes (3 data and 3 meta nodes), the configuration files were all over the place. Unfortunately, this left them in a position where it was taking several days to deploy an InfluxDB Cluster.  Additionally, as business needs dictate, there would be a need for at least one cluster in each environment (Dev, QA, Production, and DR).

THE SOLUTION

Working with internal engineers and management, Blue Sentry was able to devise a plan to ease the administrative overhead involved with deploying new InfluxDB clusters. In AWS, you can use Auto Scaling Groups to scale in and out your servers based on a template. You can use UserData in your templates to automate the bootstrap of your servers such as installing and configuring packages. You can also build automation using Lambda Functions with custom triggers like AutoScaling Lifecycle Hooks, have a Load Balancer in front of your servers to balance the traffic as well as have DNS management in Route53. Integrating all of these AWS services and tooling into a proven infrastructure as code template allowed them to move from a model that was burdened by a lot of manual work and configuration to a model where resources can be spun up very quickly in a repeatable fashion.  This allowed cluster deployments to go from taking several days to deploy by hand to taking around an hour to launch a brand new cluster in any environment.

THE PAYOFF 

Blue Sentry created a Terraform module that uses the AWS provider to deploy resources. The module deploys a CloudFormation Stack responsible for managing the EC2 Resources such as AutoScaling Groups, Launch Configurations, EC2 Instances, and Security Groups. The decision was made to use CloudFormation for this part of the automation in order to use the CFN Bootstrap Tools (cfn-init and cfn-hup).  This allows the Lukka the ability to easily manage all the configuration and Influx license files across the servers. This AWS tooling also provides the ability to ensure specific services are running and also the ability to restart the services whenever a specified file is modified. The module also creates an EFS to store Influx Backups that are taken through an Influx native tool in one of the meta nodes. The DNS management for the servers is done through Route53 records which are all automated through a Lambda Function that is triggered whenever the AutoScaling group scales in or out. It also counts on SSM Parameter Store to store some important information such as Influx Licenses and updating the license across the running servers is now as easy as updating an SSM Parameter Store.  This flexible, scalable and repeatable architecture led the Cloud Director to state: “I am extremely impressed with the level of expertise and dedication demonstrated by the tech team at BlueSentry. Their deep knowledge of the latest advancements in cloud platform technology, coupled with their unwavering commitment to quality, make them an invaluable partner in helping us build mission-critical grade crypto trading applications that are trusted by global institutional companies. Their ability to deliver robust and scalable solutions on time and within budget is truly remarkable, and I would highly recommend them to any company looking to stay ahead of the curve in this rapidly evolving industry.”