A social media company, founded in 2008, is in the business of extracting sentiment from what they call “galactic” amounts of unstructured social content, and transforming it into highly actionable indicators in diverse fields such as finance, brand management and politics.

The Challenge

Their clients rely on the company’s ability to ingest and analyze large amounts of data, derived from Twitter feeds and other social media channels, at scale — literally, the ability to process anywhere from a few hundred social media mentions to tens of millions, in an instant. When Blue Sentry was first engaged, they were using a single instance of MongoDB — greatly limiting its ability to scale and analyze data with the dynamism that is expected.

The Solution

Blue Sentry created and deployed a reproducible sharded MongoDB cluster, using infrastructure provisioning automation, and, to the extent possible, configuration management tools. Based upon client preferences, Blue Sentry developed a config management engine powered by AWS CloudFormation and a third-party tool, SaltStack. Once deployed, the cluster would be load tested using metrics gathered from Datadog as a means of providing guidance on the possible resize (scale-up/ scale-down) needed for cluster hardware. The resulting cluster (see diagram at right) was designed to handle 200 records PUTs/s, burstable up to 6,000 PUTs/s. Other AWS services used in the deployment included Amazon Kinesis, AWS Kinesis Firehouse, EC2, VPC and Cloud Former

The Benefit

This Social Media Company enjoys the benefit of a reproducible infrastructure, which features templates that can be deployed, on-demand, based upon the unique needs of each client. Blue Sentry over delivered on the project by creating the configuration management stack that was not delineated in the initial scope of work, yet became necessary as the project unfolded. Now, their clients benefit from a stable environment that can scale and flex to handle their dynamic needs.