Case Study: Serverless Research Platform in Azure

The Challenge

An established asset manager approached me with a research platform challenge. Their application brought together market data with their own research and opinions into one view to create and evaluate their strategies.

Their legacy implementation ran on several servers as a monolithic application, with a hand managed and complex infrastructure. The overall performance was poor, which is unacceptable in a trading research environment where timeliness, speed and accuracy are paramount. In addition, the ongoing cost of this setup was hard to justify.

Requirements

• Ingest various feeds of market data sources
• Combine the price data, with holdings and position calculations
• Present dashboards and analytics to research analysts
• Ongoing low effort maintenance, cost-efficient and automated

Key Considerations

1. The solution had to be able to process several data feeds reliably and merge with the in-house research data
2. Security was paramount, with the highly sensitive trading research information held within the system
3. The application had to be responsive, dynamic with auto-scaling and auto-healing capabilities

Technologies Used

• Azure App service sharing a single App Service plan
• Bamboo for app build and deployment to registry
• Cosmos DB with a MongoDB interface
• Azure Batch execution to handle daily jobs

The Solution

After reviewing the challenges, my team were able to create an elegant serverless solution in Azure leveraging the huge power and scale of the cloud, which is simply not possible in a private cloud. Where possible we used cloud native services for their low cost, breadth of features available, and ease of management.

We took the existing legacy application and broke it up into microservices. The code was entirely rewritten into a modern cloud native stack, separating real-time applications from batch processing to be able to handle these appropriately.

The platform has built-in multi-region resiliency across the Azure cloud and is also hugely auto-scalable based on workloads and user demand. Within the application there are health checks and integrated alerts, automated recovery and restarts if any service fails.

Utilising such a cloud native serverless solution allows us to obtain huge processing power when needed, and only pay for what was actually used.

Technical Details

The entire platform has been built as a serverless solution. The batch work now uses Azure Batch; scripts have been turned into Docker containers which are scheduled onto temporary machines used for the smallest amount of time necessary. This allows for simplified management, huge parallelism, and low cost.

The application servers are now hosted with Azure App Service to leverage the reliability and huge number of management tools available. My team built out the data pipelines to move data from the large number of feeds into an Azure Cosmos DB. A managed database was used to take away the headaches of managing a database on a machine, such as patching or disk management, while getting access to features such as geo-redundant automatic backups and one-button scaling.

We also developed the API server that allows individual services to get access to the data, as well as a web app that the client interacts with to perform their data analysis.

Bitbucket and Bamboo are used as the CI/CD pipelines, which test and build the application and batch Docker containers and then deploy to the container registry, all automatically when the code is updated. This can be adapted to use any repository and build server products if needed, such as Azure DevOps.

Security is crucial, with all access to the application handled natively within Azure and integrated with Azure AD for a seamless and authentication experience with world-class security.

The Benefits of Azure App Service

Everything you need to secure and manage a web app is contained in Azure App Service:

• The servers are created as Docker containers, making them easy to create, version, and deploy
• Deployment tools such as blue/green deployments, split traffic, and slot dependent environment variables for minimal interruption in deployments
• All authentication handled through Azure Active Directory. This leveraged Azure class security while simplifying the platform development as app security did not need to be included in the app itself
• One button upgrading / downgrading of hardware, with a change taking around 2 minutes
• Custom domain handling and TLS/SSL set up in a few button presses
• Network security, such as restricting access by IP
• Log compilation and log streaming
• Encrypted connection strings