Business Need

The client’s existing on-premise data center was limited in scalability and flexibility while being time-consuming and expensive to maintain. During the COVID-19 pandemic, stock trading surged to 3x within 5-6 months, and the on-premise data center reached its total capacity. Planning and adding more capacity to the on-premise data center required a lot of time and huge costs. Therefore, the client needed a modern, flexible, scalable option with improved speed, reliability, and security.

Business Challenge

In today’s digital era, cloud adoption is essential for scalability, flexibility, and cost efficiency. However, India’s leading stock exchange was constrained by an on-premise data center limited to 2x trading volume capacity.

During the 2020 lockdown, stock trading surged 3x within months, maxing out their data center. Expanding on-premise infrastructure proved costly and slow, stalling key surveillance, regulatory compliance, and analytics initiatives.

The client urgently needed a scalable, agile solution to meet dynamic demands with minimal disruption. Evaluating a move to the cloud, they faced strict data residency regulations, requiring data to stay within India and the challenge of migrating over 560 TB quickly despite limited documentation and transparency.

Business Solution

With our domain expertise and vast experience, NuSummit evaluated the options and selected Snowflake on Amazon Web Service (AWS). Being a cloud-native platform, Snowflake eliminated physical infrastructure requirements, enabling agility, flexibility, and scalability. The AWS data center was in Mumbai, thus meeting the regulatory requirement of data residing within India. After selecting the preferred cloud service, we migrated the high data volume (~3,000 Greenplum tables comprising >560 TB data) to the cloud. We established a 10GBPS dedicated line from the stock exchange data center to the AWS data center to ease the migration.

NuSummit, with our domain expertise and talented workforce, sprang into action, and we:

• Scalability and Speed with Snowflake on AWS: Partnered with Snowflake on AWS to achieve unmatched scalability, flexibility, and accelerated data processing.
• Efficient Data Migration and Audit Control: Utilized a metadata-driven, Spark-based framework to migrate over 560 TB of data within just 24 days, implementing an audit control mechanism that enabled a reconciliation framework for exceptional data accuracy.
• Seamless Incremental Data Replication: Developed a replication service to sync incremental data daily from the on-premise data warehouse to Snowflake, ensuring up-to-date and consistent data across systems.

Our partnership with DataSwitch enabled efficient automation and accuracy in key areas:

• DDL Conversion: Achieved 95% accuracy in DDL conversion, reducing manual effort and saving 80% of the time.
• Informatica to Snowflake Migration: Converted Informatica workflows into Apache Airflow DAGs and mappings into SnowSQL scripts, achieving over 65% accuracy and significantly reducing coding time.
• UNIX to Snowflake SQL Conversion: Transformed UNIX scripts embedded with Greenplum SQLs into Snowflake-compatible SQLs, saving more than 75% of the effort required for code conversion.

Tech Stack

Cloud Platforms and Data Warehousing:

• AWS (Amazon Web Services)
• Snowflake

• Apache Spark