Engaged in a pivotal role within the E-commerce US Data Lake project, dedicated to crafting a centralized data lake platform on Azure. The primary goal is to seamlessly ingest data from diverse sources, encompassing both batch and real-time data, and empower business stakeholders across departments to construct data warehouses. The project, labeled ECS US Data Lake, involves the following responsibilities: - Facilitating effective communication with business stakeholders across various departments to comprehend requirements and orchestrate design discussions with the team. - Orchestrating the setup of Infrastructure Platform components and implementing security measures using automation tools, specifically Terraform. - Devising versatile Azure Data Factory pipeline templates to streamline the ingestion and consolidation of data from numerous source systems, spanning both batch and real-time environments. - Crafting and implementing consolidation strategies, along with business models and reports in Databricks using PySpark. - Design discussions with the platform and DevOps engineers to develop CI/CD processes and pipelines Working on Microsoft Fabric Evaluation and Migration Impact Analysis POC for Fabric Data Factory, Lakehouse and One Lake/Direct Lake

Engaged with a mining industry client, spearheading efforts to efficiently handle and refine data from diverse source systems through Azure Cloud technologies, including Data Factories, pipelines, and Azure Databricks. Project/Product: Centralized Data Provisioning Key Responsibilities: - Lead, mentor, and provide technical guidance to a team of 5 data engineers. - Innovatively designed a streaming ingestion and curation pattern utilizing Azure EventHub services. - Developed an ingestion and curation data pipeline for building a BI analytics-focused data lake. - Engineered an event-based system for consuming and curating data from event grid topics using Azure cloud services.

Contributed to the modernization of a platform product by leveraging Spark, Scala, and an AWS EC2 cluster for aggregations and transformations. This project focused on standardizing terabytes of incremental Market Ownership dataset for millions of companies. Project/Product - Ownership Incremental Standardization Key Achievements: - Implemented Spark-based parallel ingestion and processing framework, significantly improving performance compared to legacy systems. - Led the end-to-end delivery of a complex system involving multiple frameworks for ingestion, initialization, standardization, and aggregation. - Successfully optimized performance for complex problems, such as hierarchical queries and aggregates, using techniques like caching, repartitioning, and broadcasting. Project/Product - Modernization of Loaders using Kafka Key Contributions: - Executed the implementation of a real-time Kafka pipeline, handling millions of transactional CDC messages and processing them with business logic before persisting in the target database. - Led the end-to-end pipeline delivery, designing and developing the Kafka consumer with adherence to best practices and guidelines. - Achieved performance optimization and addressed latency issues in message consumption through Kafka optimization techniques and Docker in production.

Project/Product - CHIEE (Clinical Health Information Engagement Enablement) - Anthem (Nov 2016 - Jan 2018) - Led the development of a data lake storing clinical healthcare data from wearable devices through a multi-layer framework.Responsibilities included designing a real-time data ingestion and processing framework using Kafka and Spark streaming,building data models for MongoDB, and integrating layers for an end-to-end data lake solution. Project/Product - D-rive Telematics (Usage-Based Insurance) - (Jan 2015 - Oct 2016) - Contributed to a Data Analytics Telematics solution capturing user data through sensors on a mobile app. Responsibilitiesinvolved designing and implementing rules and algorithms using Java MapReduce and Hive scripts, as well as developingOozie workflows. Project/Product - Anthem Healthcare Modernization (Jun 2014 - Dec 2014) - Focused on the ingestion and transformation of healthcare data from Teradata to Hadoop (Hive) in parquet format using SparkScala. Responsibilities included designing and building an ingestion and processing framework.

Project/Product: Emerson Process Management (EPM) - Develop framework models and reports containing crucial measures for analyzing business status. These reports will empower management in making strategic decisions for organizational growth. Roles/Responsibilities: - Design and develop DataMarts using a framework manager to meet user requirements. - Implement security at different levels, including data, object, and package. - Design intricate reports and dashboards based on end-user specifications, incorporating features such as drill-through and master-detail functionality.

Completed Microsoft DotNet Framework 3.5 training with a CGPA of 4.78 (5). Created a Windows Phone 7 application, 'EnRAllocation Management System,' using Silverlight and C#.