About Me

I am a Data Engineer with 8 years of experience working with AWS services, ETL workflows, and code optimization. I specialize in optimizing ETL pipelines through SQL, PySpark, and Python, helping companies harness the power of their data.

My role as a Lead AWS Data Engineer involves spearheading the development of data pipelines, notably optimizing ETL processes through SQL, Pyspark, Python, dbt, Snowflake, AWS services (AWS Glue, AWS EMR, AWS Kinesis, AWS Lambda, SQS, SNS, EC2, S3, etc.) and orchestrating data workflows with Airflow. I have successfully integrated advanced data transformations, contributing to significant profitability growth via insightful, strategic analytics solutions.

With a Postgraduate Program in Artificial Intelligence and Machine Learning from The University of Texas at Austin, I combine academic prowess with practical expertise in PySpark, SQL, and Python to deliver comprehensive Quarterly Business Reviews. These efforts have not only addressed critical business challenges but have also earned multiple honors, reflecting our commitment to excellence in data-driven innovation.

Experience

Manager - Lead Software Engineer

Duration: Sept 2024 - Present

Responsibile for creating complex data pipelines for our client NFL (National Football League)
Utilising several AWS services, PySpark, Python, SQL, and DBT

Team Lead - Data Engineering & Analytics

Duration: April 2022 – Sept 2024

Led the development of complex Dbt-Snowflake queries, procedures, and workflows to establish robust data pipelines and facilitate seamless data transformations
Collaborated closely with cross-functional teams to develop data pipelines, utilizing SQL, Python, PySpark, and Tableau dashboards to drive profitability growth.
Demonstrated proficiency in AWS services i.e. Glue, Lambda, EMR, Kinesis, SNS, SQS for efficient data integration and transformation.
Analyzed KPIs and built Tableau dashboards for pricing of software SKUs (SAAS products) such as Discount on prices, inbuilt YOY price increases, multi-year deals, Renewal price increases & volume increases.
Implementing Customer Segmentation to predict ARR (Annual Recurring Revenue) through machine learning techniques, utilizing K-means clustering and Random Forest algorithms.

Consultant - Data Engineering & Analytics

Duration: August 2019 – April 2022

Conducting landscape studies for new technologies in international markets by analyzing technical data using Pyspark, AWS services, SQL, Snowflake, python and Advance Excel, and converting the analyzed data into useful insights by using visualization tool i.e., Tableau
Portfolio analysis of top automotive companies and identifying their major technology and research areas by quickly analyzing their technical literature portfolio with the help of SQL and Advance Excel and making client interactive presentations.

Research Associate – R&D Analytics & Data Engineering

Duration: July 2017 – August 2019

R&D investment analysis of top automotive company’s v/s their Intellectual Property strength and identifying the key technology area gaining investment from multiple companies using Pyspark, SQL, snowflake and data visualization techniques.
Actively interacting with international clients on calls and e-mails for creating a clear understanding of their demands.

Skills

Snowflake (SQL)

A cloud-based data warehousing solution that integrates with modern data platforms for efficient querying and analytics.

Tableau

A leading data visualization tool that helps in creating powerful dashboards for insightful data analysis.

Python

A versatile programming language used for data analysis, machine learning, and automation.

PySpark

A Python API for Apache Spark, designed for large-scale data processing and machine learning.

AWS Services

A suite of AWS services for data storage, processing, and orchestration to build scalable data pipelines.

dbt

A data transformation tool that enables analytics engineering workflows by transforming raw data into clean models.

Airflow

An open-source tool for programmatically authoring, scheduling, and monitoring workflows and data pipelines.

ML Algorithms

Experienced in both supervised and unsupervised machine learning techniques for prediction and clustering.

Advanced Excel & VBA

Proficient in Excel for data manipulation and VBA for automating repetitive tasks.

CICD & Bitbucket

Skilled in setting up Continuous Integration and Deployment pipelines using Bitbucket and other tools.

Agile & JIRA

Expertise in Agile methodologies with JIRA for project management and sprint planning.

Education

HBTU Kanpur

Duration: 2013-2017
Bachelor of Technology
(Ranked 23rd for Government Engineering colleges by India Today 2024)

The University of Texas at Austin

Duration: 2022-2023
Postgraduate Program, Artificial Intelligence and Machine Learning
(Ranked 66th in the QS World University Rankings 2024)

Hello, I'm Mridul

Talk to my AI!