As a dedicated Data Engineer, my passion lies in crafting solutions that efficiently transform raw data into actionable insights. I successfully orchestrated a robust data pipeline for an e-commerce giant, seamlessly managing diverse data sources. Leveraging cutting-edge technologies such as Apache Pyspark, Apache Airflow, and Azure Data Factory, this pipeline empowers data-driven decision-making, paving the way for the company’s growth and expansion.
Collaborating with you on this transformative journey has been a pleasure. Throughout our partnership, I’ve harnessed a diverse toolkit encompassing technologies like MySQL, MongoDB, Kafka, Python, Microsoft SQL Server, and more, to execute comprehensive data processing, analytics, and visualization tasks. My expertise extends to Microsoft Azure Data Factory (ADF), SQL Server Integration Services (SSIS), and SQL Server Analysis Services (SSAS).
I’m committed to continuous learning, always eager to acquire new skills and tools to augment my data engineering capabilities. A collaborative team player, I thrive on working with fellow data professionals and stakeholders. Unfortunately, due to character limitations, I’m unable to provide additional details on my extensive technical expertise in data engineering. Let’s connect and discuss how I can contribute to your data-driven success!
Add work experience to your profile. (optional)
Led the design and execution of scalable data pipelines for telecommunication clients, developing end-to-end data architecture and technical solutions.
Implemented ETL processes using Apache Kafka, Apache NiFi, and Apache Airflow, improving data ingestion speeds by 30%.
Engineered and maintained data infrastructure, including data lakes and warehouses, ensuring 99.9% data availability and integrity.
Enforced data governance, including access control and anonymization, enhancing compliance with privacy regulations.
Provided leadership and mentorship to junior team members, resulting in a 25% improvement in team performance.
Led the design and execution of scalable data pipelines for telecommunication clients, developing end-to-end data architecture and technical solutions. Implemented ETL processes using Apache Kafka, Apache NiFi, and Apache Airflow, improving data ingestion speeds by 30%. Engineered and maintained data infrastructure, including data lakes and warehouses, ensuring 99.9% data availability and integrity. Enforced data governance, including access control and anonymization, enhancing compliance with privacy regulations. Provided leadership and mentorship to junior team members, resulting in a 25% improvement in team performance.
Skills: Data Engineering · SQL Server Integration Services (SSIS) · Microsoft SQL Server · Azure Databricks · PySpark
Led the migration of data from legacy databases to newly designed databases, ensuring full compliance with Malaysian taxation rules and improving data accuracy by 30%.
Managed ETL processes for taxation data, transforming and cleansing data to meet the specific business rules for Malaysia’s taxation system.
Developed complex SQL queries and optimized stored procedures for data transformation, significantly enhancing data processing speeds and reducing query times by 40%.
Created and optimized Power BI reports, integrating data from multiple sources, and delivered insightful dashboards for tax reporting.
Ensured data integrity during migration by implementing validation and verification processes, resulting in a seamless transition to the new system.
Developed and optimized T-SQL stored procedures and managed ETL processes using SSIS.
Automated data pipelines and implemented real-time data streaming solutions with Apache Kafka and Spark Structured Streaming.
Managed and optimized Apache Spark and PySpark jobs, reducing processing times and enabling the analysis of large structured and unstructured datasets.
Collaborated with data scientists and stakeholders to design scalable, secure data storage solutions, supporting both data engineering and analysis efforts.
Developed and maintained end-to-end operations of ETL data pipelines using Azure Data Factory.
Fetched data from different sources (MongoDB, SQL Server (OLTP, OLAP), Kafka) using PySpark for analysis and reporting.
Utilized DAX and MDX for reporting by Power BI and Tableau, ensuring accurate representation of data for stakeholders.
Key Projects
Economic Council Database: Designed and implemented SQL Server databases and web services for reporting and integrated data from multiple sources into a data warehouse using SSIS and SSAS.
Real-Time Data Streaming: Implemented real-time data streaming capabilities using Apache Kafka and Spark Structured Streaming, enabling efficient data processing for IoT devices and real-time analytics.
Designed and implemented systems for reservation management and compliance with IATA regulations.
Developed data integration and ETL processes for travel agency systems using SQL Server.
Enhanced data querying and processing through the development of complex stored procedures.
Designed and managed SQL databases, integrating them with C# .Net applications for efficient data processing and reporting.
Collaborated with software development teams to ensure smooth integration of data into business systems.
We will review the reports from both freelancer and employer to give the best decision. It will take 3-5 business days for reviewing after receiving two reports.