🚀 Lead Data Engineer | Senior Cloud Data Engineer | Analytics & Data Integration | Independent Consultant
In data warehousing, Slowly Changing Dimensions (SCD) are essential for accurately tracking and managing changes in data over time. This guide provides an in-depth exploration of SCD types 0 through 4 and their practical implementation in modern data...
Photo by Ant Rozetsky on Unsplash Fault tolerance is an important requirement in distributed systems. Apache Spark provides robust fault tolerance mechanisms to ensure reliable data processing. In this blog, we will explore how Spark achieves fault t...
Photo by Jim Wilson on Unsplash Have you ever wanted to pivot a Spark dataframe to change rows into columns or vice versa? Let me tell you that Spark provides a simple function to do it. In this blog, we will explore what the pivot function does. Wha...
https://unsplash.com/fr/photos/ItLQ6scEmz0 In the realm of distributed data processing, Apache Spark stands as a prominent figure, transforming the landscape of big data analytics with its lightning-fast performance and user-friendly APIs. When it co...
Unsplash image of Maarten van den Heuvel In today’s data-driven landscape, managing and optimizing data storage is a critical factor in achieving efficient data processing and analysis. We’re starting reading from a storage and finishing writing on a...
Delta Lake is an open-source storage layer that brings reliability to data lakes. Delta tables are a core abstraction in Delta Lake, which are similar to tables in a traditional database. Delta tables can become fragmented over time due to updates, d...
Image by Kai Pilger on Unsplash Delta Lake, an open-source storage layer that brings reliability to data lakes, allows you to store and manage data in data lakes. Delta tables are a core concept of Delta Lake, which enables data versioning, transacti...
Photo by Aidan Bartos on Unsplash Data engineering is a crucial field for organizations looking to make data-driven decisions. However, the costs associated with learning and practicing data engineering can be a significant barrier for beginners. The...
Photo by kris on Unsplash Introduction As a data engineer, you need to enhance your productivity from both technical and functional perspectives to deliver substantial impact and value to your organization. In this comprehensive blog post, I will dis...