Srinimf -Blogs

If you’ve been building data pipelines on Databricks, you’ve likely used Spark Structured Streaming — the powerful low-level API for real-time data processing. But with the introduction of Spark Declarative Pipelines (SDP), formerly known as Delta Live Tables, Databricks has reimagined how pipelines should be built. So which one should you use? In this post, we break down both approaches — what…

Why Linking Words Matter Many English learners know grammar and vocabulary, but they often struggle to connect their ideas smoothly while speaking. For example, they may have several good points to share, but the conversation sounds disconnected. In other words, linking words help transform separate sentences into a clear and organized message. There are several benefits to using linking words,…

Modern data engineering requires handling both real-time streaming data and traditional batch workloads efficiently. Databricks has evolved Delta Live Tables (DLT) into Lakeflow Declarative Pipelines, enabling data engineers to build reliable and scalable pipelines with less code. Using the dp module, developers can define streaming tables and materialized views declaratively. Databricks…

Artificial Intelligence is changing the world faster than ever, and one of the biggest innovations today is AI Agents. From automating tasks to making intelligent decisions, AI agents are becoming powerful digital assistants for businesses and individuals. If you are new to AI, this beginner-friendly guide will help you understand: What AI agents are How they work Real-world use…

Use these for interviews, practice, or YouTube/blog content. Includes table creation + query questions. 1. Find 2nd Highest Salary Table: CREATE TABLE employees ( emp_id INT, emp_name VARCHAR(50), salary INT);INSERT INTO employees VALUES(1,'John',5000),(2,'Mary',7000),(3,'David',7000),(4,'Sam',4000); Question: Find the 2nd highest salary. Answer: SELECT MAX(salary) AS second_highestFROM…

Introduction One common challenge beginners face while working with AWS Glue is handling S3 paths that contain dynamically changing folder names. This usually happens when files are written by applications, APIs, Lambda functions, or ETL jobs that generate hash-based or UUID-based folders for every run. For example, your S3 path may look like this: s3://my-bucket/output/ 8d3f1a92/result/…

In modern data engineering, building scalable and reusable systems is essential. Writing separate SQL queries for every KPI quickly becomes messy and hard to maintain. A better approach?👉 Use a Generic Stored Procedure powered by Dynamic SQL, and trigger it using AWS Lambda. In this blog, you’ll learn: What a generic stored procedure is Why dynamic SQL is important Step-by-step KPI…

In the ever-evolving landscape of big data analytics, tools and technologies play a pivotal role in how businesses derive insights from their data. One such revolutionary tool is Databricks Genie. This article will explore what Databricks Genie is, its features, and how it enhances data analytics for organizations looking to leverage their data for competitive advantage. What is Databricks…

Modern applications often allow users to upload files—documents, invoices, images, or datasets. But a production-grade upload pipeline must be secure, scalable, and well-organized. In this article, we will build a complete end-to-end architecture where: A frontend user submits data An API receives the request The system validates the request against a database A hashed folder key is…

Artificial Intelligence is rapidly transforming the way organizations manage and process data. One of the most exciting developments in this space is the emergence of AI agents. These intelligent systems are capable of analyzing information, making decisions, and performing tasks autonomously. For data engineers, AI agents represent a major shift in how data pipelines, analytics workflows, and…

Artificial Intelligence tools are on the rise, from writing assistants to coding helpers and automation platforms. However, many professionals struggle to compare these tools effectively. This is where the AI Stack becomes important. Modern AI tools like ChatGPT, NotebookLM, and Antigravity serve different purposes, and understanding their roles helps in: Avoiding tool overload Building…

Strong workplace communication is one of the most valuable professional skills today.Whether you’re giving project updates, speaking to clients, or collaborating with teams — the ability to speak clearly and confidently can set you apart. However, many professionals struggle with: Speaking smoothly in meetings Structuring thoughts while talking Avoiding filler words Sounding confident during…

Setting up a Relational Database Service (RDS) database in Amazon Web Services (AWS) is an essential skill for developers and data engineers. This blog post outlines four different methods for creating an RDS database, providing step-by-step instructions that are easy for beginners to follow. Additionally, we will discuss how to create schemas after establishing the database. 1. Creating RDS…

Every developer or data engineer starts the same way — with a database running on their laptop. It works. Until it doesn’t. The moment your application grows beyond testing, local databases become risky: No backups No high availability No scalability No disaster recovery This is where managed cloud databases come in. And one of the easiest ways to make your first move from local to…

Artificial Intelligence tools are proliferating — from writing assistants to coding copilots to workflow automation platforms. But here’s the challenge: Most professionals compare AI tools directly without understanding their role. Instead of asking: 👉 Which AI tool is best? A better question is: 👉 What role does this AI tool play? That’s where the AI Stack comes in. Modern AI tools like…

SQL interviews at FAANG (Facebook/Meta, Amazon, Apple, Netflix, Google) are not about syntax. They are designed to test logical thinking, edge cases, execution order, and data correctness at scale. Many strong candidates fail—not because they don’t know SQL, but because they fall into subtle traps. In this blog, we’ll walk through real FAANG-style SQL traps, why candidates fail them, and how to…

Introduction Databricks has become a premier platform for data engineering, especially with its robust integration of Apache Spark and Delta Lake. However, even experienced data engineers encounter challenges when building and maintaining pipelines. In this blog post, we’ll explore common Databricks pipeline errors, provide practical fixes, and discuss performance optimization strategies to…