#data preprocessing

Implementing Proactive Ambient AI Agents is increasingly becoming a necessity for businesses looking to enhance user engagement and operational efficiency. By proactively addressing customer needs and streamlining processes, companies can benefit greatly from these advanced AI capabilities.

The evolution and implementation of Proactive Ambient AI Agents should be approached with a well-defined strategy. Here are several best practices that organizations can adopt when transitioning to these sophisticated AI systems.

Effective Data Preprocessing

Data preprocessing forms the backbone of any successful AI initiative. Ensuring data accuracy through rigorous data ingestion and cleaning processes allows Proactive Ambient AI Agents to learn and evolve effectively. Organizations must invest in robust feature engineering practices that help identify the most relevant data points, which can significantly impact the learning and adaptation of AI models in real-time.

Model Training and Optimization

Continuous training and validation of AI models are vital for maintaining performance and reliability. Applying techniques like transfer learning can enhance model accuracy while reducing training time and costs. Additionally, integrating user feedback during model optimization phases helps ensure the AI agents meet user expectations, fostering a more reliable user experience. For organizations interested in accelerating their AI initiative, exploring AI-based development strategies can be highly beneficial.

Conclusion

Introduction

So far in this masterclass, we’ve explored individual feature engineering techniques—handling missing data, encoding categories, scaling features, creating new variables, and reducing dimensionality. In real-world machine learning projects, however, these techniques are never applied in isolation.

Feature engineering in practice is about combining methods correctly, avoiding common pitfalls, and building reproducible pipelines that work reliably across training, validation, and production environments.

This final episode ties everything together with practical guidance, real-world considerations, and a complete end-to-end workflow.

Building a Feature Engineering Pipeline

In production-grade machine learning, feature engineering should always be systematic and automated, not ad hoc.

A proper feature engineering pipeline typically includes:

Missing value handling

Categorical encoding

Feature scaling or transformation

Feature creation and selection

Model training

Using pipelines ensures that:

The same transformations are applied consistently

Training and inference behave identically

Human errors are minimized

Pipelines also make models easier to maintain, debug, and deploy.

Avoiding Data Leakage

One of the most critical mistakes in feature engineering is data leakage—when information from the future or from the test set leaks into training.

Common leakage sources include:

Calculating statistics (mean, median, scaling factors) on the full dataset before splitting

Using target-based encodings without proper cross-validation

Creating features using future timestamps

Performing feature selection before train-test split

Best practices to prevent leakage:

Always split data before fitting transformations

Fit preprocessing steps only on training data

Apply learned parameters to validation and test sets

Be especially careful with time-series and target encoding

Avoiding leakage is often the difference between a model that looks great in experiments and one that fails in production.

Cross-Validation Considerations

Feature engineering must align with your validation strategy.

When using cross-validation:

Feature transformations should be fitted inside each fold

Target encoding must be recalculated per fold

Feature selection should be repeated per fold, not once globally

This ensures performance metrics reflect real generalization rather than hidden information reuse.

In time-based data:

Use time-aware splits

Never shuffle data randomly

Create features only from past observations

Automated Feature Engineering Tools

Manual feature creation can be time-consuming, especially with relational or transactional data.

Automated feature engineering tools help by:

Generating aggregations automatically

Creating time-based and relational features

Reducing manual trial-and-error

A popular example is Featuretools, which uses:

Deep Feature Synthesis

Entity relationships

Automated aggregation and transformation primitives

While automated tools accelerate experimentation, they should be used with:

Strong domain understanding

Careful validation

Feature importance analysis

Automation complements expertise—it does not replace it.

Case Study: Before and After Feature Engineering

Consider a simple classification problem using raw data:

Minimal preprocessing

Basic encoding

No feature creation

Initial model performance:

Moderate accuracy

High variance

Poor generalization

After proper feature engineering:

Missing values handled correctly

Categorical features encoded appropriately

Numerical features scaled where required

New interaction and time-based features added

Irrelevant features removed

Results:

Improved accuracy

More stable validation scores

Better interpretability

Stronger performance on unseen data

This demonstrates that feature engineering often contributes more to performance gains than changing models.

Key Takeaways

Feature engineering is a workflow, not a single step

Pipelines ensure consistency and reproducibility

Preventing data leakage is essential

Validation strategy must align with feature creation

Automated tools can accelerate, but not replace, expertise

Well-engineered features outperform complex models with poor features

Final Thoughts

Feature engineering is where data understanding meets machine learning performance. Models may change, algorithms may evolve, but strong features remain the foundation of successful machine learning systems.

Introduction

As datasets grow larger and more complex, models often face a common challenge: too many features. While having more data can be beneficial, high-dimensional feature spaces can lead to slower training, overfitting, noisy patterns, and poor generalization. This is known as the curse of dimensionality.

Dimensionality reduction techniques aim to address this issue by reducing the number of input features while preserving as much meaningful information as possible. In this episode, we explore both feature extraction and feature selection approaches, understand when to use each, and learn how dimensionality reduction improves model performance and interpretability.

The Curse of Dimensionality

High-dimensional data introduces several problems:

Increased computational cost

Sparse data distribution

Higher risk of overfitting

Difficulty in visualizing patterns

Reduced model interpretability

As dimensionality increases, the amount of data required to learn reliable patterns grows exponentially. Dimensionality reduction helps combat these effects by simplifying the feature space.

Principal Component Analysis (PCA)

PCA is a linear dimensionality reduction technique that transforms original features into a smaller set of uncorrelated components called principal components.

Key characteristics of PCA:

Captures directions of maximum variance

Produces orthogonal components

Reduces redundancy from correlated features

Works best with standardized numerical data

PCA is commonly used for:

Improving model efficiency

Reducing multicollinearity

Noise reduction

Preprocessing before regression or clustering

However, PCA reduces interpretability since transformed components no longer correspond directly to original features.

t-SNE for Visualization

t-SNE (t-Distributed Stochastic Neighbor Embedding) is a non-linear technique designed primarily for visualization.

Key points:

Preserves local structure of data

Excellent for visualizing clusters

Commonly used with embeddings and high-dimensional representations

Not suitable for direct model training

t-SNE is most effective for:

Exploring patterns

Understanding class separability

Presenting results visually

Because it is computationally expensive and non-deterministic, t-SNE is best used for analysis rather than production pipelines.

UMAP for Structure Preservation

UMAP (Uniform Manifold Approximation and Projection) is another non-linear dimensionality reduction method that balances local and global structure.

Advantages of UMAP:

Faster than t-SNE

Preserves both local and global relationships

Scales well to large datasets

Can be used as a preprocessing step

UMAP is increasingly popular for:

Exploratory data analysis

Feature compression

Visualizing embeddings in NLP and computer vision

Feature Selection Approaches

Unlike PCA or UMAP, feature selection keeps original features and removes less useful ones.

Filter Methods

These rely on statistical properties of data:

Correlation analysis

Variance thresholding

Mutual information

Chi-square tests

They are fast, model-agnostic, and useful for initial pruning.

Wrapper Methods

These evaluate feature subsets using a model:

Recursive Feature Elimination (RFE)

Forward or backward selection

They are more accurate but computationally expensive.

Embedded Methods

These perform feature selection during model training:

Lasso (L1 regularization)

Elastic Net

Tree-based feature importance

Embedded methods balance performance and efficiency and are widely used in practice.

Using Feature Importance from Tree Models

Tree-based algorithms such as Random Forests and Gradient Boosting provide built-in feature importance scores.

These scores help:

Identify influential variables

Remove low-impact features

Improve model interpretability

Reduce noise

While powerful, feature importance should be interpreted carefully, especially when features are correlated.

Choosing the Right Technique

The choice of dimensionality reduction depends on:

Dataset size and feature count

Model type

Need for interpretability

Computational constraints

Purpose (training vs visualization)

Linear methods suit structured numerical data, while non-linear techniques excel in complex representations and exploratory analysis.

Key Takeaways

High-dimensional data can hurt performance and generalization

PCA reduces redundancy through linear transformations

t-SNE and UMAP are best for visualization and exploration

Feature selection preserves interpretability

Tree-based importance helps guide feature pruning

In our previous post, we built the foundation of a serverless ETL pipeline. We used AWS Glue and PySpark to ingest, clean, and split retail sales data. This data was from Kaggle’s Store Item Demand Forecasting Challenge. That version was intentionally minimal. It dropped null rows. The version also performed basic data type conversions to get the pipeline running end-to-end. Now that the initial…

Inside the Power of Scikit-learn Pipelines Perhaps one of the most underrated yet powerful features that scikit-learn has to offer, pipelines are a great ally for building effective and modular machine learning workflows. A pipeline combines multiple preprocessing steps – like scaling, encoding categorical variables, and feature selection – into a single, reusable unit. This dramatically…

Di era di mana kecerdasan buatan (AI) telah menjadi motor penggerak utama inovasi di berbagai sektor, kemampuan untuk membangun model AI bukan lagi domain eksklusif para ilmuwan data. Kini, dengan akses ke library dan framework yang kian canggih, siapa pun dengan fondasi yang tepat dapat memulai perjalanan ini. Namun, membangun model AI yang efektif bukanlah sekadar menjalankan beberapa baris…

Data is the new oil, and just like crude oil, it needs refining before it can be utilized effectively. Data cleaning, a crucial part of data preprocessing, is one of the most time-consuming and tedious tasks in data analytics. With the advent of Artificial Intelligence, particularly Large Language Models (LLMs), the landscape of data cleaning has started to shift dramatically. This blog delves into how LLMs are revolutionizing data cleaning in 2024 and what this means for businesses and data scientists.

The Growing Importance of Data Cleaning

Data cleaning involves identifying and rectifying errors, missing values, outliers, duplicates, and inconsistencies within datasets to ensure that data is accurate and usable. This step can take up to 80% of a data scientist's time. Inaccurate data can lead to flawed analysis, costing businesses both time and money. Hence, automating the data cleaning process without compromising data quality is essential. This is where LLMs come into play.

What are Large Language Models (LLMs)?

LLMs, like OpenAI's GPT-4 and Google's BERT, are deep learning models that have been trained on vast amounts of text data. These models are capable of understanding and generating human-like text, answering complex queries, and even writing code. With millions (sometimes billions) of parameters, LLMs can capture context, semantics, and nuances from data, making them ideal candidates for tasks beyond text generation—such as data cleaning.

To see how LLMs are also transforming other domains, like Business Intelligence (BI) and Analytics, check out our blog How LLMs are Transforming Business Intelligence (BI) and Analytics.

Traditional Data Cleaning Methods vs. LLM-Driven Approaches

Traditionally, data cleaning has relied heavily on rule-based systems and manual intervention. Common methods include:

Handling missing values: Methods like mean imputation or simply removing rows with missing data are used.

Detecting outliers: Outliers are identified using statistical methods, such as standard deviation or the Interquartile Range (IQR).

Deduplication: Exact or fuzzy matching algorithms identify and remove duplicates in datasets.

However, these traditional approaches come with significant limitations. For instance, rule-based systems often fail when dealing with unstructured data or context-specific errors. They also require constant updates to account for new data patterns.

LLM-driven approaches offer a more dynamic, context-aware solution to these problems.

How LLMs are Transforming Data Cleaning

1. Understanding Contextual Data Anomalies

LLMs excel in natural language understanding, which allows them to detect context-specific anomalies that rule-based systems might overlook. For example, an LLM can be trained to recognize that “N/A” in a field might mean "Not Available" in some contexts and "Not Applicable" in others. This contextual awareness ensures that data anomalies are corrected more accurately.

2. Data Imputation Using Natural Language Understanding

Missing data is one of the most common issues in data cleaning. LLMs, thanks to their vast training on text data, can fill in missing data points intelligently. For example, if a dataset contains customer reviews with missing ratings, an LLM could predict the likely rating based on the review's sentiment and content.

A recent study conducted by researchers at MIT (2023) demonstrated that LLMs could improve imputation accuracy by up to 30% compared to traditional statistical methods. These models were trained to understand patterns in missing data and generate contextually accurate predictions, which proved to be especially useful in cases where human oversight was traditionally required.

3. Automating Deduplication and Data Normalization

LLMs can handle text-based duplication much more effectively than traditional fuzzy matching algorithms. Since these models understand the nuances of language, they can identify duplicate entries even when the text is not an exact match. For example, consider two entries: "Apple Inc." and "Apple Incorporated." Traditional algorithms might not catch this as a duplicate, but an LLM can easily detect that both refer to the same entity.

Similarly, data normalization—ensuring that data is formatted uniformly across a dataset—can be automated with LLMs. These models can normalize everything from addresses to company names based on their understanding of common patterns and formats.

4. Handling Unstructured Data

One of the greatest strengths of LLMs is their ability to work with unstructured data, which is often neglected in traditional data cleaning processes. While rule-based systems struggle to clean unstructured text, such as customer feedback or social media comments, LLMs excel in this domain. For instance, they can classify, summarize, and extract insights from large volumes of unstructured text, converting it into a more analyzable format.

For businesses dealing with social media data, LLMs can be used to clean and organize comments by detecting sentiment, identifying spam or irrelevant information, and removing outliers from the dataset. This is an area where LLMs offer significant advantages over traditional data cleaning methods.

For those interested in leveraging both LLMs and DevOps for data cleaning, see our blog Leveraging LLMs and DevOps for Effective Data Cleaning: A Modern Approach.

Real-World Applications

1. Healthcare Sector

Data quality in healthcare is critical for effective treatment, patient safety, and research. LLMs have proven useful in cleaning messy medical data such as patient records, diagnostic reports, and treatment plans. For example, the use of LLMs has enabled hospitals to automate the cleaning of Electronic Health Records (EHRs) by understanding the medical context of missing or inconsistent information.

2. Financial Services

Financial institutions deal with massive datasets, ranging from customer transactions to market data. In the past, cleaning this data required extensive manual work and rule-based algorithms that often missed nuances. LLMs can assist in identifying fraudulent transactions, cleaning duplicate financial records, and even predicting market movements by analyzing unstructured market reports or news articles.

3. E-commerce

In e-commerce, product listings often contain inconsistent data due to manual entry or differing data formats across platforms. LLMs are helping e-commerce giants like Amazon clean and standardize product data more efficiently by detecting duplicates and filling in missing information based on customer reviews or product descriptions.

Challenges and Limitations

While LLMs have shown significant potential in data cleaning, they are not without challenges.

Training Data Quality: The effectiveness of an LLM depends on the quality of the data it was trained on. Poorly trained models might perpetuate errors in data cleaning.

Resource-Intensive: LLMs require substantial computational resources to function, which can be a limitation for small to medium-sized enterprises.

Data Privacy: Since LLMs are often cloud-based, using them to clean sensitive datasets, such as financial or healthcare data, raises concerns about data privacy and security.

The Future of Data Cleaning with LLMs

The advancements in LLMs represent a paradigm shift in how data cleaning will be conducted moving forward. As these models become more efficient and accessible, businesses will increasingly rely on them to automate data preprocessing tasks. We can expect further improvements in imputation techniques, anomaly detection, and the handling of unstructured data, all driven by the power of LLMs.

By integrating LLMs into data pipelines, organizations can not only save time but also improve the accuracy and reliability of their data, resulting in more informed decision-making and enhanced business outcomes. As we move further into 2024, the role of LLMs in data cleaning is set to expand, making this an exciting space to watch.

One the most amazing experiences of my PhD project was to develop and employ a particle detector for a particle accelerator at CERN. 

This work also involved a quite deal of data preprocessing and analysis, so to determine the efficiency of the detector. It is a great example of how creative data analysis can overcome limitations from the hardware design and improve detection efficiency by up to 40%!

One of my main design philosophy was to build a detector that was as cheap as possible, from salvaged equipment in the laboratory. To overcome limitations from old components, I developed an algorithm to find the timing of the particles using a constant-fractional discrimination technique. This algorithm finds the timing from the rise time of pulse, overcoming artificial increase in particle timing from large pulses. 

One of the great advantages of my algorithm is its speed. While some will fit a special function to the entire pulse in the waveform, my algorithm takes advantage of the rise time being linear to fit a straight line to it. 

My algorithm not only improved the timing measurements by 40%, as it was so efficient that it could be incorporated to the on-the-fly analysis, to maximize the quality of the data being acquired. Every spill in a beam line is precious, so we must ensure the quality of the data is maximal!