#preprocessing

"Forgive me, Father, I have sinned."

Cardinal!Thrawn

See all images on Patreon

Hey everyone, This is video related to #datapreprocessing and #EDA for #Sentiment #Analysis Case Study. In this project we … source

Let’s get real: raw data is often a hot mess. It’s messy, incomplete, sometimes downright weird. Before you can analyze it or feed it into a machine learning model, you’ve got to clean it up. Luckily, Python is like your trusty sidekick, packed with tricks to tame even the wildest datasets.

Ready to level up your data game? Here are the top Python hacks that’ll turn your messy data into gold.

Spot & Fix Missing Data — Because Nothing Likes Blanks 😬 Missing data is like that one friend who ghosts you—awkward and confusing.

Use Python to find those pesky gaps.

Then decide: drop them if they’re few or fill them with smart guesses like the average.

Bye-Bye Duplicates — Because Twins Can Be Trouble 👯‍♂️ Duplicate rows? Double trouble for your analysis.

Python’s got your back—one line and those sneaky duplicates vanish.

Fix Your Data Types — Because Numbers Should Act Like Numbers 🔢 Ever seen numbers saved as text? It’s like trying to do math with words—doesn’t work.

Convert them the right way so your calculations make sense.

Clean Up Text — Because Extra Spaces & Weird Caps Are No Fun 🧹 Text data often comes with unwanted spaces, random capital letters, or odd characters.

Strip, lowercase, and replace like a boss to make it neat.

Scale It Right — Because Not All Numbers Play Fair ⚖️ If one number is in millions and another is between 0 and 1, your model gets confused.

Normalize or scale your numbers so everything plays nice.

Turn Categories Into Numbers — Because Models Don’t Speak Human 🗣️➡️🤖 Machine learning models love numbers, not words.

Convert categories into neat numeric codes or one-hot vectors to keep models happy.

Handle Outliers — Because Weird Data Points Can Wreck Your Day 🚨 Outliers are those oddballs that don’t fit the pattern and can mess up your results.

Spot them and decide if you want to fix, remove, or keep them (but watch out!).

Feature Engineering — Because Sometimes You’ve Got to Create Magic ✨ Creating new features from your data can unlock hidden insights.

Extract dates, combine columns, or create bins—the possibilities are endless.

Why Should You Care? Because clean data = better insights + smarter models + less headache. Skipping these steps is like trying to bake a cake without measuring ingredients—it just won’t turn out right.

Dealing with missing data

In real world data, there are some instances where a particular element is absent because of various reasons, such as, corrupt data, failure to load the information, or incomplete extraction. Handling the missing values is one of the greatest challenges faced by analysts, because making the right decision on how to handle it generates robust data models. Let us look at different ways of imputing…

In 2018, I extracted my Kakaotalk history with my friend. It was... very long. Looking at it, I didn’t want to add two more years to that pile so I decided to start with what I had.

...though what I had was apparently not good enough.

The above are encoding errors I got when Excel converted UTF-8 to ANSI while importing the text data. The weird black box things were supposed to be apostrophes, Korean letters, and emoticons. While the Korean letters weren’t exactly useful to VADER (and I could discard them), the apostrophes and emoticons were.

Trying to import the text data into Excel so that I could convert it into csv also produced extra, unwanted columns:

When I looked through the data, I found the culprits:

Since I imported the text data with a comma delimiter, Excel added a column every time there was a comma... even when they were grammar-related.

To fix this issue, I made up a completely ridiculous delimiter to use:

I don’t think I’ll ever use “@@@” consecutively in a conversation anywhere, so it was perfect.

And it worked...!

With the encoding problem:

Those were supposed to be apostrophes, so I just replaced them all in Excel... which was easy. What was not easy were the emoticons, and just as I was about to give up entirely, I discovered something that I had overlooked completely:

XD

Which solved most of my issues.

The last problem was this:

There were so many errors.

Which... obviously should not be a date. After some digging, I found the source of the problem:

Basically, any Kakaotalk message that was not in a single line was exported as separate lines... and Excel couldn’t read them in the format I was trying to coerce it into. So I went in, ctrl + f searched for all of the above instances and fixed them manually.

I still haven’t found out what the other errors were indicating, but I figured that 262 against 224k rows of data was negligible enough to discard them.

And this was the end of my confusing and frustrating preprocessing attempts :’D

python anaconda 

https://www.anaconda.com/distribution/#download-section

r language

https://cran.cnr.berkeley.edu/

r studio ( r language ide )

https://www.rstudio.com/products/rstudio/download/#download

R, PYTHON에서 작업하기 이전에 데이터셑이 있는 폴더를 working directory로 지정하고 작업을 해야한다.

python 에서 기본적으로 필요한 라이브러리 numpy는 mathmatics 기능을 도와준다.

matplotlib은 plot 그리는데 필요한 도구. pandas는 dataset을 가져오는데 사용한다.

python에서 dataset.iloc[:, :-1].values 의 의미는 모든 행을 이용하고 칼럼은 뒤에서 하나를 빼고 값들을 가져온다.

dataset.iloc[:, 3].values 의 의미는 모든 행을 이용하고 칼럼은 앞으로 부터 4번째만 값을 가져온다.

참고자료 https://stackoverflow.com/a/46692001/3151712

R에서 데이터 가져오는 방법

MISSING DATA

missing data가 있는 경우 그 데이터를 삭제하기 보다 중간값이나 자주등장하는 값 등으로 교체해서 처리하는 방법이 있다.

X[:, 1:3] 에서 2번째 , 3번째 칼럼을 말한다. ( lower bound는 포함, upper bound는 포함하지 않는다. ) 

command + i  단축키를 통해 class 의 notation를 확인가능하다.

code 선택후 command + enter 를 통해 부분 실행이 가능하다.

CATEGORICAL DATA

이 강의 예시에서 국적 칼럼의 경우 그 값이 문자열로 되어있고 딱 세개로 되어있고 그 값에 특별한 우열이 없는 경우 정리가 필요하다. 우선 문자열을 수치 계산이 가능한 숫자로 바꾼다. 또 우열이 없으므로 각각 동등한 값을 가지도록 각 국적별 칼럼을 만들고 값이 1이 되게 한다. 

그 정리 결과가 위와 같게 한다.

참고 자료) label encoder와 one hot encoder의 역할 설명

https://medium.com/@contactsunny/label-encoder-vs-one-hot-encoder-in-machine-learning-3fc273365621

SPLITTING DATASET INTO TRAINING SET AND TEST SET

training set과 test set으로 구분하는 작업

결과 예시는 위와 같이 된다. training features and results , test features and results

train_test_split에서 random_state 는 숫자가 커질수록 무작위의 데이터가 순서에 상관없이 training set과 test set으로 나뉘어 들어가게 된다. test_size는 전체에서 차지하는 percent이다. 

FEATURE SCAILING 

각각 feature값의 범위차가 너무 큰경우 특정 feature가 미치는 영향이 너무 커지므로 어느정도 범위를 맞추어 주어야 한다. 보통 두가지 방법이 있다.

standardisation를 이용한 예시를 보여주고 있다. 우측 데이터들을 보면 비슷한 범위의 값들로 정리된것을 볼수 있다. feature scailing 의 장점의 하나로 계산속도의 향상을 들수 있다. 낮은 수준의 값으로 정리되기 때문이다.

DATA PREPROCESSING TEMPLATE

매 작업마다 기본적으로 수행되는 부분을 template으로 따로 화일을 만들어서 반복 coding하는 번거로움을 줄이는 작업이다. 기본적으로 라이브러리 import 작업, features matrix만드는 작업, 결과 vector만드는 작업. training set 과 test set 구분하는 작업이 들어간다.

Mass Detection and Classification System for Mammography Image Preprocessing

By S. Thilagavathi | Dr. M. S. Irfan Ahmed"Mass Detection and Classification System for Mammography Image Preprocessing"

Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-2 | Issue-1 , December 2017,

URL: http://www.ijtsrd.com/papers/ijtsrd8223.pdf

http://www.ijtsrd.com/medicine/other/8223/mass-detection-and-classification-system-for-mammography-image-preprocessing/s-thilagavathi

call for paper international journal, special issue publication, economics journal, indexed journal