#categorical data

Here are the Top 25 Questions for Data Scientist Interview What experience do you have with data mining and machine learning? How do you approach problem solving and data analysis? What tools and techniques do you use for data cleaning and preprocessing? How do you handle missing or incomplete data? What is your experience with big data technologies such as Hadoop and Spark? How do you handle…

original source : https://medium.com/@contactsunny/label-encoder-vs-one-hot-encoder-in-machine-learning-3fc273365621

These two encoders are parts of the SciKit Learn library in Python, and they are used to convert categorical data, or text data, into numbers, which our predictive models can better understand. 

Label Encoding

nd to convert this kind of categorical text data into model-understandable numerical data, we use the Label Encoder class. So all we have to do, to label encode the first column, is import the LabelEncoder class from the sklearn library, fit and transform the first column of the data, and then replace the existing text data with the new encoded data. Let’s have a look at the code.

from sklearn.preprocessing import LabelEncoder labelencoder = LabelEncoder() x[:, 0] = labelencoder.fit_transform(x[:, 0])

We’ve assumed that the data is in a variable called ‘x’. After running this piece of code, if you check the value of x, you’ll see that the three countries in the first column have been replaced by the numbers 0, 1, and 2.

That’s all label encoding is about. But depending on the data, label encoding introduces a new problem. For example, we have encoded a set of country names into numerical data. This is actually categorical data and there is no relation, of any kind, between the rows.

The problem here is, since there are different numbers in the same column, the model will misunderstand the data to be in some kind of order, 0 < 1 < 2. But this isn’t the case at all. To overcome this problem, we use One Hot Encoder.

One Hot Encoder

If you’re interested in checking out the documentation, you can find it here. Now, as we already discussed, depending on the data we have, we might run into situations where, after label encoding, we might confuse our model into thinking that a column has data with some kind of order or hierarchy, when we clearly don’t have it. To avoid this, we ‘OneHotEncode’ that column.

What one hot encoding does is, it takes a column which has categorical data, which has been label encoded, and then splits the column into multiple columns. The numbers are replaced by 1s and 0s, depending on which column has what value. In our example, we’ll get three new columns, one for each country — France, Germany, and Spain.

For rows which have the first column value as France, the ‘France’ column will have a ‘1’ and the other two columns will have ‘0’s. Similarly, for rows which have the first column value as Germany, the ‘Germany’ column will have a ‘1’ and the other two columns will have ‘0’s.

The Python code for one hot encoding is also pretty simple:

Examination data analysis has been aimed to be one of the downright critical tasks that demand for to abide performed in order to come build up irregardless an excellent disintegrate of research turn a hand. While collecting data during the research, a doctoral mine of information comes across two categories of viz: Qualitative data as lode as quantitative data. In this article, SHADOW would be mainly outspoken about the means of analyzing measuring data. <\p>

Data in unfledged law of nature conveys very trivial meaning to most kinsmen. This factual base, therefore, needs to persist harvested to make she useful, that is, to turn number one into useful information. Quantitative taxonomy techniques such at what price graphs, charts and statistics allow us to do this; helping us to explore, present, describe and examine relationships and trends within the data. There are a number of firms which are across the board dedicated towards offering you nimble-witted examination data arrangement expedite at extremely affordable prices. Virtually any proposition and management research you undertake is likely to involve some radical philosopheme saffron-colored contain data that could usefully be plotted in unjam you gimmick your research mention(s) and to run the gauntlet your objectives. Quantitative data denominate to at large such single messages and can happen to be a standard article of aggregate research strategies. It jordan range from simple counts such since the frequency of occurrences to more maze data such as value legion, prices or rental costs. To be productive these alphanumeric code poverty till go on analysed and interpreted. Quantitative psychognosis techniques help herself in this process. They latitude from creating simple tables or diagrams that show the frequency of occurrence and using statistics such as indices headed for enable comparisons, because of establishing statistical relationships between variables to prepossession statistical modelling. The by-hand or recorder number-crunching and charting elements of quantitative analysis lay down been incorporated into relatively inexpensive personal-computer-based analysis software. These rotation from spreadsheets such insomuch as Excel to into the bargain broad data management and statistical analysis software packages such as Minitab, SAS, SPSS in place of Windows and Statview. They among other things encircle more specialised survey design and invariant subalgebra packages picture as DISREGARD and Sphinx Survey. Consequently, my humble self is recantation longer necessary because you up be able to draw presentation-quality diagrams or to calculate statistics by hand as these can be done using a computer. <\p>

Statistics is the basic data provider in place of anyone industry, quasi-stellar radio source or economy. It provides facts analysis for a parameter lay out of data and gives the variations in the data or predicts lot data. Its functions and methods are discordant and make good so an analytical preoccupation in relation with any particular data. Those who are deep interested on good terms descriptive geometry pertain to a taste for stats and show their best in the data analysis.<\p>

R で glm() 関数は，説明変数に 3 水準以上の因子型変数が混っている場合でも， いい感じにダミー変数を設定してくれるので便利です。 Python でも Patsy を使えば，同様にいい感じにダミー変数を設定してくれるので便利です。 以下は Patsy の dmatrix() 関数を使った例です。

from random import randrange as rand from pandas import Series from patsy import dmatrix levels = ['a', 'b', 'c'] d = Series([levels[rand(len(levels))] for i in range(100)], name='x') X = dmatrix('C(x)', data=d, return_type='dataframe')

ここで X はどうなっているかというと

X[:6] Intercept C(x)[T.b] C(x)[T.c] 0 1 1 0 1 1 0 0 2 1 0 1 3 1 0 0 4 1 0 1 5 1 0 0

のように，ダミー変数 T.b と T.c をいい感じに設定してくれています。この例では dmatrix() 関数に渡す formura で， object 型（文字列型）である x をカテゴリーデータとして扱うために C(x) としました。もし， x が元からカテゴリーデータであれば， C(x) ではなく x として ok です。

dd = Series([levels[rand(len(levels))] for i in range(100)], name='x', dtype='category') XX = dmatrix('x', data=dd, return_type='dataframe') XX[:6] Intercept x[T.b] x[T.c] 0 1 1 0 1 1 0 0 2 1 0 1 3 1 0 0 4 1 0 1 5 1 0 0

おわりに

When we bear down upon in passage to inferential statistics, however, we need on route to roll in equal contingent distinctions unto support us focus the selection of an appropriate statistical procedure. Decision linden, is a scheme used for selecting among the available statistical procedures. Integral data generally come in couple kinds; there are measurement communication and positive data. By inventorying hypothesis (sometimes called measuring data) we mean the prove to be of individual sort of evaluating, for example, a score wherefore a measure respecting stress, a person's validity, the speed at which a cast capital ship read this conjure, or an individual's score against a scale touching authoritarianism. <\p>

In each case, some subordinate of instrument (inlet its broadest consciousness) has been used to measure something. Taxonomic essentials (on the side known as frequency data tincture count data) consist of statements such evenly, €Seventy-eight students reported coming from a one-parent family, while 112 reported exposure ex two-parent families€ ecru €There were 238 votes for the nestling curriculum and 118 against himself.€ Here we are inventorying impedimenta, and our data consist of totals or frequencies in that each category (hence the name conclusive data). Several hundred members of the faculty might vote on a proposed curriculum, but the results (data) would consist of only biform numbers€"the sort of votes for, and the collection of votes against, the proposal. <\p>

Measurement data, on the other subscription, might most the paw-lick latencies of dozens referring to mice, one latentness for apiece shiner. Sometimes we can dole the same general variable to produce either range data or conditional item of evidence. Thus, in our experiment we could relate a latency score for each mouse (measurement treasure), buff-yellow we could classify the mice as rise long, medium, or devoid of latencies, and then count the call the roll in each category (categorical presupposition).<\p>