Data Analysis and Interpretation Wesleyan University

Results:

A Lasso Analysis was conducted first to reduce the predictors number and keep only the predictors that relevant in predicting the communication technology user’s response variable.

As a result, 77 variables were shrunk to 0 and 8 variables retained in the model:

1-  Adjusted Savings: Energy Depletion (% Of Gni)

2- Adjusted Savings: Energy Depletion (Current Us$)

3- Adolescent Fertility Rate (Births Per 1,000 Women Ages 15-19)

4- Agriculture, Value Added (% Of Gdp)

5- Cause Of Death, By Non-Communicable Diseases (% Of Total)

6- Industry, Value Added (% Of Gdp)

7- Internet users (per 100 People).

8- Personal Remittances, Paid (Current Us$).

Descriptive Statistics:

Table 1 shows descriptive statistics for communication technology users and the quantitative predictors. The average communication technology users were 156.58 (sd=55.28), with a minimum of 32.86 users and a maximum of 262.2 users.

Bivariate Analyses:    

Scatter plots for the association between the communication technology response variable and quantitative predictors (Figure 1) revealed that communication users were decreased when there was a greater number of Adolescent Fertility Rate (Births Per 1,000 Women Ages 15-19) (Pearson r=-0.84, p<.0001) and greater number of Agriculture, Value Added (% Of Gdp) (Pearson r=-0.85,P<0.0001), but increased when internet users(per 100 people) increased(Pearson r=0.87,P<0.0001).

Communication Technology users was not significantly associated with the Adjusted Savings: Energy Depletion (% Of Gni) (Pearson r=0.099, p=.472) and Adjusted Savings: Energy Depletion (Current Us$), (Pearson r=0.27, p=0.043).

The Pearson correlation “r” values and the associated p-values, summarized in the Table 2 below:

Lasso Regression Analysis:

Table 3 shows the retained predictor variables and their respective model co-efficients.

From the above table 2 and figure 2 it can be seen that Internet users (Per 100 people) is     most strongly associated with Communication technology users (Per 100 people) and this is followed by Adolescent Fertility Rate (Births Per 1,000 Women Ages 15-19). This corroborates the -Pearson correlation “r” results. The Communication technology users (Per 100 people) increases when Internet users (Per 100 People) and Industry, Value Added (% Of Gdp) increases and decreases when Adolescent Fertility Rate (Births Per 1,000 Women Ages 15-19) and Agriculture, Value Added (% Of Gdp) increases.

Together, these 8 predictors accounted for 83% of the variance in Communication technology users. The mean squared error (MSE) for the test data (MSE=346.88) differed very little from the MSE for the training data (MSE=512.15), which suggests that predictive accuracy did not decline when the lasso regression algorithm developed on the training data set was applied to predict communication technology users in the test data set Figure 3 below:

Conclusions/Limitations:

This project used lasso regression analysis to identify the best predictors for communication technology users (Per 100 people) of countries from multiple World Bank development indicators in N= 248 countries and regions of the world for the year 2012 which is made up of national, regional and global estimates. The communication technology users (per 100 people) for this period ranged from 32.86 (Per 100 people) to 262.20 (Per 100 people) indicating that there was considerable variability in the communication technology users (Per 100 People) of countries and regions of the world for that year.

The prediction accuracy of the model was (83%) on the training dataset and (88%) when ran on the test dataset. Hence, the algorithm helped to identify the best predictors for communication technology users (Per 100 people) of countries for the year 2012 given the indicators available in this World Bank dataset.

There was significant increase in the MSE when the training set lasso regression algorithm was used to predict the communication technology users (per 100 people) in the test data set. This suggests that the predictive accuracy of the algorithm may not be very stable for future datasets and hence has to be looked into further by using other analytic methods such as Multiple Regression.

The Pearson correlation “r” values and associated p-values revealed that all the retained variables were significantly associated with communication technology users (Per 100 people) as can be seen in the Table 2 above. From the Scatter Plots and the Pearson “r” values and also from the multivariate analysis it can be seen that internet users (per 100 people) is the strongest relation to the communication technology users followed by Industry, Value Added (% Of Gdp).

The Internet users (per 100 People), Industry, Value Added (% Of Gdp), Adjusted Savings: Energy Depletion(% Of Gni), Adjusted Savings: Energy Depletion (Current Us$) all indicated a positive co-efficient ,meaning an increase in these variables are likely to lead to increase in the communication technology users(Per 100 people) of countries and regions.

So, for companies and stakeholders who want invest in communication technology search for countries that have more internet users and strong industry and energy sectors.

On the other hand, the analysis find that communication technology users decrease when Adolescent Fertility Rate (Births Per 1,000 Women Ages 15-19) and Agriculture, Value Added (% Of Gdp) increase and this mean that countries that have great growth in population and its economy depend mostly on agriculture need to reduce  the population growth and focus more on industry and energy sectors and increase internet users if they want to invest efficiently in communication technology sector.  

Limitations

Although the p-values associated with the Pearson correlation “r” values indicated that all the retained 8 variables were significantly associated with The communication technology users, it can be seen that some of the associations were not linear hence the Pearson correlation could not capture the true relationship between those indicators and communication technology users ,therefore could not help in interpreting communication technology users variability given those non-linear associations.

In effect, though the model accurately predicts 83% of the variance in communication technology users, with all the indicators combined, a further examination of the variables using other analysis like Multiple Regression will further enlighten the true associations between these non-linear curves and communication technology users. What is more, as indicated in the codebook, this World Bank data set is a subset of data extracted from the primary World Bank collection of development indicators, which means there are other development indicators which were not provided and hence not captured in this analysis. It is possible that the indicators identified as important predictors of communication technology users, among the set of predictors analyzed in this project are confounded by other factors not considered in this analysis.

1-Sample

The sample included N=248 countries and regions of the world for the years 2012 and 2013. The sample is made up of national, regional and global estimates. I am only looking at the 2012 data for this sample data. I will also be looking at all the 86 variables in the World Bank data for 2012 which will help predict the communication technology users (internet + Mobile) for that year – 2012 and also help make future predictions of subsequent years.

2-Measures:

The response variable is communication technology users is a quantitative variable combined from 2 quantitative variables:

- Internet users (Per 100 people)

- Mobile cellular subscriptions (Per 100 People)

Predictors used in the algorithm are all quantitative include all world bank indicators I will use Lasso Regression model; Lasso Regression Analysis is a shrinkage and variable selection method for linear regression models. The goal of lasso regression is to obtain the subset of predictors that minimizes prediction error for a quantitative response variable.

Below is a description for all used predictors:

3-Analyses:                                    

Capstone Project Title :

Prediction of modern communication technology users (internet +Mobile) in the world countries.

Introduction to the Research Question:

The purpose of this study is to identify the best predictors for modern communications users (internet and mobile users) around the countries of the world from multiple World Bank development indicators such as:

Adjusted net national income per capita, Foreign direct investment, Access to electricity, Population density, Population growth, Population Age (0-14), Population Age (15-64), Secure internet Services, Urban Population, Adjusted net national income….

 As  an engineer who has a great interest in the modern communication technology and its rapid growth in the world countries and the work of the World Bank in helping the development of countries and increasing people communication around the world, I will like to find out about the best development factors that will lead to an increase or decrease in the using modern communication tools like internet and mobiles in the world Countries.

Such knowledge will enable countries, governments and stakeholders (internet and mobile companies) to make informed decisions and focus on the factors that will help to increase using the modern communication tools without wasting much time and resources on indicators that do not count much. This, will in effect, help drive forward the development of each country and increase the communication between individuals in different nations.

Sample

The sample included N=248 countries and regions of the world for the years 2012 and 2013. The sample is made up of national, regional and global estimates. I am only looking at the 2012 data for this sample data. I will also be looking at all the 86 variables in the World Bank data for 2012 which will help predict the communication technology users (internet + Mobile) for that year – 2012 and also help make future predictions of subsequent years, such as the year, 2013

Measures:

The response variable is communication technology users is a quantitative variable combined from 2 quantitative variables:

-  Internet users (Per 100 people)

-  Mobile cellular subscriptions (Per 100 People)

For this week I will continue with gapminder dataset to identify the impact of similar groups of variables on Life expectancy using K-Means cluster analysis by identifying subgroups of countries based on their similarity response to the following 6 variables: income per person incomeperperson, employment rate: employrate, HIVrate, alcohol consumption rate :alcconsumption, urban rate: urbanrate, amount of CO2 emission  co2emissions.A series of k-means cluster analyses were conducted on the training data specifying k=1-9 clusters, using Euclidean distance. The variance in the clustering variables that was accounted for by the clusters (r-square) was plotted for each of the nine cluster solutions in an elbow curve to provide guidance for choosing the number of clusters to interpret.

Import libraries and load the dataset :

subset clustering variables :

standardize predictors to have mean=0 and sd=1 :

split data into train and test sets :

k-means cluster analysis for 1-9 clusters :

Plot average distance from observations from the cluster centroid to use the Elbow Method to identify number of clusters to choose :

Interpret 4 cluster solution :

plot clusters :

Begin multiple steps to merge cluster assignment with clustering variables to examine cluster variable means by cluster.

create a unique identifier variable from the index for the cluster training data to merge with the cluster assignment variable:

create a list that has the new index variable:

create a list of cluster assignments :

combine index variable list with cluster assignment list into a dictionary :

convert newlist dictionary to a dataframe :

rename the cluster assignment column :

Repeat the same for the cluster assignment variable.

create a unique identifier variable from the index for the cluster assignment dataframe to merge with cluster training data :

merge the cluster assignment dataframe with the cluster training variable dataframe by the index variable :

cluster frequencies :

Finally calculate clustering variable means by cluster :

validate clusters in training data by examining cluster differences in lifeexpectancy using ANOVA .

First have to merge lifeexpectancy with clustering variables and cluster assignment data:

split lifeexpectancy data into train and test sets :

Means for lifeexpectancy by cluster :

standard deviations for lifeexpectancy by cluster :

The elbow curve was inconclusive, suggesting that the 2, 4,5, 6,7 and 8-cluster solutions might be interpreted. The results above are for an interpretation of the 4-cluster solution.

Lasso Regression Analysis is a shrinkage and variable selection method for linear regression models. The goal of lasso regression is to obtain the subset of predictors that minimizes prediction error for a quantitative response variable. The lasso does this by imposing a constraint on the model parameters that causes regression coefficients for some variables to shrink toward zero. Variables with a regression coefficient equal to zero after the shrinkage process are excluded from the model. Variables with non-zero regression coefficients variables are most strongly associated with the response variable. Explanatory variables can be either quantitative, categorical or both.

For this week example I am still using Gapminder dataset, my response variable is Life expectancy lifeexpectancy and my explanatory variables are:

incomeperperson, employrate, HIVrate, alcconsumption, urbanrate, co2emissions, polityscore, relectricperperson, suicideper100th, oilperperson.

The below code using for import libraries and load data and convert variables to numeric:

select predictor variables and target variable as separate data sets :

standardize predictors to have mean=0 and sd=1 :

split data into train and test sets: 

specify the lasso regression model:

print variable names and regression coefficients:

plot coefficient progression:

plot mean square error for each fold:

MSE from training and test data:

R-square from training and test data:

Data were randomly split into a training set that included 70% of the observations (N=39) and a test set that included 30% of the observations (N=17). The least angle regression algorithm with k=10 fold cross validation was used to estimate the lasso regression model in the training set, and the model was validated using the test set. The change in the cross-validation average (mean) squared error at each step was used to identify the best subset of predictor variables.

Of the 10 predictor variables, 5 were retained in the model. During the estimation process, income per person and Urban rate were most strongly associated with life expectancy.

For this week I will continue with the same gapminder dataset to build a random forest and explore the relationship between my response variable life expectancy lifeexpectancy and explanatory variables: incomeperperson, employrate, HIVrate, alcconsumption, urbanrate.

The code used in random forest is mostly same that we used to generate the decision tree.

The below code showing import necessary libraries and reding data from gapminder dataset :

The below code binned the response variable to 2 categories 

The below code is using to clean the data:

Split into training and testing sets: 

Build model on training data 

Display the confusion matrix 

Display the accuracy score 

fit an Extra Trees model to the data :

display the relative importance of each attribute:

Running a different number of trees and see the effect of that on the accuracy of the prediction :

Random forest analysis was performed to evaluate the importance of a series of explanatory variables in predicting a binary, categorical response variable. The following explanatory variables were included as possible contributors to a random forest evaluating Life expectancy (my response variable), incomeperperson, employrate, HIVrate, alcconsumption, urbanrate. The explanatory variables with the highest relative importance scores were incomeperperson 36%, urbanrate 24% then employrate 15%, and the less importance scores were: alcconsumption 12%, HIVrate 9%.

For this week I will explore the machine learning technique using classification tree to help analyzing the association between my response variable life expectancy lifeexpectancy and explanatory variables: incomeperperson, employrate, HIVrate, alcconsumption, urbanrate  in gapminder dataset.

This technique requires a binary categorical response variable with 2 levels so I will binned my response variable lifeexpectancy to 2 levels high life expectancy and low life expectancy using median point.

The below code showing import necessary libraries and reding data from gapminder data set:

The below code binned the response variable to 2 categories:

The below code is using to clean the data:

Split into training and testing sets:

Build model on training data:

Display the confusion matrix:

Display the accuracy score:

Display the decision tree: 

For this week I will examine the association between alcohol dependence and manic disorder and the potential confounding variable is Major depression so I will start with this research question:

Is the Manic disorder, and Major depression associated with alcohol dependence among current young adult consumers?

My hypothesis there are a strong association between manic disorder, Major depression and alcohol dependence.

My explanatory variable is Manic disorder NMANDXLIFE categorical with 2 levels: No=0 and yes =1.

The response variable is alcohol dependence ALCABDEP12DX is categorical with 3 levels I will use the below code to convert it to 2 levels:

The potential confounding variable is Major depression MAJORDEPLIFE is categorical variable with 2 levels No =0 and Yes =1.

First, I will test the association between alcohol dependence and Manic disorder using Logistic Regression model by executing the below code:

The result is below table:

From the above table we notice that Number of observations is N=2786 and P=0.0001 this mean that manic disorder and alcohol dependence are significantly associated and this support my hypothesis. The odds of having manic disorder is 4.12 times higher for observations with alcohol dependence than observations without alcohol dependence (OR=4.12,95% CI=3.05-5.58, P=0.0001).

Now I will examine if Major depression is confounding Manic disorder by adding MAJORDEPLIFE to explanatory variables list as below code:

The results in the below table:

From the table above we notice that manic disorder and major depression significantly associated with alcohol dependence for manic disorder P=0.0001 and the odds OR=3.44 and 95% CI=3.05-5.58 ,This mean that odds of having manic disorder and major depression is 3.44 times higher for observations with alcohol dependence than observations without alcohol dependence .

Summary:

For this week I will start with research question:

Is Alcohol dependence, manic disorder, Age and Sex associated with the alcohol drinking Level among current young adult consumers?

The primary explanatory variable is Alcohol Dependence ALCABDEP12DX is categorical variable I will make some data management to convert it to 2 levels using below code:  

The confounding variables are:

- Manic disorder NMANDXLIFE categorical 2 level 0 No depression and 1 yes

- SEX categorical variable 2 levels 1 Male and 2 Female

- AGE is quantitative I will be centered it using below code:

The response variable NUMDRIMO_EST is number of drinks per month we calculated in the course Data analysis Tools as the following using the below code:

- Regression Model with only Number of drinks per month NUMDRIMO_EST and Alcohol dependence ALCABDEP12DX:

From above result we notice that P<0.0001 and this mean there are significant association between Alcohol dependence and level of alcohol consumed and R=0.12(12%) indicating that linear association of Alcohol dependence is capturing only 12% of the variability in drinking level. The coefficient regression parameter for the alcohol dependence ALCABDEP12DX is (9.9) this mean that alcohol dependence positively associated with level of drinking.

- Adding Manic dis order to Alcohol dependence ALCABDEP12DX:

From the result above we notice that P-Value for the Alcohol dependence ALCABDEP12DX is less than 0.001 and still significantly associated with level of drinking after controlling for manic disorder and we notice that P-value for the manic disorder is 0.83 too large and this mean that manic disorder not associated significantly with number of drinking per month and we cant reject the null hypothesis .Also we notice that R-Squared is still 12% this mean that adding manic disorder didn’t affect the variability of the drinking level

- Adding age_c and SEX to the Regression model:

After adding age_c and SEX to the model we notice from the result above that alcohol dependence is still significantly associated with number of drinks per month were the p still less than 0.0001 and also we notice that ALCABDEP12DX and ag_c associated positively with drinking level but SEX is associated negatively with drinking level where it is coefficient value =-2.78 and R-square value increased from 12% to 14%.we notice also that manic disorder is still not significantly associated with drinking level where it is P=0.54>0.0001

Diagnostic Plots:

-QQ plot

using the code below:

In this regression model, the residual is the difference between the expected or predicted number of drinks, and the actual observed number of drinks per month. The easiest way to evaluate residuals, is to graph them as shown as Q-Q plot above. A Q-Q plot, plots the quantiles of the residuals that we would theoretically see if the residuals followed a normal distribution, against the quantiles for the residuals estimated from our regression model.

our regression Q-Q plot model shows that the residuals generally follow a straight line, but deviate at the lower, center and higher quantiles. This indicate that our residuals did not follow perfect normal distribution. This could mean there might be other explanatory variables that we might consider including in.

Plot of standardized residuals for all observations:

Using the code below we can plot the standardized residuals for all observations:

To evaluate the overall fit of the predicted values of the response variable to the observed values and to look for outliers, I created a plot for the standardized residuals of each observation. As we can see, most of the residuals fall between -1 and 1, but many of them fall also above 1. This indicates that we have several outliers, basically above the mean of 0. Furthermore, some of these outliers fall above 2 (extreme outliers) which suggests that the fit of the model is relatively poor and could be improved.

Additional Plot Diagnostic:

using the code below:

I choose to test the correlation of Age in drinking level. The plot in the upper right-hand corner shows the residuals for each observation at different values of age. There’s clearly look like a slightly funnel shaped pattern to the residuals, where we can see that the absolute value of the residuals are significantly larger at lower value of age. This is consistent with the other regression diagnostic plot, that indicate that this model predicts drinking level well for adults that have either high or low age. The Model is particularly good predicting drinking level for adults with low age.

Leverage Plot:

Using below code:

For this week I will use NESARC Data set to answer the research question:

Is the Manic disorder associated with the alcohol drinking quantity among current young adult consumers?

The explanatory variable Manic disorder NMANDXLIFE is categorical with two levels Where 0 is no Manic disorder and 1 there is Manic disorder. So, the explanatory variable doesn’t need any management.

The response variable NUMDRIMO_EST is number of drinks per month we calculated in the course Data analysis Tools as the following using the below code:

The results for the response variable NUMDRIMO_EST is below:

Bivariate graph for the explanatory variable:

The code below plotting the bivariate chart for the Manic disorder variable NMANDXLIFE:

 The results below:

Test a linear regression model:

The code below tests the linear regression:

Results:

From the above result table, we notice that manic disorder (Beta=2.58, p=0.00156) is significantly and positively associated with number of drinking per month.

The regression equation of this model is:

Sample:

Adults between the ages of 18 and 25 who consumed at least 1 alcoholic drink per day in the past 30 days (n=3175) were considered for the analysis from the National Epidemiologic Study of Alcohol and Related Conditions (NESARC) Dataset.  The sample is from the first wave of the survey which was conducted in 2001-2002 and represents the civilian, non-institutionalized adult population of the United States.

Procedure:

Data were collected by trained U.S. Census Bureau Field Representatives during 2001–2002 through computer-assisted personal interviews (CAPI). One adult was selected for interview in each household, and interviews were conducted in respondents’ homes following informed consent procedures.

Measures:

The explanatory variable Life time Manic disorder (i.e. those experienced in the past 12 months and prior to the past 12 months) - NMANDXLIFE was assessed using the NIAAA, Alcohol Use Disorder and Associated Disabilities Interview Schedule – DSM-IV (AUDADIS-IV) with a value of 1-yes and 0-No.

The response variable Current alcohol consumption (NUMDRIMO_EST) number of drinks per month was evaluated through the quantitative variable ranged from 1 -98 (S2AQ8B- number of drinks of any alcohol usually consumed on days when drank alcohol in last 12 months). Furthermore, created drink category based on cumulative distribution of number of drinks per month.

For this week I will continue with NESARC data set and return to our research question is alcohol consumption associated with alcohol dependence?

I will create another drinking variable reflecting how many drinks each adult drink per day as below:

-  0 non-daily drinkers

-  3 indicate who drink 1-5 drinks per day

-  8 indicate who drink 6-10 drinks per day

- 13 indicate who drink more than 10 drinks per day

Now I will perform a chi-squared test of independence to examine the association between alcohol dependence and the new variable USQAN:

Results: 

From the results above we notice large Chi-square value and very small value for P, this mean that drinking quantity and alcohol dependence are significantly associated.

 And we can also notice that the percentages increasing from 0.19 to 0.89.

 To represent these results graphically I will use the code below:

From the graph we notice the strong association between alcohol dependent and number of drinking, where alcohol proportion increase with increasing the drinking number.

Testing Potential moderator:

To test the third variable, moderate the association between alcohol dependence and quantity of drinking I will use third variable manic disorder so the question will be:

Dose manic disorder affect either the strength or direction of the relationship between drinking and alcohol dependence?

Or is drinking related to alcohol dependence for each level of this third variable?

To perform this test, I will use the code below: 

After executing the above code, the results below:

From the results above we notice that for those without manic disorder the chi-square value is big and the p value is quite small So we can say that this is a statistically significant relationship for those without manic disorder.

For those with manic dis order also we notice chi-square is small and the p value is also small 0.0026.now if we use a line graph to examine the rates of alcohol dependence by different levels of drinking using the code below:

From the graphs we notice that both the direction and the size for the relationship is similar for drinking and alcohol dependence. For those with manic disorder and for those without.

Although those with manic disorder do show higher rates of alcohol dependence at every level of drinking quantity. Still, in this case, we would say that a diagnosis of manic disorder does not

For this week assignment I will use Gapminder data set, I select the following variables to examine the relationship between them using Pearson correlation:

Incomeperperson: 2010 Gross Domestic Product per capita in constant 2000 US$. The inflation but not the differences in the cost of living between countries has been taken into account.

co2emissions: 2006 cumulative CO2 emission (metric tons), Total amount of CO2 emission in metric tons since 1751

oilperperson: 2010 oil Consumption per capita (tones per year and person)

 First, I will create a scatter plot for to show the association between Person income rate and oil per person:

Scatter plot for the person income rate and Co2 level:

- Running Pearson correlation:

Results:

From the results above we notice that person income rate and oil use rate showing a strong relationship where the r =0.54 and r2=0.54*0.54=0.2916

I will use Chi-Square independence to test the hypotheses about drinking frequency and alcohol dependence.

Is drinking frequency and alcohol dependence independent or dependent? Or are the rates of alcohol dependence equal or not equal among individuals from different drinking frequency categories?

For this analysis, I'm going to use a categorical explanatory variable with six levels.

The number of days each individual drank per month USFREQMO that I calculated in week 3 from Data analysis and visualization course, with the following categorical values:

-          1 day per month

-          2.5 day per month

-          14 day per month

-          22 day per month

-          30 day per month

 The response variable alcohol dependence ALCABDEP12DX is categorical variable with 4 categories as below:

I will use some data management to reduce this variable to 2 categories by using the code below: 

After executing the code above the results below: 

From the results we notice the count of the individuals who drank at least one drink per month is 784 without alcohol dependence and 36 with alcohol dependence, and in the other end of the table among drinking daily at 30 days a month 58 participants without alcohol dependence and 60 participants with alcohol independence.

We notice also that percentages are more meaningful by showing the percent of individuals with or without alcohol dependence within each drinking frequency category or level.

If we look at the Chi-square results, we notice that this value is too large 398.355and the P value is quite small 6.709e-84 Which clearly tells us that drinking and alcohol dependence are significantly associated.

To create the graph of percent of young adult consumers with alcohol dependence

within each drinking frequency category, I use the below code:

From the graph above we notice clearly that there is a positive linear relationship that is more days per month young adult consumes alcohol the more likely they are to have alcohol dependence. 

Run Post hoc tests for Chi-Square test:

The explanatory variable has six levels, if we reject the null hypothesis, we must perform a comparison for each pair of alcohol dependent's rates across the six drinking frequency categories, this mean we actually make 15 pair comparisons, and rather than evaluating significance at the p .05 level, we would adjust the p value to make it more difficult to reject the null hypothesis .The adjusted p value is calculated by dividing p 0.05 by the number of comparisons that we plan to make, So the new P value will be 0.05/15=0.003.

the code below showing comparison for the pair 1,2.5:

From the table above if we look at the percentages for the alcohol dependence individuals (1) we notice there is a difference between them but the question is this difference significant? So, if we look at the p value is very small 4.63e-10<0.003 this mean we will reject the null hypothesis and say there is a relationship between the drinking level and alcohol dependence in these two groups.

The below code is for the other comparisons:

The results below:

If we looking on the results above, we can notice that all groups have p values less than 0.003 except the 3 comparisons:(14,22) have p=0.891, (14,30) have P=0.579 and (22,30) have P=0.84

For this week I will continue with NESARC Study and alcohol dependence between young adults.

The research question: Is the Manic disorder associated with the drinking quantity among current young adult consumers?

The null hypothesis H0: mean number of drinks consumed per month is same for the individuals with and without manic disorder

The alternate hypothesis Ha: mean number of drinks consumed per month is different for the individuals with and without manic disorder

The explanatory variable Manic disorder NMANDXLIFE is categorical with two levels Where 0 is no Manic disorder and 1 there is Manic disorder.

The response variable NUMDRIMO_EST is number of drinks per month.

Run ANOVA test (2 levels category variable):

From the results above we notice that F=10.02 and P=0.00156<0.05

So, we can reject the Null hypotheses H0 and accept the alternate hypotheses that state mean number of drinks consumed per month is different for the individuals with and without manic disorder, in other words there is a significant association between number of drinks consumed and the manic disorder.

Now we will calculate the mean and std of drinks for the adults with and without Manic disorder per month using the code below:

From the results above we notice that the mean number of drinks per month for the adults with Manic disorder 22.57 per month with std 12.57 and for individuals without Manic disorder 20.16 with std 11.46 the deference is 2.59.

Run ANOVA test (more 2 levels category variable):

In the case where the explanatory variables represent more than two groups, a significant ANOVA does not tell us which groups are different from the others. To determine which groups are different from the others, we would need to perform a post hoc test. In this case I’m going to use the Tukey HSDT or Honestly Significant Difference Test for the post hoc test.

I will continue with alcohol dependence and see which group of individuals drink more than other or in which group ethnicity alcohol dependence more than other groups.

The explanatory variable ETHRACE2A is categorical variable with 5 categories:

1. White, Not Hispanic or Latino

2. Black, Not Hispanic or Latino

3. American Indian/Alaska Native, Not Hispanic or Latino

4. Asian/Native Hawaiian/Pacific Islander, Not Hispanic or Latino

5. Hispanic or Latino

From the above table displaying the Tukey post hoc paired comparisons we notice differences in drinking for each ethnic group , and we can see from the last column reject True mean we reject the null hypotheses and accept the alternate hypotheses, and we can see clearly that group 1 (white) drink more than group 5(Hispanic or Latino ) , and group 2(Black) drink also mor than group 5(Hispanic or Latino).

Appendix:Python script

I will continue with my research question in the first week:

Are alcohol consumption and alcohol dependence associated among the young adults who have drank in the last 12 Months?

Going back to the code book for the NESARC study there no single variable measures the alcohol dependence, instead there are many variables used as signs for the alcohol dependence I will select the following signs:

1-  Variable S2BQ1B1: EVER FIND USUAL NUMBER OF DRINKS HAD LESS EFFECT THAN BEFORE IN LAST 12 MONTHS.

2-  Variable S2BQ1B4: EVER INCREASE DRINKING BECAUSE AMOUNT FORMERLY CONSUMED NO LONGER GAVE DESIRED EFFECT IN LAST 12 MONTHS.

3-  Variable S2BQ1B6: MORE THAN ONCE TRY UNSUCCESSFULLY TO STOP OR CUT DOWN ON DRINKING In last 12 months.

The code book for the above variables can be found below:

The frequency distribution for these variables as below for the adults(18-25):

S2BQ1B1-ever find usual number of drinks had less effect than before last 12 months

    Frequency    Percent

1.0        513     16.17%

2.0       2642     83.27%

NaN         18     0.57%

S2BQ1B4-EVER INCREASE DRINKING BECAUSE AMOUNT FORMERLY CONSUMED NO LONGER GAVE DESIRED EFFECT

    Frequency    Percent

1.0        266     8.38%

2.0       2889     91.05%

NaN         18     0.57%

S2BQ1B6-MORE THAN ONCE TRY UNSUCCESSFULLY TO STOP OR CUT DOWN ON DRINKING last 12 months

    Frequency    Percent

1.0        113     3.56%

2.0       3047     96.03%

NaN         13     0.41%

Create Univariate graphs for selected variables:

I will create graphs for each variable:

- for the variable S2BQ1B1: I will recode this variable to indicate 1 for yes and 0 for NO then create the graph using the code below: 

recod4={1:1,2:0}

alco_dep_a['S2BQ1B1']=alco_dep_a['S2BQ1B1'].map(recod4)

 alco_dep_a['S2BQ1B1'] = alco_dep_a['S2BQ1B1'].astype('category')

c1=alco_dep_a['S2BQ1B1'].value_counts(sort=False,dropna=False)

print(c1)

seaborn.countplot(x='S2BQ1B1', data=alco_dep_a)

plt.xlabel('Alcohol Dependence past 12 months')

plt.title('Alcohol Dependence first sign in the Past 12 Months Among Young Adult Consumers in the NESARC Study')

- for the variable S2BQ1B4:

#univariate bar graph for S2BQ1B4

alco_dep_a['S2BQ1B4']=alco_dep_a['S2BQ1B4'].map(recod4)

 alco_dep_a['S2BQ1B4'] = alco_dep_a['S2BQ1B4'].astype('category')

c1=alco_dep_a['S2BQ1B4'].value_counts(sort=False,dropna=False)

print(c1)

seaborn.countplot(x='S2BQ1B4', data=alco_dep_a)

plt.xlabel('Alcohol Dependence past 12 months')

plt.title('Alcohol Dependence second sign in the Past 12 Months Among Young Adult Consumers in the NESARC Study') 

for the variable S2BQ1B6:

#univariate bar graph for S2BQ1B6

alco_dep_a['S2BQ1B6']=alco_dep_a['S2BQ1B6'].map(recod4)

 alco_dep_a['S2BQ1B6'] = alco_dep_a['S2BQ1B6'].astype('category')

c1=alco_dep_a['S2BQ1B6'].value_counts(sort=False,dropna=False)

print(c1)

seaborn.countplot(x='S2BQ1B6', data=alco_dep_a)

plt.xlabel('Alcohol Dependence past 12 months')

plt.title('Alcohol Dependence third sign in the Past 12 Months Among Young Adult Consumers in the NESARC Study')

for the variable NUMDRIMO_EST: secondary variable we created in the week 3 to calculate the number of drinks per month, this variable is Quantitative I used the code below to create the histogram for this variable:

#Univariate histogram for quantitative variable:  

seaborn.distplot(alco_dep_a['NUMDRIMO_EST'].dropna(), kde=False);

plt.xlabel('Number of drinks per Month')

plt.title('Estimated Number of drinks per Month among Young Adult consumers in the NESARC Study'

From the above histogram we notice that the shape is slightly skewed to the right in the left part and little skewed to the left in the right part.

Measures of the center and spread:

To determine the variability of the of the number of the drinks per month I will calculate the standard deviation stdv for this variable using the below code:

# standard deviation and other descriptive statistics for quantitative variables

print ('describe number of drinks per month')

desc1 = alco_dep_a['NUMDRIMO_EST'].describe()

print (desc1)

the result is:

describe number of drinks per month

count    3156.000000

mean       20.333016

std        11.551675

min         7.000000

25%         9.000000

50%        20.000000

75%        25.000000

max        44.000000

Name: NUMDRIMO_EST, dtype: float64

From the result above we notice that each adult drink in the mean 20 drinks per month with standard deviation std 11.55 this mean that each adult drink 20 drinks per month +/-12 drinks. We notice from this there is a big variability in the number of drinks per month.

Also notice other useful measures like MAX,MIN..and other measures.

We can calculate the median and mode from the below code:

mean1 = alco_dep_a['NUMDRIMO_EST'].mean()

mode = alco_dep_a['NUMDRIMO_EST'].mode()

med = alco_dep_a['NUMDRIMO_EST'].median()

std=alco_dep_a['NUMDRIMO_EST'].std()

min = alco_dep_a['NUMDRIMO_EST'].min()

max=alco_dep_a['NUMDRIMO_EST'].max()

desctable=pd.DataFrame({'Mean':mean1,' ':' ', 'Mode':mode,' ':' ','Median':med,' ':' ','Std':std,' ':' ','Min':min,' ':' ','Max':max})

print(round(desctable,2))

The results:

Mean     Mode  Median   Std   Min   Max

20.33     15.0    20.0     11.55   7.0    44.0

Visualizing the relationship between the variables:

To answer the research question: Are alcohol consumption and alcohol dependence associated?

We first must determent the role for each variable, for my research the number of the drinks per month NUMDRIMO_EST is explanatory variable and the Alcohol dependence represented by the three variables we selected are the Response variables

The three alcohol dependence variables are categorical variables and in each variable 2 category: 1 for yes there is dependence and 0 for No.

The type of my explanatory variable NUMDRIMO_EST is quantitative, so I must bin this variable in to categories, to achieve this I will need to some data managements, I will create a new variable to calculate the number of bottles each adult drink per month by considering the standard bottle size 750ml contain 5 drinks as below:

# calculate number of bottels consumed per month considering bottle size 750ml contain 5 drinks

alco_dep_a['BOTPERMONTH']=alco_dep_a['NUMDRIMO_EST'] / 5

c2= alco_dep_a.groupby('BOTPERMONTH').size()

print (c2)

Results:

BOTPERMONTH

1.4    436

1.6    221

1.8    161

2.0     53

3.0    660

4.0    475

5.0    528

6.0    134

8.4    360

8.8    128

Now I will create 3 groups depending on number of bottles consumed per month: 0-2,3-4,5-9 then convert this new variable to category:

alco_dep_a['BOTCATEGORY'] = pd.cut(alco_dep_a.BOTPERMONTH, [0, 2, 4, 6,9])

# change format from numeric to categorical

alco_dep_a['BOTCATEGORY'] = alco_dep_a['BOTCATEGORY'].astype('category')

 print ('describe BOTCATEGORY')

desc3 =alco_dep_a['BOTCATEGORY'].describe()

print (desc3)

Results:

describe BOTCATEGORY

count       3156

unique         4

top       (2, 4]

freq        1135

Bottol category counts

(0, 2]     871

(2, 4]    1135

(4, 6]     662

(6, 9]     488

Name: BOTCATEGORY, dtype: int64

Now we can create category to category bar chart by using the below code, I will use this code for the three variables I selected as signs of alcohol dependence,:

# bivariate bar graph C->Q for variable S2BQ1B1 EVER FIND USUAL NUMBER OF DRINKS HAD LESS EFFECT THAN BEFORE last 12 months  

alco_dep_a['S2BQ1B1'] =pd.to_numeric( alco_dep_a['S2BQ1B1'])

ax=seaborn.factorplot(x='BOTCATEGORY', y='S2BQ1B1', data=alco_dep_a, kind="bar", ci=None)

plt.xlabel('Number Of Botteles per Month')

plt.ylabel('Proportion Alcohol Dependent')

plt.title('Alcohol dependence first sign(ever find usual number of drinks had less effect than befor in last 12 months)')

-For variable S2BQ1B4:

# bivariate bar graph C->Q for variable S2BQ1B4 EVER INCREASE DRINKING BECAUSE AMOUNT FORMERLY CONSUMED NO LONGER

#GAVE DESIRED EFFECT last 12 months

alco_dep_a['S2BQ1B4'] =pd.to_numeric( alco_dep_a['S2BQ1B4'])

seaborn.factorplot(x='BOTCATEGORY', y='S2BQ1B4', data=alco_dep_a, kind="bar", ci=None)

plt.xlabel('Number Of Botteles per Month')

plt.ylabel('Proportion Alcohol Dependent')

plt.title('Alcohol dependence second sign(ever increas drinking becouse ammount consumed no longer gave desiared effect in last 12 months)' )

variable S2BQ1B6:

# bivariate bar graph C->Q for variable S2BQ1B6 MORE THAN ONCE TRY UNSUCCESSFULLY TO STOP OR CUT DOWN ON DRINKING last 12 months

alco_dep_a['S2BQ1B6'] =pd.to_numeric( alco_dep_a['S2BQ1B6'])

 seaborn.factorplot(x='BOTCATEGORY', y='S2BQ1B6', data=alco_dep_a, kind="bar", ci=None)

plt.xlabel('Number Of Botteles per Month')

plt.ylabel('Proportion Alcohol Dependent')

plt.title('Alcohol dependence third sign(more than once try unsuccessfully to stop or cut down on drinking in last 12 months)' )

Interpretation:

From the three charts above we notice strong association between amount of alcohol consumed and each alcohol dependence sign, for example in the first one we notice that adults who in start drinking (0-2) bottles per month about 5.2% alcohol dependence then increase to 17% among those consume 2-4 bottles per month and this increased to reach to about 33% among those consume 6-9 bottles per month and we can see this pattern in other signs.

I will continue this week with my selected variables and make a data management decision as follow:

1-managing valid data:

For participants who answer no to the question Drank at least 1  alcoholic drink  in the last 12 months? (variable S2AQ3), No need to answer the questions for variables S2AQ8A- How  often  drank any alcohol drink in last 12 months, And S2AQ8B- number of drinks last 12 months.

so, we need to recode this data as valid data rather to consider it as missing data, to implement this I used the following code:

alco_vars=['CONSUMER','S2AQ1','S2AQ2','S2AQ3','S2AQ8A','S2AQ8B']

for var in alco_vars:

  data[var]=pd.to_numeric(data[var],errors='coerce')

#Recover valid data for the observations who answer no for the question Drank at least 12 alcoholic drink in last 12 months?

#set valid data for the variable S2AQ8A to dummy value 11    

   data.loc[(data[alco_vars[3]]!=9) &(data[alco_vars[4]].isnull()),alco_vars[4]]=11

   #set valid data for the variable S2AQ8B to dummy value 100

   data.loc[(data[alco_vars[3]]!=9) &(data[alco_vars[5]].isnull()),alco_vars[5]]=100

Results as below:

S2AQ8A-How often drank any alcohol in last 12 months

     Frequency   Percent

1.0        1865      4.33%

2.0        1210      2.81%

3.0        2619      6.08%

4.0        2914      6.76%

5.0        3261      7.57%

6.0        3557      8.25%

7.0        2663      6.18%

8.0        1805      4.19%

9.0        3210      7.45%

10.0       3637      8.44%

11.0      16116     37.4% ------- Recovered valid data (from S2AQ3)

99.0        205      0.48%

NaN          31      0.07%-------Unknown from S2AQ3

S2AQ8B-number of drinks of any alcohol consumed on days when drank alcohol in last 12 months

      Frequency    Percent

1.0        10350     24.02%

2.0         7880     18.29%

3.0         3678      8.54%

4.0         1724       4.0%

5.0          759      1.76%

6.0         1247     2.89%

7.0          189      0.44%

8.0          275      0.64%

9.0           45       0.1%

10.0         163      0.38%

11.0           7      0.02%

12.0         246      0.57%

13.0          11      0.03%

14.0           7      0.02%

15.0          40      0.09%

16.0           3      0.01%

17.0           4      0.01%

18.0          24      0.06%

19.0           1       0.0%

20.0          14      0.03%

24.0          21      0.05%

25.0           6      0.01%

30.0           8      0.02%

32.0           1       0.0%

36.0           1       0.0%

42.0           1       0.0%

45.0           1       0.0%

48.0           1       0.0%

98.0           1       0.0%

99.0         238      0.55%

100.0      16116     37.4%------- Recovered valid data (from S2AQ3)

NaN           31     0.07%-------- Unknown from S2AQ3

I used dummy code(11) for the valid data in S2AQ8A and 100 for valid data in S2AQ8B.

2-managing missing data:

Set messing data (99) in S2AQ8A, S2AQ8B to nan:

##subset data to young adults age 18 to 25 who have drank at least 12 alcoholic drink in the last 12 months

sub1=data[(data['AGE']>=18) & (data['AGE']<=25) & (data['S2AQ2']==1)]

#make a copy of  new subsetted data alco_dep_a =sub1.copy()

 # frequency distritions on new alco_dep_a data frame print("Frequency and percentages for subset data( young adults age 18 to 25 who have drank at least 12 alcoholic drink in the last 12 months):") alco_a_vars=['AGE','S2AQ2','S2AQ8A','S2AQ8B']

for var in alco_a_vars:

   if var==alco_a_vars[0]:        print("adults age 18-25 by-{} tab".format(var))    elif var==alco_a_vars[1]:

       alco_dep_a[var]=alco_dep_a[var].replace(99,numpy.nan)        print("{} adults who drank at least 12 alcoholic drink in the last 12 months".format(var))    elif var==alco_a_vars[2]:         #recode 99 values as missing for the variable S2AQ8A:        alco_dep_a[var]=alco_dep_a[var].replace(99,numpy.nan)

       print("{}- How often drank any alcohol in last 12 months".format(var))    else:         #recode 99 values as missing for the variable S2AQ8B:        alco_dep_a[var]=alco_dep_a[var].replace(99,numpy.nan)        print("{}-Number of drinks of any alcohol consumed on days when drank alcohol in last 12 months".format(var))    c=alco_dep_a[var].value_counts(sort=False,dropna=False).sort_index(ascending=True)

   p=alco_dep_a[var].value_counts(sort=False,normalize=True,dropna=False).sort_index(ascending=True).mul(100).round(2).astype(str) + '%'    ferq_table=pd.DataFrame({'Frequency':c,' ':' ', 'Percent':p})    print(ferq_table)

Results as below:

Frequency and percentages for subset data( young adults age 18 to 25 who have drank at least 12 alcoholic drink in the last 12 months):

adults age 18-25 by-AGE tab

   Frequency    Percent

18        266      8.38%

19        358     11.28%

20        348     10.97%

21        453     14.28%

22        436     13.74%

23        443     13.96%

24        471     14.84%

25        398     12.54%

S2AQ2 adults who drank at least 12 alcoholic drink in the last 12 months

  Frequency    Percent

1       3173     100.0%

S2AQ8A- How often drank any alcohol in last 12 months

     Frequency    Percent

1.0         134      4.22%

2.0         128      4.03%

3.0         360     11.35%

4.0         475     14.97%

5.0         528     16.64%

6.0         660      20.8%

7.0         436     13.74%

8.0         221      6.97%

9.0         161     5.07%

10.0         53      1.67%

NaN          17      0.54% -------------missing data

S2AQ8B-Number of drinks of any alcohol consumed on days when drank alcohol in last 12 months

     Frequency    Percent

1.0         385     12.13%

2.0         802     25.28%

3.0         599     18.88%

4.0         397     12.51%

5.0         243      7.66%

6.0         326     10.27%

7.0          81      2.55%

8.0          89       2.8%

9.0          21      0.66%

10.0         59      1.86%

11.0          4      0.13%

12.0         66      2.08%

13.0          7      0.22%

14.0          3      0.09%

15.0         22      0.69%

16.0          1      0.03%

17.0          3      0.09%

18.0          8      0.25%

20.0          7      0.22%

24.0          8      0.25%

25.0          3     0.09%

30.0          2      0.06%

36.0          1      0.03%

42.0          1      0.03%

NaN          35       1.1%--------------missing data

3- Create secondary variables:

for variable S2AQ8A I will create a secondary NUMDRIMO_EST variable to compute number of drinks each participant consumes per month.

To achieve this, I will re-code this variable from 1-6 as below:

1-  Every day

2-  Nearly every day (5-6 times a week)

3-  3 to 4 times a week

4-  Once to 2 times a week

5-  2 to 3 times a month

6-  Once or less a month

I think this coding is better and more accurate because a person cannot remember exactly what he consumed if the period become more than month.

- recoding values for S2AQ8A into a new variable, USFREQ the new code from 1-6:

#recording values for S2AQ8A and create new variables    #recoding values for S2AQ8A into a new variable, USFREQ the new code from 1-6 print('recoding values for S2AQ8A-How often drank alcohol in last 12 months?- in to a new variable,USFREQ ,the new code from 1-6')     recode1 = {1:1,2:2,3:3,4:4,5:4,6:5,7:6,8:6,9:6,10:6} alco_dep_a['USFREQ']= alco_dep_a['S2AQ8A'].map(recode1) c1=alco_dep_a['USFREQ'].value_counts(sort=False,dropna=False).sort_index(ascending=True) p1=alco_dep_a['USFREQ'].value_counts(sort=False,normalize=True,dropna=False).sort_index(ascending=False).mul(100).round(2).astype(str) + '%' ferq_table=pd.DataFrame({'Frequency':c1,' ':' ', 'Percent':p1}) print(ferq_table)

Result:

recoding values for S2AQ8A-How often drank alcohol in last 12 months?- in to a new variable,USFREQ ,the new code from 1-6

    Frequency    Percent

1.0        134     4.22%

2.0        128     4.03%

3.0        360     11.35%

4.0       1003     31.61%

5.0        660     20.8%

6.0        871     27.45%

NaN         17     0.54%

- reverse the code values to indicate high frequency values take the high values and low frequency values take low values:

#reverse the code values to indicate high frequency values in the first and low frequency values in the last print('reverse the code values to indicate high frequency take high values and low frequency take low values ')        #recoding values for S2AQ8A into a new variable, USFREQ1 recode2 = {1: 6, 2: 5, 3: 4, 4: 3, 5: 2, 6: 1} alco_dep_a['USFREQ1']= alco_dep_a['USFREQ'].map(recode2) print(alco_dep_a['USFREQ1'].value_counts(sort=False,dropna=False).sort_index(ascending=True))

Result:

reverse the code values to indicate high frequency take high values and low frequency take low values

1.0     871

2.0     660

3.0    1003

4.0     360

5.0     128

6.0     134

NaN      17

Name: USFREQ1, dtype: int64

It is reasonable to re-code the variable in order to capture more of its quantitative features. So, I choose values that reasonably

correspond to the number of times each individual drink in a usual Month. For example, if a person drinks every day this mean he drinks 30 days in usual month, if he drinks nearly every day (5-6 times a week) he drinks 22 days in a usual month and so on:

·         Every day----------------------------------30 days a month

·         Nearly every day (5-6 times a week) --------22 day a month

·         3 to 4 times a week------------------------14 days a month

·         Once to 2 times a week----------------------5 days a month

·         2 to 3 times a month------------------------2.5 days a month

·         Once or less a month------------------------1 day a month

 The code that perform this task is below:

#recoding values for S2AQ8A into a new variable, USFREQMO print('Create secondary variable USFREQMO to indicate how often you drank last 12 months as quantitative value per month')         recode3= {1: 30, 2: 22, 3: 14, 4: 5, 5: 2.5, 6: 1} alco_dep_a['USFREQMO']= alco_dep_a['USFREQ'].map(recode3) c1=alco_dep_a['USFREQMO'].value_counts(sort=False,dropna=False).sort_index(ascending=True) p1=alco_dep_a['USFREQMO'].value_counts(sort=False,normalize=True,dropna=False).sort_index(ascending=False).mul(100).round(2).astype(str) + '%' ferq_table=pd.DataFrame({'Frequency':c1,' ':' ', 'Percent':p1}) print(ferq_table)

Results:

Create secondary variable USFREQMO to indicate how often you drank last 12 months as quantitative value per month

     Frequency    Percent

1.0         871     27.45%

2.5         660      20.8%

5.0        1003     31.61%

14.0        360     11.35%

22.0        128      4.03%

30.0        134      4.22%

NaN          17      0.54%

From this table we notice that most adults (31.61%) drink 5 day per month.

- Create the secondary variable: NUMDRIMO_EST and display the firs 25 rows:

#secondary variable multiplying the number of days drank/month and the approx number of drinks drank/day alco_dep_a['NUMDRIMO_EST']=alco_dep_a['USFREQMO'] * alco_dep_a['S2AQ8A'] sub3=alco_dep_a[['IDNUM','S2AQ8A','USFREQMO','NUMDRIMO_EST']] print(sub3.head(25))

#calculate the Frequency and precentages for the secondary variable NUMDRIMO_EST

c1=alco_dep_a['NUMDRIMO_EST'].value_counts(sort=False,dropna=False).sort_index(ascending=True) p1=alco_dep_a['NUMDRIMO_EST'].value_counts(sort=False,normalize=True,dropna=False).sort_index(ascending=False).mul(100).round(2).astype(str) + '%' ferq_table=pd.DataFrame({'Frequency':c1,' ':' ', 'Percent':p1}) print(ferq_table)

Results:

  IDNUM  S2AQ8A  USFREQMO NUMDRIMO_EST

9       10     8.0       1.0           8.0

19     20     5.0       5.0          25.0

30     31     7.0       1.0           7.0

39     40     8.0       1.0           8.0

45     46     5.0       5.0          25.0

59     60     9.0       1.0           9.0

76     77     9.0       1.0           9.0

80     81     5.0       5.0          25.0

88     89     9.0       1.0           9.0

93     94     4.0       5.0          20.0

102    103     3.0     14.0          42.0

104   105     5.0       5.0          25.0

120   121     6.0       2.5          15.0

121   122     6.0       2.5          15.0

124   125     7.0       1.0           7.0

149   150     7.0       1.0           7.0

154   155     6.0       2.5          15.0

173   174     8.0       1.0           8.0

177   178     4.0       5.0          20.0

183   184     7.0       1.0           7.0

187   188     7.0       1.0           7.0

209   210     5.0       5.0          25.0

269   270     8.0       1.0           8.0

278   279     1.0      30.0          30.0

291   292     6.0       2.5          15.0

Frequency and precentages for the NUMDRIMO_EST      Frequency    Percent 7.0         436     13.74% 8.0         221      6.97% 9.0         161      5.07% 10.0         53      1.67% 15.0        660      20.8% 20.0        475     14.97% 25.0        528     16.64% 30.0        134      4.22% 42.0        360     11.35% 44.0        128      4.03% NaN          17      0.54%

we notice that most adults(20.8%) drink 15 drinks per month

4- Divide the AGE variable to 3 groups:18-20,21-22,23-25:

# categorize quantitative variable based on customized splits using cut function # splits into 3 groups (18-20, 21-22, 23-25) print ('AGE - 3 categories - quartiles') alco_dep_a['AGEGROUP3']=pd.cut(alco_dep_a.AGE,[17,20,22,25]) c=alco_dep_a['AGEGROUP3'].value_counts(sort=False, dropna=True) print(c) #crosstabs evaluating which ages were put into which AGEGROUP3 print('crosstabs evaluating which ages were put into which AGEGROUP3:') print (pd.crosstab(alco_dep_a['AGEGROUP3'],alco_dep_a['AGE']))

#frequency distribution for AGEGROUP3 print('frequency distribution and percentages for AGEGROUP3') c=alco_dep_a['AGEGROUP3'].value_counts(sort=False, dropna=True) p=alco_dep_a['AGEGROUP3'].value_counts(sort=False,normalize=True,dropna=True).sort_index(ascending=False).mul(100).round(2).astype(str) + '%' ferq_table=pd.DataFrame({'Frequency':c,' ':' ', 'Percent':p}) print(ferq_table)

Results:

 AGE - 3 categories - quartiles

(17, 20]     972

(20, 22]     889

(22, 25]    1312

Name: AGEGROUP3, dtype: int64

crosstabs evaluating which ages were put into which AGEGROUP3:

AGE         18   19   20   21   22   23   24   25

AGEGROUP3                                        

(17, 20]   266  358 348    0    0   0    0    0

(20, 22]     0    0   0  453  436   0    0    0

(22, 25]     0    0   0    0    0 443  471  398

frequency distribution and percentages for AGEGROUP3

         Frequency    Percent

(17, 20]        972     30.63%

(20, 22]        889     28.02%

(22, 25]       1312     41.35%

From this table we notice that most adults 41.35% in the third group (22-25).

Summary of Frequency Distributions:

for variable S2AQ8A I created a secondary NUMDRIMO_EST variable to compute number of drinks each participant consumes per month.

To achieve this, I  re-coded  this variable from 1-6 as below:

1- Every day

2- Nearly every day (5-6 times a week)

3- 3 to 4 times a week

4- Once to 2 times a week

5- 2 to 3 times a month

6- Once or less a month

I think this coding is better and more accurate because a person cannot remember exactly what he consumed if the period become more than month.

we notice from the results above that most adults (20.8%) drink 15 drinks per month(variable NUMDRIMO_EST).

- from the age groups we notice from the results above that most adults(41.35%) falls in the group 22-25).

Appendix:(full python code):

#import pandas,Numpy library import pandas as pd import numpy data=pd.read_csv('nesarc_pds.csv',low_memory=False,encoding='utf-8',na_values=' ') print('number of rows:') print(len(data))          # number of rows print('number of variables (columns):') print(len(data.columns))  # number of variables (columns)

#counts and percentages (i.e. frequency distributions) for each variable

alco_vars=['CONSUMER','S2AQ1','S2AQ2','S2AQ3','S2AQ8A','S2AQ8B']

for var in alco_vars:    data[var]=pd.to_numeric(data[var],errors='coerce') #Recover valid data for the observations who answer no for the question Drank at least 12 alcoholic drink in last 12 months? #set valid data for the variable S2AQ8A to dummy value 11        data.loc[(data[alco_vars[3]]!=9) &(data[alco_vars[4]].isnull()),alco_vars[4]]=11

   #set valid data for the variable S2AQ8B to dummy value 100    data.loc[(data[alco_vars[3]]!=9) &(data[alco_vars[5]].isnull()),alco_vars[5]]=100          if var==alco_vars[0]:        print("{}-Drinking status,Current drinker=1, EX-drinker=2, Lifetime Abstainer=3".format(var))

   elif var==alco_vars[1] :        print("{}-Drank at least 1 alcoholic drink in life ,yes=1, No=2".format(var))

   elif var==alco_vars[2]:      print("{}-Drank at least 12 alcoholic drinks in last 12 months ,yes=1, No=2,Unknown=9".format(var))

   elif var==alco_vars[3]:            print("{}-Drank at least 1 alcoholic drink in last 12 months ,yes=1, No=2,Unknown=9".format(var))

   elif var==alco_vars[4]:        print("{}-How often drank any alcohol in last 12 months".format(var))

   else:            print("{}- drinks of any alcohol consumed on days when drank alcohol in last 12 months ".format(var))

   c = data[var].value_counts(sort=False,dropna=False).sort_index(ascending=False)

   p=data[ var].value_counts(sort=False,normalize=True,dropna=False).sort_index(ascending=True).astype(float).mul(100).round(2).astype(str)+'%'

   ferq_table=pd.DataFrame({'Frequency':c,' ':' ', 'Percent':p})

   print(ferq_table)

##subset data to young adults age 18 to 25 who have drank at least 12 alcoholic drink in the last 12 months

sub1=data[(data['AGE']>=18) & (data['AGE']<=25) & (data['S2AQ2']==1)]

#make a copy of  new subsetted data alco_dep_a =sub1.copy()

 # frequency distritions on new alco_dep_a data frame print("Frequency and percentages for subset data( young adults age 18 to 25 who have drank at least 12 alcoholic drink in the last 12 months):") alco_a_vars=['AGE','S2AQ2','S2AQ8A','S2AQ8B']

for var in alco_a_vars:

   if var==alco_a_vars[0]:        print("adults age 18-25 by-{} tab".format(var))    elif var==alco_a_vars[1]:

       alco_dep_a[var]=alco_dep_a[var].replace(99,numpy.nan)        print("{} adults who drank at least 12 alcoholic drink in the last 12 months".format(var))    elif var==alco_a_vars[2]:         #recode 99 values as missing for the variable S2AQ8A:        alco_dep_a[var]=alco_dep_a[var].replace(99,numpy.nan)

       print("{}- How often drank any alcohol in last 12 months".format(var))    else:         #recode 99 values as missing for the variable S2AQ8B:        alco_dep_a[var]=alco_dep_a[var].replace(99,numpy.nan)        print("{}-Number of drinks of any alcohol consumed on days when drank alcohol in last 12 months".format(var))    c=alco_dep_a[var].value_counts(sort=False,dropna=False).sort_index(ascending=True)

   p=alco_dep_a[var].value_counts(sort=False,normalize=True,dropna=False).sort_index(ascending=True).mul(100).round(2).astype(str) + '%'    ferq_table=pd.DataFrame({'Frequency':c,' ':' ', 'Percent':p})    print(ferq_table)

   #recording values for S2AQ8A and create new variables    #recoding values for S2AQ8A into a new variable, USFREQ the new code from 1-6 print('recoding values for S2AQ8A-How often drank alcohol in last 12 months?- in to a new variable,USFREQ ,the new code from 1-6')     recode1 = {1:1,2:2,3:3,4:4,5:4,6:5,7:6,8:6,9:6,10:6} alco_dep_a['USFREQ']= alco_dep_a['S2AQ8A'].map(recode1) c1=alco_dep_a['USFREQ'].value_counts(sort=False,dropna=False).sort_index(ascending=True) p1=alco_dep_a['USFREQ'].value_counts(sort=False,normalize=True,dropna=False).sort_index(ascending=False).mul(100).round(2).astype(str) + '%' ferq_table=pd.DataFrame({'Frequency':c1,' ':' ', 'Percent':p1}) print(ferq_table) #reverse the code values to indicate high frequency values in the first and low frequency values in the last print('reverse the code values to indicate high frequency take high values and low frequency take low values ')        #recoding values for S2AQ8A into a new variable, USFREQ1 recode2 = {1: 6, 2: 5, 3: 4, 4: 3, 5: 2, 6: 1} alco_dep_a['USFREQ1']= alco_dep_a['USFREQ'].map(recode2) print(alco_dep_a['USFREQ1'].value_counts(sort=False,dropna=False).sort_index(ascending=True))        #recoding values for S2AQ8A into a new variable, USFREQMO print('Create secondary variable USFREQMO to indicate how often you drank last 12 months as quantitative value per month')         recode3= {1: 30, 2: 22, 3: 14, 4: 5, 5: 2.5, 6: 1} alco_dep_a['USFREQMO']= alco_dep_a['USFREQ'].map(recode3) c1=alco_dep_a['USFREQMO'].value_counts(sort=False,dropna=False).sort_index(ascending=True) p1=alco_dep_a['USFREQMO'].value_counts(sort=False,normalize=True,dropna=False).sort_index(ascending=False).mul(100).round(2).astype(str) + '%' ferq_table=pd.DataFrame({'Frequency':c1,' ':' ', 'Percent':p1}) print(ferq_table)

#secondary variable multiplying the number of days drank/month and the approx number of drinks drank/day alco_dep_a['NUMDRIMO_EST']=alco_dep_a['USFREQMO'] * alco_dep_a['S2AQ8A'] sub3=alco_dep_a[['IDNUM','S2AQ8A','USFREQMO','NUMDRIMO_EST']] print(sub3.head(25))

# categorize quantitative variable based on customized splits using cut function # splits into 3 groups (18-20, 21-22, 23-25) print ('AGE - 3 categories - quartiles') alco_dep_a['AGEGROUP3']=pd.cut(alco_dep_a.AGE,[17,20,22,25]) c=alco_dep_a['AGEGROUP3'].value_counts(sort=False, dropna=True) print(c) #crosstabs evaluating which ages were put into which AGEGROUP3 print('crosstabs evaluating which ages were put into which AGEGROUP3:') print (pd.crosstab(alco_dep_a['AGEGROUP3'],alco_dep_a['AGE']))

#frequency distribution for AGEGROUP3 print('frequency distribution and percentages for AGEGROUP3') c=alco_dep_a['AGEGROUP3'].value_counts(sort=False, dropna=True) p=alco_dep_a['AGEGROUP3'].value_counts(sort=False,normalize=True,dropna=True).sort_index(ascending=False).mul(100).round(2).astype(str) + '%' ferq_table=pd.DataFrame({'Frequency':c,' ':' ', 'Percent':p}) print(ferq_table)

#upper-case all DataFrame column names - place afer code for loading data aboave data.columns = map(str.upper, data.columns)