Education Methodology and Statistics

Rafferty (1995) in Table 6 outlines changes in BIC as rules of thumb. 0-2 points difference on BIC means there's weak evidence of improvement.

2-6 points suggest positive evidence of improvement

6-10 Strong evidence of improvement

Maas, C. J., & Hox, J. J. (2004). The influence of violations of assumptions on multilevel parameter estimates and their standard errors. Computational statistics & data analysis, 46(3), 427-440.

THIS PAPER

Materials for Quantile Regression workshop from AERA 2018, including data files, R and SAS code, slides, and additional readings. If you want to read more about the QUANTREG package in R, instructions and detailed information are freely available here: 

When fitting longitudinal CFA-based models (say one latent factor measured twice using the same indicators), it is customary to correlate the residual variances at time 1 with the residual variances from the same model at time 2. I often need a citation for this practice so here we go: 

“Measurement residual variance contains variance due to measurement error as well as variance unique to an indicator but not common to the other indicators of the factor. Because measurement error is random, by definition, the measurement error component of the variance of the measurement residual cannot be correlated with other factors. The Specific variance component on the other hand... may have auto-correlation over time”

“if not taken into account, the stability of the model may be overestimated...... Although the inclusion of correlated measurement residuals.. could be decided on the basis of significance tests, they are generally included in longitudinal models a priori (Mheaton, Muthen, Alwin, & Summers, 1977). Except for the cost of the degrees of freedom, there is typically no harm in including these estimates. 

Sometimes I need to work with a dataset that has multiple rows representing each person; sometimes called a univariate dataset. In a typical univariate dataset in education research, people (usually students) are assessed multiple times; for example the beginning and end of the school year. So that data set might look something like this:

ID       SCORE       TIME

1            12            1

1            18            2

2            13            1

2            15            2

The first column represents the ID number, the second represents the score they got, and the third represents the time at which the kid was assessed (in this case, either 1 or 2). So we can tell pretty easily that the score of 12 belongs to kid 1 and was given at time 1.

Today, I had a dataset like this, but kids were measured anywhere from 1 to 5 times. My challenge was that there was no “time” variable, just a date. So I knew which came first, but I didn’t know whether a kid’s score came from the 1st time we talked to them or the 3rd (etc). So I had to create one.  

Today I am working in SPSS. I decided to create a new variable called count. Then used the “lag” function in SPSS to re-compute count if the id on the current row is equal to the id on the previous row. I used the lag function again to tell SPSS to add 1 to whatever was in the previous row on the variable count. It looked like this (variable names are in bold):

compute count = 1.

IF id = lag(id) count = 1+lag(count).

execute.

  And results in:

  ID       SCORE       COUNT

1            12            1

1            18            2

1            13            3

2            15            1

2            15            2

2            20            3

2            21            4

3            10            1

3            13            2

Millsap (2007)

Yo. If your model doesn’t fit perfectly it might be because you already trimmed it. And that’s ok. In fact, it’s expected. 

Rex Kline (on p194) giving us all a way to deal with marginal fit statistics. Cites below the jump.

Kline suggests that we instead start providing a matrix of residual correlations in any latent factor model. His rule of thumb is that any residual correlation > .10 suggests poor explanation of the observed correlation between that pair of indicators. 

Kline, R. B. (2013). Exploratory and confirmatory factor analysis. Applied quantitative analysis in the social sciences, 171-207. 

Kline cites this with (the apparently extremely hysterical) Millsap, 2007