Outlier Research & Evaluation

Greetings my name is Maurice Samuels and I’m a Lead Evaluation and Research Associate at Outlier. A recent trend in evaluation is to increase the number of evaluators that can conduct culturally competent evaluations.  Our group recently hosted an American Evaluation Association Graduate Education Diversity Intern (GEDI). This was a wonderful opportunity for me and my colleagues to to influence the development of a new member of the field.  Our intern Crystal Coker had experience conducting an evaluation and more importantly we supported her thinking about and practice of cultural competence in evaluation. Below are several helpful tips to introduce evaluators to cultural competence in evaluation: 

Tip 1. Immerse yourself in the literature about cultural competence: It is important to have an understanding of evaluation frameworks and approaches (e.g., culturally responsive evaluation, contextually responsive evaluation, cross-cultural evaluation) that are sensitized to culture and context in order to stimulate thinking about the role of culture in evaluation. Equally important is to have a comprehensive understanding of how culture has been characterized in other fields such as anthropology, health, and social work.  This is particularly helpful due to the fact that culture can be understood in many different ways. For articles on the role of culture in evaluation check out these publications from the Center for Culturally Responsive Evaluation and Assessment.

Tip 2.  Use resources available through the American Evaluation Association (AEA) – The AEA has several Topical Interest Groups (TIGs) that have an explicit commitment to culture and diversity. In addition, the organization has a published statement and guidelines on cultural competence in evaluation. Specifically, commit yourself to AEA’s Cultural Competence in Evaluation and review their Introduction to the Cultural Readings of The Program Evaluation Standards and the Guiding Principles for evaluators. Finally, the AEA365 blog is an excellent way to become acquainted with how members of AEA are using cultural competence in their evaluation practice.

Tip 3. Create opportunities to engage in dialogue about cultural competence – Connecting with people in the field with similar interest and those who are enacting cultural competence is important to making the practice concrete. Further, this encourages open conversations about culture, which helps to refine ones notions of cultural competence and provides multiple perspectives to draw upon.

Tip 4. Encourage strong field work practices and self-reflection – It is important for evaluators to build strong relationships with clients and stakeholders, understand the context of the program and the surrounding community, and give back to the community in tangible ways such as volunteering at the program or attending program sponsored events that are not related to the evaluation.  As for self-reflection, it is important to document and share decisions made and assumptions when in the field through journaling and debriefing with a colleague.           

Sandra Holt, Research and Evaluation Associate

Computer science is all the buzz in K-12 education right now. Quite a few initiatives have been set forth with the goal of bringing computer science education to public schools across the country. Cultivating more computer scientists is crucial for the future of our nation’s economy. According to U. S. Bureau of Labor Statistics, IT is one of the fastest-growing sectors and will add over 1.4 million job openings by 2020. More than two-thirds of these jobs will go unfilled due to a lack of college graduates with computer-related degrees! With that startling statistic in mind, it’s obvious why many organizations have made it their mission to make computer science education a priority.

Some of the initiatives that I’ve taken interest in are those with the goal of engaging more women and minority students in computer science. As a social worker turned education researcher and evaluator, I recognize the importance of engaging underrepresented groups and promoting equality in computing. There’s a lot of value in diversity. It is a well-known and documented fact that there is a drastic lack of representation on behalf of women and minorities in the field of computing. Currently, the statistics for women in computer science look something like this:

Getting more girls and students of color interested and excited about computer science is a need, but who is doing something about it? Below are a few of organizations that have made this a priority. You may have heard of them in the news or on social media!

• Black Girls Code is a nonprofit organization committed to empowering girls of color to pursue STEM fields through exposure to computing and technology.

• Code.org is a nonprofit organization dedicated to increasing the participation in computer science, especially by women and minorities, with their “Hour of Code” initiative and work to bring computer science courses to K-12 schools across the nation.

• Girls Who Code is a nonprofit organization working to educate and inspire girls to pursue opportunities in computing fields.

• “Made with Code” is a campaign recently launched by Google. It is dedicating $50 million dollars over three years to get young women interested in coding.

• The National Center for Women and Information Technology (NCWIT) is a nonprofit committed to help recruit, retain and advance women in the field of information technology. Here’s a great resource from NCWIT on how to engage underrepresented students in computing.

We must make computer science education more available in and out of schools, especially in large urban districts, and break down the stereotypes about computer scientists. It is important that young girls and minorities see strong role models in the field of computer science. These organizations have definitely gotten the conversation started and have made great progress, but there’s still a lot of work to do.

What else can be done to help get underrepresented students interested and excited about computer science? We’d love to know what you think!

July 9, 2014

Sarah Wille, Researcher

In 2011, one of my first projects at Outlier Research & Evaluation was a study of inclusive STEM schools in Ohio that are part of Ohio STEM Learning Network's (OSLN) Platform Schools Initiative. Part of our research plan included my favorite type of data collection: facilitating student focus groups. These focus groups were an essential piece of our work, enabling us to hear the student perspective on STEM schools.

I worked closely with two of the schools from 2011 - 2013. While all of the schools were initially designed around a set of common design principles, the schools’ contexts and histories differed. Yet during focus group data collection, students at these two schools seemed to be saying some similar things. I was struck by how frequently students talked about their relationships with teachers and students in their school, how they felt supported, and their descriptions of school as a comfortable space to learn.

In Winter 2014, I began qualitative analysis of the student focus group data to systematically examine how students perceived as their overall experience and membership in their STEM schools. I analyzed all three years of the high school student focus group transcripts and found that many students across the OSLN schools emphasized relationships in their school and the learning supports available to them.

Although students described diverse experiences in their schools, five common themes emerged across all four Ohio STEM Learning Network high schools:

1. The school is a family. Students reported that adults and other students in the school care about and support each other.

2. Students have decision-making power in the school. Students generally feel that their voices are heard and many students reported that they feel respected.

3. The school is a comfortable and safe space for all learners. Students indicate that teachers help students academically through personalized instruction and support, and teachers make time to help students.

4. Students know they are preparing for their future. Students realize that their school is academically challenging and provides college and career preparation resources. They understand that teachers believe and expect students to learn and that students are responsible for their learning.

5. It's a different kind of school.

When STEM schools discuss their strategies, there is a lot of dialogue around subject content, pedagogy, preparation for STEM majors, and supporting the STEM pipeline. Yet the data indicate that the student experience in STEM schools also provide students with a sense of connectedness in their schools. In general, students that participated in our focus groups thought their school was different and beneficial to them not just because of the STEM content, but because of the learning opportunities and supports available to them, their interactions with adults in the school, and the value placed on them and their perspectives.

STEM schools can contribute to learning in many ways, from rigorous content and pedagogy to important psychological supports, community, and belonging.

Courtney Heppner, Associate Project Director

One of the only eNewsletters that I read religiously is AEA365. Even on the days when my inbox is overflowing with a backlog of email, I still find time to skim over the “Rad Resources” section. True to name, the resources I have discovered in these emails are totally rad. Rad Resources suggest websites and tips that have been particularly helpful in learning new ways to communicate evaluation findings. Gone are the days of the thick evaluation report, full of text, boring charts and graphs. The future of communicating evaluation and research is in online reporting, data visualization, and infographics. In that vein, I thought I would share my own list of evaluator rad resources (both internal and external), a la AEA365-style:

Stephanie Evergreen: Stephanie is a great resource for utilizing data visualization in your evaluation work. Stephanie ran one of the best webinars I ever attended on Power Point. She provided PowerPoint slide design guidelines that encourage you to use engaging graphics and few words to get a point across. Stephanie’s book is another great resource for creating well-designed visuals and best practices in communicating findings.

Kylie Hutchinson: Like Stephanie, Kylie hosts webinars and workshops on evaluation. She also hosts the fantastic  “Adventures in Evaluation” podcast. 

Ann Emery: Ann challenges the status quo of presenting information.

FreshSpectrum: Chris Lysy helps evaluators translate findings through data visualization and infographics. He has some great cartoons!

Online and Interactive Evaluation Reports: My colleague Sarah Rand and I wrote an AEA365 post last year about lessons learned from developing online reports for clients. Take a look at a recent online report Outlier developed for the Boston Schoolyard Initiative Evaluation.

Logic Models: My colleague Maurice Samuels wrote a great blog post in April about logic modeling in our evaluation work, and even included some rad resource links of his own. The Kellogg Foundation Logic Model Development Guide is particularly helpful.

Heather King, Associate Project Director

You may have heard of the Hour of Code, but have you heard of the “Hour of Implementation”? You might have come across a STEM network, but have you heard of “implementation networks”? No?

Well, you haven’t heard of them because they don’t exist. But they should. Implementation research may not have the exposure or appeal that computer science education or STEM education are currently enjoying, but it’s very important nonetheless. Let’s walk through a simple example.

I’ve abandoned my cookbooks for the Internet and all it can offer, as many of us probably have. Among my favorite things about finding recipes online are the reviews;  looking for high ratings is a great way to sort through seas of recipes and find the ones worth investing time in.

If you’ve ever actually read reviews on a website like allrecipes.com, however, you’ll start to notice something interesting. Take a recipe for coleslaw; it’s very simple, just a cabbage, a carrot, mayonnaise, sugar, vinegar, dried onion, salt, and pepper. But many of the reviews––especially the 5-star ones––didn’t actually follow the recipe, and they say so in their review. One reviewer added horseradish, several added additional ingredients such as dried cranberries, apples, or nuts, and one even used pickle juice instead of vinegar. I’ve learned to read the reviews very carefully because those adaptations can make a real difference, and I want to know whether I should also adapt the recipe to get a 5-star dish.

This is a perfect example of why implementation research matters. Personal taste aside, if I assumed that reviewers on allrecipes.com followed the recipe to the letter before posting their review, I could simply look at the ratings and decide which recipe to use. But people adapt recipes to their needs all the time; maybe they didn’t have regular vinegar and that’s why they used pickle juice instead. Maybe they’re vegan, and so they took the mayo out entirely. In each case, what’s written in the recipe differed from how the recipe was implemented, and ultimately the taste of the coleslaw is affected.

If significant adaptations can happen when making a simple coleslaw recipe, imagine how different any of the hundreds of classrooms implementing a mathematics curriculum might look. Or any of the dozens of STEM schools across the United States, especially since many of these had different "recipes" to begin with. That’s not to say that adaptation is bad; it’s not. It’s good. It’s adjusting “to taste,” so to speak, to accommodate local contexts and conditions. Implementation research helps you know not just what the “recipe” is, but how it played out in real world settings.

In keeping with the theme of this month’s postings, I’d like to leave you with some resources to get your feet wet. Implementation resources are few, but here are two that can get you started:

Implementation Research: A Synthesis of the Literature. This monograph by Fixsen et al. provides a synthesis of implementation research and includes information from the fields of mental health, social services, education, employment, and justice.

Measuring Implementation Strength from the London School of Hygiene and Tropical Medicine. Much of the implementation literature comes from the health sciences, and this literature review outlines general approaches others have used to measure implementation.

Measuring Enactment of Innovations and the Factors that Affect Implementation and Sustainability: Moving Toward Common Language and Shared Conceptual Understanding. Current and past members of Outlier wrote this 2012 article. It proposes a conceptual framework that can be used by researchers and others to more effectively communicate about and understand implementation.

Sarah Rand, Associate Communications and Project Director

STEM (Science, Technology, Engineering, and Math) education is a major part of the national conversation about how to improve education. Everyone is talking about STEM; Obama, Mayim Bialik from the Big Bang Theory TV show, and musician Will.i.am are all talking about STEM, as well as many schools and districts across the U.S.

Outlier is also talking about STEM. We are currently working on an NSF-funded research study examining STEM school models in seven states. To someone just entering the conversation, or even someone who has been working in the field for a long time, STEM education may seem like an overwhelming topic to learn about. Here are a few of the resources we use to stay up to date on STEM education in the political and educational landscape.

STEM Education Coalition: This organization works to raise awareness of STEM in Washington D.C. politics. Their weekly newsletter and website is full of education policy news, information about events in STEM, and resources for promoting STEM education.

Triangle Coalition for STEM Education: Another Washington D.C.- based organization, the Triangle Coalition’s mission is to bring together business, government and education to create a STEM-literate public. They have an annual conference coming up in September about sustaining STEM education. Follow them on Twitter for STEM-related news.

The National Girls Collaborative Project (NGCP): Promoting the important mission of motivating girls to pursue careers in STEM, the NGCP offers great resources. Their program directory is a comprehensive list of STEM programs for girls. You can search by city and state to find a program near you.

Pinterest: If you haven’t yet explored the Pinterest community, take a look and be prepared to find very useful and perhaps unexpected resources. The search for STEM Education boards yields many activities having to do with kids both in and outside of the classroom.

Also, check out STEM school resources on the Outlier website, such as our STEM school infographic. 

Amelia Baxter-Stoltzfus, Research Assistant

As an educator, researcher, or evaluator, you know that searching for resources can be an arduous and time-consuming task. This month we will share posts about resources that we use in computer science education, STEM education, evaluation, and implementation research.

Resources may be hard to come by, particularly for those working in computer science (CS) education. These are five venues that offer interesting resources for computer science education. These are sources of useful information for CS classroom teachers, advocates of CS education, and those who want to learn more about CS. Whether you are a newcomer to or a veteran of the CS education community, or are a well-established member of the community I’m sure you will find something valuable in the resources listed below.

The National Center for Women & Information Technology is a non-profit organization dedicated to increasing women’s participation in computing and technology. NCWIT’s resources page is extensive and includes case studies, talking points, and advice for school leaders, teachers, and students considering a future in computing.

Mark Guzdial, a professor at Georgia Institute of Technology, writes a blog about computing education. He mixes research, personal reflection, and advocacy, and also has a great resources page for new computer science teachers.

Code.org is a non-profit organization that aims to spread and expand access to and participation in computer science through policy changes, petitions, and social media. Their educator resources page contains coding tutorials and tools for local advocacy.

The Computer Science Teachers Association is a membership-based community for computer science and technology teachers, providing connections, curriculum standards, professional development opportunities, and more. (And it’s free to join!)

Our 2013 Building an Operating System for Computer Science Education report is a great resource for teachers, policy makers, school district leaders, and others. The report describes the current landscape of professional development available to high school CS teachers, provides information on CS teacher professional development, describes characteristics of CS teachers, and provides an in-depth description of CS education in schools using the voices of teachers and their school administrators.

June 4, 2014

Melanie LaForce, Associate Director

At Outlier Research & Evaluation, we work with people from all corners of the education world. We collaborate with teachers, principals, policy makers, program developers, scientists, and others. Much of our work at Outlier involves evaluation. Museums, for- and non-profit organizations, school districts, and other institutions engage us to help them understand whether their education programs are working. We help our clients understand their program’s successes, areas for improvement, and outcomes.

Most of our evaluation clients are not researchers, and many of them are unfamiliar with evaluation. Therefore, we often find ourselves in a position to act not only as a critical friend to our clients but also as a teacher of evaluation. It is important to Outlier that our clients fully understand our evaluation methods and can actively collaborate with us.

Clients often approach us because they want to know if their programs are “successful,” that is, whether or not they are achieving the desired outcomes. We help them understand the benefit of looking at the implementation as well as the outcomes of their programs. Without understanding how the program is being implemented, it is difficult to interpret outcomes. For example, if a program is not achieving its goals for student learning, examining the program’s implementation will help the organization identify a cause. Perhaps teachers delivering program content were not sufficiently trained, or technology resources are insufficient. In our evaluation work we measure implementation carefully to help clients understand not only whether their program is working, but how.

It is difficult in any research to definitively say whether something “works.” Proving cause and effect––for example, “This program causes an increase in learning”––can be achieved only with highly rigorous methodologies. The gold standard for demonstrating cause and effect is to conduct a randomized control trial (RCT). In an RCT, one group of the targeted population is randomly assigned to a treatment (program participating) group, while another target population is randomly assigned to a control (non-participating) group. In addition, a desired outcome for each group––such as interest in computer science or math intelligence–– is measured both before and after the program participation. This way, growth can be compared for both groups.

As you can imagine, it’s unlikely that a program (or its evaluation team) will have the capacity to randomly divide participants into different groups. It’s not generally feasible to randomly select half of a student population to receive a treatment or program (e.g., double-dose algebra or a new science curriculum). Rather, most education programs serve an entire group of students or population that needs it. This practice may be based on policy or  equity. Consider an example where a district decides to institute a double-dose algebra course for all students who failed Algebra 1.  It’s unlikely that a researcher would be able to randomly assign only a portion of those students to the double-dose course. That could be seen as inequitable or logistically improbable. In addition, many clients seek out evaluation after their program has already begun, making it impossible to measure the desired outcome both before and after the program.

We do our best at Outlier to provide clients with the most rigorous and appropriate methodologies. We can often use quasi-experimental designs, which provide strong clues about a program’s effectiveness. We can also look at participant reflections to understand whether the participants themselves feel that the program was effective. There are many ways we can understand a program’s value, even if a strict cause-and-effect relationship cannot be determined. We also take care to to educate clients about various methodologies, which in turn helps us determine which strategy will provide them with the information most relevant and useful to their program’s success.

May 29, 2014

Courtney Heppner, Associate Project Director

A couple of weeks ago my colleague Liz Bancroft wrote a great post about the struggles of sharing research with schools in a timely way. She noted that schools are fast-moving and ever-changing, and that high-quality research (instrument development, data collection, data analysis, report writing, etc.) takes time. However, even if the rate of school change and research were more in sync, there would still be the problem of translating research to practice. I can’t tell you how many principals and administrators I have visited who have stacks of reports and journal articles as high as their desks because they just don’t have the time to sit down and read them. As much as educators want to be current with all the latest and greatest research, the volume of what is out there, coupled with the traditional modes of dissemination (i.e., journal articles or lengthy reports), is just not conducive to consumption by the incredibly busy practitioner. So, what to do?

May 23, 2014

Sarah McKellar, Associate Project Director

Educators need to have a clear understanding of how participating in research can benefit them and their students. This declaration of purpose might be obvious for some researchers and challenging for others. Unlike studies done in labs or other settings, data collection in schools requires the development of partnerships and a shared understanding of how the study will benefit the participants.

  A financial incentive is not enough for teachers or other educators to participate meaningfully in a study. Teaching and education administration demand an enormous amount of time during the school year, and any time not spent on something that will directly benefit students is often seen as a waste. When I was a teacher, my time was gold, and anything I could do for my students was more important than compensation for participating in a study. When money is the only incentive for teacher-participants, they may not be particularly thorough when responding to questions. If researchers can find ways to combine purpose with participation, they will be more successful collecting data for their projects.

  Researchers also can’t presume which factors, such as potential broader impacts, will motivate teachers to participate in a study. Those collecting data must gain an understanding of participants’ reasons for engaging in research. This won’t happen if researchers do not ask about, understand, or in some way contribute to educators’ needs. In one study, our team found that administrators participated in research in order to have access to descriptive data for decision-making. In the same study, we found that teachers wanted to participate mainly to have their voices heard by administrators and curriculum developers. This illustrates how important it is to develop strong partnerships by asking relevant questions.

  During this process, researchers should be cautious not to overpromise or misrepresent what they can offer schools or how much commitment from schools they are asking. (This goes back to communication, Lesson 3). Failure by schools and researchers to communicate and understand each other’s purposes can lead to unfulfilled promises, failed studies, and educators who swear to never again participate in research. Communication and understanding goals can help ensure a positive and successful experience for all.

  Our team has learned––and relearned––these five lessons, and we seek ways to continually improve. This series of blogs is just the beginning of exploring researcher-practitioner relationships; there is much more to say about each of these lessons and the many lessons left to share. As our team learns and reflects on our practices, we hope to share even more, and we welcome conversations on this topic.

  In case you missed a post, here are the links to each of our “Top 5 Lessons Learned When Conducting Research in Schools.”

  Lesson 1: Create partnerships with individuals in all organizational levels 

Lesson 2: Dedicate the time and resources necessary for successful partnerships

Lesson 3: Customize communication for all partners

Lesson 4: Anticipate change and be flexible

May 22, 2014

Sarah McKellar, Associate Project Director

When conducting research in schools, external factors can interrupt or even derail research plans. Such factors include transitions in leadership, new evaluation policies, and changes in academic standards among many others. With the many factors that affect districts, schools, and classrooms, research projects often require midcourse adjustment. Our team has found that anticipating external factors and responding to them is key, even when that means modifying initial plans. Don’t fear changing small or large aspects of the project if the changes will help you better answer your research question(s) and are consistent with both your ethics and your Internal Review Board.

Here are some of the ways our team anticipates and responds to factors that affect our partnerships with schools:

-We foster commitment and partnership with several leaders and individuals within a district to ensure the study continues even if one of the key participants leaves the district or a principal changes positions.

-From questionnaires and ongoing communication, we gather feedback about teachers’ experiences and adapt data collection accordingly. In one case, we reduced questionnaire administration from twice to once a year after teachers communicated that our questionnaire took too much time and we couldn’t cut questions.

-We tailor incentives for participating to different populations and change them based on response rates. For example, in one study we incentivized teachers to complete our questionnaires by having a few drawings for larger payment amounts. However, we learned that this wasn’t encouraging enough for teachers. So we changed our approach to providing every subject with a Starbucks or Dunkin' Donuts gift card for each survey completed and our response rates improved.

- We work to communicate clearly with partners, as described in Lesson 3. For example, we realized that educators didn’t understand our project from the informational materials we provided, so we revised our handout after working with a graphic designer (see before and after handout).

Before:

After: 

May 21, 2014

Sarah McKellar, Associate Project Director

Like all organizations, districts and schools are made up of individuals with different agendas and priorities. Researchers who can successfully communicate with each of these individuals will not only collect good data but also ensure that participating in the project is a good experience for all parties. Researchers must understand the way communication works in a district and that the way they communicate with teachers needs to be different from the way they communicate with district leaders, which needs to be different from the way they communicate with school leaders. Further, the mode of communication can vary from place to place. In one district we wondered, “Why don’t teachers and administrators just f*&!@g read their email?” until we learned that the best way to communicate with them was to place paper flyers in their school mailboxes.

Here are some strategies our team has used to communicate with practitioners:

1) Develop a user-friendly website with all project information. See our S3 and BASICS project websites for examples. 

2) Send out emails with links to surveys and appropriately follow up.

3) Plan semiannual meetings with principals to share findings and answer questions.

4) Pick up the phone and call district leaders (instead of sending multiple emails).

5) Have in-person conversations with teachers and school and district leaders.

Sarah McKellar, Associate Project Director

May 20, 2014

As a researcher, you first must ask yourself whether you have the time to create a true partnership and develop a project that is meaningful to your participants. This is especially important because some district and school leaders have become wary of participating in research that will take up teachers’ valuable time while offering little in return. In your budget, consider the time and resources you will need to develop reports, products, and tools that educators will find useful. For an example, see a previous blog post by Maurice Samuels on how investing the time to generate logic models reaps benefits for both evaluators and practitioners. For another example, our team has developed visualizations and online reports that make findings directly accessible to study participants; during our study of STEM schools in Ohio, we found that educators wanted to compare their own definitions of STEM schools to others’, so our team created an online infographic of STEM school model components.

Districts’ and schools’ decisions to participate in research sometimes come down to a cost benefit analysis, so it is important to keep the following questions in mind:

-Will we have time to create meaningful products that benefit teachers, school leaders, and district leaders?

-Does someone in the district have the time to invest in a partnership and the logistics involved?

-If the research team has a strong connection with only one district leader, will the project suffer if that individual leaves the district?

-Which school staff can serve as liaison to principals and teachers for building trust and investment?

-Will teachers have sufficient time to participate in the research activities?

Here are a few steps our team takes to ensure all parties have enough time for quality partnerships:

1) Identify a dedicated district point of contact who is paid to be a liaison (this person is often a retired teacher who doesn’t already have a demanding schedule, or someone whose job already includes ongoing conversations with schools).

2) Identify a dedicated school point of contact.

3) Ask the district to set aside time for teachers to complete surveys or logs.

4) Administer questionnaires or visit schools during non-taxing times, e.g., after testing or in the fall after students have settled in from summer break.

5) Keep questionnaires under 20 minutes for teachers.

May 19, 2014

Sarah McKellar, Associate Project Director

First, researchers must form partnerships with schools and educators. When I say partnerships, I mean reciprocal relationships with two or more invested parties. In educational research, a partnership means that researchers and the educational organization mutually benefit from the project so that both are motivated and committed to collecting sufficient data for project success.

  The first priority for most researchers is to design a project that will advance knowledge. With this focus, they often make the mistake of viewing data collection in schools and districts as unidirectional and fail to form partnerships with their educator participants. The desire to publish in journals often comes at the expense of creating products that provide information meaningful to the participants. Remember that the group of educators affected by participation in a research study encompasses many people, including district leaders, principals, instructional coaches, teachers, students, and parents. In one of our studies, we worked with curriculum coordinators at the district level and selected a district point of contact to communicate with principals and teachers. While this was an excellent initial step, we quickly learned that we needed to put effort into directly engaging principals and teachers in the partnership. For example, we provided district-level data to district administrators and school-level data to teachers and principals so that all participants received the data that was most meaningful to them.

  An even greater challenge for researchers is the concern that researchers and practitioners developing stronger partnerships might lead to biased results or conflicts of interest. I argue that partnerships are essential for producing quality research. Researchers must understand the needs of the educators they study in order to recognize why schools would invest time and energy into such projects. When researchers face insufficient response rates, the lack of a researcher-practitioner partnership may be the culprit. There is still far too little understanding of why many districts, schools, and teachers refuse to participate, or withdraw from participation in valuable research studies. It is the “elephant in the room” for those who wish to do field work: How do I get a research site? How do I get a representative sample of teachers to respond to our questionnaires?

May 19, 2014

I’m Sarah McKellar, a researcher and Associate Project Director at Outlier Research & Evaluation. Conducting research in schools and with educators is near and dear to my heart, as I was previously a public school teacher whose desire to learn more about what motivated my students and my teacher colleagues led me to become a researcher. This week, I will dedicate a blog post each day to a different lesson that Outlier Research & Evaluation has learned from working with educators and in schools.

I’m Liz Bancroft, an Associate Project Director at Outlier Research & Evaluation. I’m writing this week from Knoxville, Tennessee, where I’m visiting L&N STEM Academy, one of the schools involved with our STEM School Study. This is one of my favorite parts of education research. The response to our being here has been awesome; both teachers and students have shown genuine interest in what we’re doing, what we’ve found, and, understandably, how we can provide information that is useful to them. And when.

This problem—how do we conduct rigorous research, which takes time, and provide findings that schools can use now?—is not a new one. At Outlier, we spend a lot of time thinking about how to make our work useful to not only the larger research community, but to practitioners, and a big part of this has to do with being timely in sharing information with schools. This blog is one way that we’re trying to expand from the research norm of sharing findings only in the form of publications. In the STEM School Study, we provide yearly school-level reports from survey data and use the study website to disseminate products and findings as soon as we can. But it has really struck me this week how different the pace of education in schools is from the pace of education research:

Things are moving fast in STEM schools.

New STEM schools are popping up all the time, and existing STEM schools are constantly trying to keep up with the latest technology, approaches, and practices. As STEM education researchers, we want to be a resource to schools; we want to provide them with useful information and findings about what they’re doing, what other STEM schools are doing, and how the strategies and methods used by STEM schools relate to student outcomes. But this all takes time. We need to form partnerships with schools, develop instruments, collect data, analyze that data, write up findings… it’s a long process, and one we don’t want to rush.

At the same time, STEM schools and teachers need data and information now. Students are in their classrooms every day, and teachers want to give them the best education they can. Several times this week I heard this from teachers: “When will we be able to see what you guys have found? We really want to use what you’re learning and see where we can improve.” As researchers invested in education, we want that too, and we want it soon. And while this may be a problem with no single easy solution, we’re working on it!

Visit our website to see some of the resources we’ve developed for STEM schools, with more coming soon.

Greetings from the University of Chicago!  I am Sarah Rand, the blog manager for Outlier Research & Evaluation. This month our blog posts will address the theme of “Problems and Solutions.” As experienced education researchers, we face a wide variety of challenges in the field. This month we will share some of these challenges, as well as the  lessons we’ve learned. As usual, our topics include STEM education, implementation research, computer science, and evaluation.

Watch for posts from these team members:

Liz Bancroft, Associate Project Director

Sarah McKellar, Associate Project Director

Courtney Heppner, Associate Project Director