Appendix A: Applied research and program logic models
You may hear social research referred to as pure or applied. Pure research aims to build knowledge for its own sake; applied research aims to be useful for doing things like solving problems, making the most of resources, identifying opportunities for improvement, and planning how to reach a goal. These can be useful distinctions, but they’re not mutually exclusive categories. Much pure research is eventually very useful, and much enlightening knowledge is generated in the course of conducting applied research.
When conducting applied research about a program, organization, or policy, models often play the role of theory in the research process. I’ll focus here on how models can help generate empirical research questions. The following logic model, for example, depicts how a simple afterschool tutoring program is intended to work.
The inputs include all of the resources for the program (high school student-tutors, curriculum, and the cafeteria) and the demand for the program (middle schoolers who need help with math). The activities are the main actions undertaken by the program, and the outputs are the observable, countable units of service produced. The outcomes depict the chain of intended program results—the ways the program is intended to make the world a better place.
All components of the logic model can generate applied research questions to guide inquiry that could be helpful for the entire program planning and evaluation process. Here are some examples…
- Questions to understand and establish the need for the program:
- How many middle schoolers need help?
- What are the middle schoolers’ academic strengths?
- What math concepts are especially challenging for the middle schoolers?
- What are the middle schoolers’ study habits?
- How do the middle schoolers feel about learning math?
- Questions about program resources:
- What tutoring skills do the high school students have?
- What math knowledge do the high school students have?
- How much time do the high school students have to commit to the program?
- Is the cafeteria environment conducive to learning?
- Questions about activities and outputs:
- Are the high schoolers using good tutoring practices?
- Are the high schoolers following the group tutoring curriculum?
- Are the middle schoolers staying actively engaged in the tutoring?
- Are there any barriers to middle schoolers’ participation?
- What do the middle schoolers believe is the most helpful about the program?
- What do high schoolers think is going well? What concerns do they have?
- Questions about outcomes and possible unintended consequences:
- Are the middle schoolers gaining a better understanding of the targeted math concepts?
- Are the middle schoolers’ grades in math improving?
- Are the middle schoolers developing better independent study skills?
- How are the middle schoolers’ study habits changing?
- How is the program affecting middle schoolers’ overall academic performance?
- How is the program affecting middle schoolers’ attitudes toward school and learning?
- How is the program affecting middle schoolers’ participation in co-curricular activities?
- How is the program affecting the high schoolers’ educational aspirations?
- How is the program affecting high schoolers’ academic performance?
- What changes in the students have their parents observed?
- What changes in the students have their teachers observed?
- How will the program affect middle schoolers’ academic performance next year?
- Questions linking activities, outputs, and outcomes:
- How much time in one-on-one tutoring is sufficient for improving middle schoolers’ understanding of the targeted math concepts?
- Do middle schoolers who participate more often achieve larger gains in academic performance?
- Which middle schoolers benefit the most from the group tutoring sessions?
- How does participating middle schoolers’ academic performance differ from non- participating students’ academic performance?
- How do the students feel they’ve changed due to participating in the program?
Involving stakeholders in the design of applied research projects is the most important strategy for producing useful findings. Start by identifying the stakeholders: Who could benefit or suffer based on what we learn? Who can make authoritative decisions based on what we learn? Who will need to approve those decisions? Who will be in charge of implementing changes? These stakeholders can be involved in every stage of the research process. Collaborating with stakeholders, like program managers, on the preliminary step of developing a program model is almost always beneficial for identifying gaps in knowledge or surfacing disagreements over how the program is assumed to function by different stakeholders. These are prime opportunities for developing research questions. Other research questions can be identified by asking stakeholders what decisions they hope to make based on what is learned from the research project. Applied researchers should be certain they have a shared understanding of the meaning of key concepts. (I once spent hours making sure I understood what a program evaluation client meant by life vision.) Stakeholders should agree that the operationalizations of those concepts are valid. An entire applied research project will fail if, in the end, a key stakeholder looks at some undesirable findings and dismisses them with “Well, that questionnaire really wasn’t a good indicator of our program’s outcomes.” Stakeholders can provide valuable insight even into data collection plans—they know when kids will be unable to focus because of the school band practicing next door, too drowsy to talk after lunch, and too distracted by the countdown to spring break to bother with your questionnaire. When data have been collected and it’s time for data analysis and reporting, stakeholders can participate deliberatively, and then they will be much more inclined to take the findings seriously and use them for policy and program improvement. And if you’re conducting applied research, that’s the whole point.
