University of Minnesota Extension
www.extension.umn.edu
612-624-1222
Menu Menu

Extension > Family Matters > Evaluation Essentials: Types of Evaluation

Tuesday, June 13, 2017

Evaluation Essentials: Types of Evaluation

By Emily Becher, Research Associate

Last time on Cooking with Emily — I mean, Evaluation with Emily, I used pizza to describe evaluation as an essential process for making a decision. I also pointed out that research tells us why we like pizza, and evaluation tells us what kind of pizza to order right now. Now let’s get further into the weeds with some even more jargony words: formative, process, and summative evaluations.

"Formative" and "summative" are highly specific terms. If you haven’t been exposed to them, they are the exact kind of words people tune out on. In his book Evaluation Essentials: From A to Z, Marvin Alkin discusses how formative evaluation is designed to make improvements, e.g. how to change a program to improve outcomes. Summative evaluation is designed to support major decision-making, e.g. whether or not to continue funding for a program.

But the best analogy Alkin gives about this distinction relates to food.

When the cook tastes the soup, that’s formative.

When the guest tastes the soup, that’s summative.

— Robert Stake, Professor Emeritus of Education
at the University of Illinois, Urbana-Champaign

A Swedish Soup for You (Or Who?)

Let’s think through this. Imagine you’re a chef making soup from an old Swedish recipe that you tasted long ago. You don’t have a recipe, just some basic ideas of what went into it. As you cook the soup, you add some herbs, taste the soup, add some more herbs, and taste again. Then you add some pepper, taste the soup, add some salt, taste the soup, add a little more salt, and taste again. You keep going with adding ingredients and tasting until you have what you think is a great tasting soup.

woman tasting food being made in a home kitchen

But the soup’s not just for you. So you ask some of your family members to try the soup. Dad wants more spice, Mom wants a little sweetness, and sister wants more thyme. So you slowly add in various pieces, a little at a time, tasting, having your family taste, until you have a soup that you feel confident is really good.

As you make all of these incremental changes, you’re documenting. You’re documenting not just what you’re adding and how much, but how hot or cold the soup is, how much you’re stirring it, whether you’re stirring it with a metal spoon or a wooden spoon, whether you’re tasting it straight out of the pot or from a bowl, the size of the batch and so on. This is the “process” of the soup.

On your journey to bring this Swedish soup to a bigger audience, you try to replicate the process of making the soup in the restaurant where you work. The restaurant kitchen has slightly different equipment from your home kitchen. You try to replicate the process of serving the soup as you did at home with similar bowls, spoons, and temperature. Sometimes substitutions or adaptations have to be made and you document them. For example, the bowls you’re using at the restaurant are larger and you cook with have a metal spoons at the restaurant versus the wooden spoons at home.

commercial kitchen interior

You want to see if the soup will sell well. So you give customers a free sample and ask them to rate the soup on a variety of factors (taste, texture, heat, etc.) and whether they would pay to eat this soup. You look at similar restaurants in the area to see if they have soups on their menus. If they do, you note how much they’re charging for similar soup. You might even have customers try both a competitor’s soup and your soup and then ask them what was different, which they liked better, and why.

You also talk to people in the community who don’t eat at your restaurant and ask them if they saw you were selling this soup, would they be more likely to come in? Why or why not? You even have a trial-run where you put the soup on the menu for two months to see if it sells and if customers come back to buy it again.

You keep gathering information about the soup and customers’ experience of the soup until you feel that you have enough information with which to make a decision. You use the data to make one of a the following decisions:

A. You put the Swedish soup on the menu.

B. You decide soup is still a good idea, but this particular recipe is not hitting the mark for some reason, so you go back to the drawing board with a new recipe and replicate the process.

C. You decide to not put soup on the menu at all because: 

  • Even though the Swedish soup is great, it's not a fit for your customers and the information you’ve gathered suggests that no soup will ever hit the mark, or 
  • The cost of making a soup that hits the mark doesn’t make sense for your restaurant, or 
  • It’s not a great soup and it’s not worth the investment to go back to the drawing board.

So where does formative, process, and summative come into all this? Let's take them one at a time.

Formative Evaluation: Tasting the Soup

According to Stake, formative evaluation when the cook tastes the soup. In my opinion, there are two types of formative evaluation: one where you, the chef, are the tasting the soup and one where you are getting impressions of the soup from potential customers. Information from both are used to make the soup better.

I think a great example of the first type of creator-centric formative evaluation is the I CAN Prevent Diabetes cultural adaptations project (read about it here: Family Matters: Adaptation). Facilitators kept detailed notes, both written and oral, to track their impressions of how they taught each session and how participants were experiencing the curriculum. These reflections were fed back into the adapted curricula to make them better.

An example of the second type participant-centric formative evaluation is the Partnering for School Success project. The current iteration of the program was 17 years in the making, starting with an extensive literature review, focus groups conducted in partnership with cultural communities, and several rounds of piloting. During the current CYFAR funding grant, the evaluation data collected from participants were consistently used to make the program better.

two Latinas stand next to a drawing of a tree with paper leaves
Two program facilitators of Education: Our Best Legacy.

For example, in year 2 of the program, the project leads asked parents and staff about why certain goals set at the beginning weren’t being reached. What they heard was that parents and staff needed support in developing leadership and cultural awareness. So the project created a whole new leadership training in partnership with leadership experts at the University of Minnesota that was delivered in program year 3. They listened to participants and responded by rolling a new training into the program to make it better and make it more likely the initial goals of the program would be met.

Process Evaluation: Making the Soup

A common feature of formative and summative evaluations is process evaluation. This is the practice of describing and understanding not just the soup’s ingredients, but how you created and served it.

What comes to mind for me for this type of evaluation is the thorough documentation that we are doing with the Health and Nutrition PSE Pilot Projects and Place-Based Approach. Each of these initiatives are unique and have a dynamic and evolving story. As a result of the process evaluation, we can’t say, “This is the way to do PSE place-based approaches.” We can say, “Look at examples of how we did the work in these different communities under these conditions.” Maybe from there, we can draw some lessons learned that span the work. Documenting the process of this work is essential to informing our current work and giving ideas for replication to other communities and organizations.

Summative Evaluation: The Whole Bowl

Now to the more broad summative evaluation. Here’s the summative evaluation that was involved in our soup analogy:
  • Describing the recipe of the soup (the soup's ingredients and how its made).
  • Describing the process of the soup (how to create the best experience of the soup).
  • Describing any adaptations that have been made in replicating the soup at the restaurant versus your home kitchen.
  • Asking customers to tell you about their experience of the soup.
  • Conducting a trial-run of selling the soup on the menu (a pilot).
  • Compare a group of people who ate your soup to a group who ate a different kind of soup, or who didn’t eat any soup. Ideally, you would randomly choose people to be in each group so you could say with confidence that any differences you see is because of your amazing Swedish soup.
  • Making a decision about the soup.

You may be thinking that summative evaluation sounds a lot like formative evaluation and process evaluation wrapped together. For me, the big difference is the type of decision you make at the end. With formative, it’s making decisions to form the program. With summative, it’s about making a judgment of the value of the whole program. So the question we ask in a formative evaluation is,  “What is needed to make the soup better?” The question in a summative evaluation is, “Do we sell the soup?”

The most recent example of summative evaluation that comes up for me is Parents Forever™ (surprise, surprise). What we’re trying to do is to compare parents who have taken either the in-person and online Parents Forever™ course to a group of parents who have never gone through divorce education. What we want to be able to see is if people who go through a Parents Forever™ course show different parenting, coparenting, and self-care behaviors than people who don’t go through any divorce education. We want to know if we can say, “Yes, Parents Forever™ changes people's lives for the better.”

We hope that this summative evaluation (also called an outcome evaluation) will establish the program as an evidence-based curriculum and will lead other states to approve Parents Forever™ as an appropriate choice for mandated divorce education in their state.

There we go. Evaluation broken down into a giant soup analogy. Who's hungry?

1 comment:

  • © Regents of the University of Minnesota. All rights reserved.
  • The University of Minnesota is an equal opportunity educator and employer. Privacy