4 Running and Interpreting a Two-Way ANOVA in JASP

Download Data Sets:

• JASP Data File Two-Way ANOVA
• CSV Data File Two-Way ANOVA

The Pew Research Center collected and published survey data on attitudes toward women in leadership as part of their American Trends Panel Survey in 2018. This report detailed the findings from a sample of 4,587 participants who completed an online survey. As part of their report, Parker, Horowitz, and Igielnik (2018) discussed the disparities in attitudes toward women in leadership positions – both in political and corporate settings – by gender, political party, and age. For example, they reported that the majority of women said gender discrimination is a major reason there aren’t more women in high political offices and top executive business positions; however, only 36% of men agreed it was an issue for high political offices and 44% for top executive business positions. When they looked at gender and age together, the majority of women agreed gender discrimination in a barrier to leadership across all ages, while only 35% of men 18-49 years old agreed and 38% of men 50 years and older agreed. We are going to follow up on their analyses by taking a more nuanced look at the interaction between age and gender on attitudes toward women in leadership using their data set – specifically, we are going to focus on women in political positions.

Our first factor, gender, has two levels: (1) woman and (2) man. Our second factor, age, will have three levels: (1) younger adult (30-49 years older), (2) middle-aged adult (50-64 years old), and (3) older adults (65 years and older). Our dependent variable, attitudes toward women in politics, was created by creating an average of participant responses to 23 items that accessed attitudes toward women in political positions. Higher scores on this variable mean more favorable attitudes toward women in politics.

This study utilized a quasi-experimental research design. We do not have a control group and are not randomly assigning participants to a particular gender or age. We are simply grouping participants into the gender they identified with and the age group based on their reported age when the data were collected. The table below shows our 2×3 factorial ANOVA study design.

Hypotheses

The null hypotheses:

Factor One (Main Effect of Gender)

• Conceptual H_0A: There are no significant mean differences in favorable attitudes toward women in politics between women and men.
• Mathematical H_0A: There are no significant differences between the population means for women and men on favorable attitudes toward women in politics. M₁ = M₂

Factor Two (Main Effect of Age Group)

• Conceptual H_0B: There are no significant mean differences in favorable attitudes toward women in politics between the three age groups.
• Mathematical H_0B: There are no significant differences between the population means for the 30-39 years old group, 50-64 years old group, and 65+ years old group on favorable attitudes toward women in politics. M₁ = M₂ = M₃

Factor One x Factor Two (Interaction between Gender and Age Group)

• Conceptual H_0AB: There is not a significant interaction between gender and age on favorable attitudes toward women in politics.
• Mathematical H_0AB: Gender x Age Group = 0 for each cell.

The alternative hypotheses:

Factor One (Main Effect of Gender)

• Conceptual H_1A: There is a significant mean difference in favorable attitudes toward women in politics between women and men.
• Mathematical H_1A: There is a significant difference between the population means for women and men on favorable attitudes toward women in politics. M₁ ≠ M₂

Factor Two (Main Effect of Age Group)

• Conceptual H_1B: There is a significant mean difference in favorable attitudes toward women in politics between the three age groups.
• Mathematical H_1B: There is a significant difference between the population means for the 30-39 years old group, 50-64 years old group, and 65+ years old group on favorable attitudes toward women in politics. M₁ ≠ M₂ ≠ M₃

Factor One x Factor Two (Interaction between Gender and Age Group)

• Conceptual H_1AB: There is a significant interaction between gender and age on favorable attitudes toward women in politics.
• Mathematical H_1AB: Gender x Age Group ≠ 0 for at least one cell.

JASP Analyses

In order to run analyses in JASP, the first thing we need to do is open the data set we will be working with. To do this, open JASP and complete the following steps.

Click on File → Open → Computer → Browse → Choose the Two-Way ANOVA Practice Data (Pew Research Center 2019 W36)

Before we get started with our analyses, we are going to filter our AgeGroup variable to filter out participants who were in the 18-29-year-old age group. Unfortunately, the sample size in this group was too small to compare to the other age groups in this analysis. Click the arrow in the filter column for the 18-29-year-old label. To do this, click on “AgeGroup” and click the checkmark to filter out participants who are in the 18-29-year-old age group (Value = 1). When there is an X instead of check mark, then you have filtered this group out appropriately. See the image below.

Assumption Testing

Prior to running our two-way ANOVA, we first must check to make sure we meet the assumptions for this statistical test.

Assumption One: Are our factors measured at the categorical (i.e., nominal) level with two or more groups?

Yes. Our independent variables for this study are gender and age. Our first factor, participant gender, had two levels: women and men. Participants who participated in this survey either identified themselves as a woman or a man. Thus, this factor is dichotomous and categorical. If you look at the data label icon in JASP, you can see it has the appropriate icon (three overlapping circles) for a nominal/categorical variable. Our second factor, age, had three levels: 30-39 years old, 50-64 years old, and 65+ years old. This factor is categorical and has the appropriate icon in JASP to indicate it is a nominal variable. We meet this assumption.

Assumption Two: Is the dependent variable continuous (i.e., ratio or interval)?

Yes. Our dependent variable for this study is participants’ mean score for their responses to the 23-items measuring their level of favorable attitudes toward women in politics. This variable is continuous, meaning it was measured at the interval or ratio level. If you look at the data label icon in JASP, you can see it has the appropriate icon (a ruler) for a scale or continuous variable. We meet this assumption.

Assumption Three: Are the samples independent?

Yes. Looking at our data set, we can determine that the groups of participants in each cell of our research design are independent. In our dataset, you can see that all participants identified as either a woman or a man for factor one. You can also see that participants reported that their age group was one of the following: 30-39 years old, 50-64 years old, and 65+ years old. When we display the levels of both of our factors in a table, as you see below, we have six independent cells representing the possible combinations of our two factors for our 2×3 ANOVA. Thus, knowing the research design did not allow for participants to be exposed to more than one condition or level of either factor, we can conclude that participants in all cells are independent of one another. No participant is in more than one group. We meet this assumption.

Assumption Four: Is the dependent variable normally distributed for each group of the IV?

Before we run our descriptive statistics, we first have to filter our first factor. To do this, click on “Gender” and click the arrow to filter out participants who refused to answer (value = 9) and women (value = 2). After you click the arrow you will see an x instead of a check mark (see image below). Then click on the x in the bottom righthand corner to exit out of that screen. This will filter out women participants and allow us to calculate the descriptive statistics for the top row of our 2×3 ANOVA.

To check our data for the next two assumptions, we will use the Descriptives analysis tab. Click Descriptives. When the “Descriptive Statistics” window pops up, we will need to move our dependent variable to the “Variables” box and our independent variable to the “Split” box using the arrows depicted in the photo below – this is so we can examine normality in the DV for the three levels of our second factor separately.

Once we have our variables in the appropriate boxes and our first factor filtered, we are going to ask JASP to run the various statistics and graphics we will need to interpret for our assumptions by clicking on the appropriate boxes in the test window. We will be asking JASP to provide us with all possible output we may want to look at for our assumptions; however, we will focus on interpreting the output that you will be expected to analyze for your course assignments.

1. Under the “Basic Plots” tab we will check “Distribution plots.”
2. Under the “Customizable Plots” tab we will check the “Boxplots,” “Boxplot element,” and “Label outliers” boxes.
3. Under the “Statistics” tab we will check the “Skewness,” and “Kurtosis boxes.

See the following image to make sure your test window has all the appropriate boxes selected.

As a reminder, for this course, we will focus on interpreting the skewness and kurtosis statistics to determine if our data are normally distributed. We want our skewness and kurtosis statistics to be between -2 and +2. For this study design, we have six different cells that we have to check to determine if each of those six groups of participants have a normal distribution of attitudes toward women in politics.

Looking at the values on the output above, we can see our skewness values for the 30-49-year-old group (-0.42), 50-64-year-old group (-0.24), and 65+ group (-0.31) are all within the acceptable range of -2 and +2. Additionally, the kurtosis for the 30-49-year-old group (-0.59), 50-64-year-old group (-0.67), and 65+ group (-0.65) are also within the acceptable range of -2 and +2.

Now that we have analyzed the descriptive statistics for participants in the participants who identified as a man, we have to conduct the same steps for participants who identified as a woman. To do this, click on “Gender.” Next, you will click on the check mark in the “Filter” column for Men so it is now an X. You will also need to click on the X next to Women to make sure it is now a check mark. Once this is completed, click on the x in the bottom righthand corner to exit out of that screen. This will filter out participants who identified as men and allow us to calculate the descriptive statistics for the bottom row of our 2×3 ANOVA.

Once this is completed, you will want to repeat the steps above to run Descriptives again with our remaining three groups.

Looking at the values on the output above, we can see our skewness values for the 30-49-year-old group (-0.62), 50-64-year-old group (-0.55), and 65+ group (-0.48) are all within the acceptable range of -2 and +2. Additionally, the kurtosis for the 30-49-year-old group (-0.31), 50-64-year-old group (-0.05), and 65+ group (-0.21) are also within the acceptable range of -2 and +2. The skewness and kurtosis statistics for each of our six cells are also reported in the table below.

To report these results in APA format, we could write:

Attitudes toward women in politics was normally distributed within all six cells, with skewness and kurtosis statistics between -2 and +2.

Now that we have checked our data for normality, let’s take a look at our boxplots for each cell to see if we have outliers in any of our study cells.

Assumption Five: Are there any outliers in the sample?

To assess for outliers, we will look at the boxplots in our JASP output. If we have outliers, they would be outside the top and bottom lines or whiskers. When we look at the boxplot for participants who identified as a man, you can see there are no outliers visible in 30-49-year-old age group, 50-64-year-old age group, or the 65+ age group.

However, if we look at the boxplot for participants who identified as women, you can see we have a lot of outliers across all three age groups, below the lower quartile.

To report this in APA format, we could write

There were five outliers in the 65+, women group and several outliers in both the 50-64-year-old, women group and in the 30-49-year-old, women group as assessed by the inspection of a boxplot.

As we discussed in previous chapters, when we have outliers we have to decide if they are going to correct it, keep it, delete it, or replace it. This data has already been checked and cleaned by researchers with the Pew Research Center; thus, we can assume these outliers are not due to a data entry error. As a researcher, you would then have to decide if the outliers are exerting a significant enough impact on your results to warrant deletion. Given that outliers are especially common in datasets with large samples (as ours is; N = 4,337), and that we have so many, we will keep these outliers in our dataset in order to avoid removing a significant amount of datapoints from our model (something that is generally discouraged, Faraway, 2015). Instead, we will model how to run the analyses as normal with outliers present, and at the end of this chapter, will run the analyses with those outliers removed to examine whether the results remained the same.

Assumption Six: Are there homogeneity of variances?

To determine if we have homogeneity of variances, we need to ask JASP to run our two-way ANOVA. Checking for this assumption is a part of the overall two-way ANOVA. Let’s move onto our primary analyses below, and complete checking this assumption in that section.

Primary Analyses

Before running our primary analyses, we need to turn off the filter for Gender. To do this, hover over Gender and click when you see “click here to change labels or inspect filter.” Then click on the x icon for men. You should now see a check mark next to values 1 and 2, for men and women. Click on the X in the righthand corner to exit that screen. [You want an X to remain for Value 9].

To run a two-way ANOVA, go to ANOVA at the top of the JASP screen and click on ANOVA. See the image below.

1. First, we need to move our dependent variable FavAttitudes to the “Variables” box and our factors Gender and AgeGroup to the “Fixed Factors” box. We are then going to ask JASP to run our omnibus test by clicking on the appropriate boxes in the test window.
2. Under “Display” we will check the “Descriptive statistics” and “Estimates of effect size” test boxes.
3. Under “Estimates of effect size” we will check the omega squared “ω²” box.

See the image below to make sure your test window has all the appropriate boxes selected.

Next, we are going to ask JASP to give us the statistics for our final assumption – homogeneity of variances.

4. Under “Assumption Checks” we will check the “Homogeneity tests” box.

Are there homogeneity of variances?

Now that we have asked JASP to run our first set of analyses, we can begin our interpretation of the results. The first thing we need to do is go back to our last assumption and determine is we meet the assumption for homogeneity of variances. As you can see from the output below, the p value for our Levene’s test is < .001. Because p is < .05, we can conclude that we do not meet this assumption. Although this is a bummer, we are in luck! As long as the number of participants in each group is approximately equal and not less than five per group, ANOVA is generally “robust” to a violation of this assumption. If we look at the descriptives box in our output, we can see the sample sizes in each cell are approximately equal: 671, 655, 607, 626, 720, and 600. Thus, we can proceed with our analyses. If we did not have equal sample sizes in each of our groups, we could potentially conduct a nonparametric test equivalent instead, the Scheirer-Ray-Hare test.

To report the results of the Levene’s test in APA format, it might look something like this:

There was not homogeneity of variances, as assessed by the Levene’s test for equality of variances (p < .05); however, because there was an approximately equal number of participants in each cell, the two-way ANOVA is considered robust to this violation (Maxwell & Delaney, 2004).

Interpreting the Statistical Significance of Two-Way ANOVAs

Now that we have completed the assumption checking for this statistical test, we can proceed with interpreting the primary analyses. The first thing we will interpret is the statistical significance of the omnibus test results. To do this, we are going to look at the “ANOVA – FavAttitudes” table. We have a total of three different F tests that we will interpret for each of our hypotheses.

To interpret the results, we want to look at the p values for the main effect for factor one (hypothesis one), the main effect for factor two (hypothesis two), and the interaction (hypothesis three). If p is less than our alpha level of .05, we will reject the null hypothesis.

1. There is a significant statistical difference between the group means of our first factor, gender, because p < .001. Because our p value is less than our alpha level of .05, we will reject our first null hypothesis. Thus, we would say we have a main effect of gender on participant attitudes toward women in politics.
2. There is not a significant statistical difference between the group means of our second factor, age group, because p = .18. Because our p value is greater than our alpha level of .05, we will fail to reject our second null hypothesis. Thus, we would say we do not have a main effect of age on participant attitudes toward women in politics.
3. There is a significant interaction between gender and age group because p = .01. Because our p value is less than our alpha level of .05, we will reject our third null hypothesis. Thus, we would say we do have a significant interaction between gender and age group on participant attitudes toward women in politics.

Writing the basic results of a two-way ANOVA in APA format follows this general format:

F(df1, df2) = F statistic, p < .05 or > .05, ω² = omega-squared statistic

However, with a two-way ANOVA, we have three F statistics to report: (1) main effect for our first factor, (2) main effect for our second factor, and (3) our interaction. Plugging our results above into this format, we have:

Main Effect for Factor One: F(1, 3873) = 92.39, p < .001, ω² = .02

Main Effect for Factor Two: F(2, 3873) = 1.71, p > .05, ω² = .00

Main Effect for the Interaction: F(2, 3873) = 4.56, p < .05, ω² = .002

Now we know that we have a significant interaction between our two factors – gender and age group – but what does that mean? To answer this, we are going to do two things: (1) graph the means of each cell in our interaction to help us visualize what is happening with the data and (2) conduct post hoc tests called simple main effects.

We can graph the means of the interaction using JASP or using Excel. For now, let’s take the easy way out and ask JASP to provide us with a graph of the interaction.

1. In the ANOVA analysis window, click on “Descriptive Plots.”
2. Move Gender to the “Separate Lines” box. This will graph our first factor, gender, as two separate lines on our graph.
   1. I choose this variable to have separate lines for because it will make it easier to understand two separate lines rather than three. However, you can always play around with different options to see what method works best for your study.
3. Move AgeGroup to the “Horizontal Axis” box. This will group our second factor, age group, on the x-axis of the graph.
4. Click “Display error bars.”

Looking at the Descriptive plots or line graph above, we can see that we have an ordinal interaction – the groups means do not cross or overlap in our graph. By graphing the means of each of our study cells, we can visualize that is happening with our data. For example, looking at this graph we can see women in the younger (30-49-year-old) age group reported the most favorable attitudes toward women in politics. Alternately, men in the younger age group reported the least favorable attitudes toward women in politics. This graph is limited because it does not tell us which groups are significantly different from one another. However, by asking JASP to display error bars using the confidence intervals, we are able to take an educated guess as to where the statistical differences will be. The dots on the graph indicate the group means and the error bars indicate the 95% confidence intervals around the means. A general indication of how spread out or variable participant ratings of attitudes toward women in politics were in each group. Groups that do not have overlapping error bars are more likely to be significantly different from one another; however, this is not guaranteed – so post hoc tests are needed to confirm. Looking at the line graph above, it seems likely that women across all age groups had significantly more favorable attitudes toward women in politics than men. The group with the smallest gender differences in attitudes toward women in politics was the older adult age group. To confirm our interpretation of the graph is correct, we will conduct post hoc tests. A commonly used post hoc test to help interpret significant interactions with two-way ANOVAs are simple main effects.

Our next step is to run tests of simple effects.

1. In the ANOVA analysis window, click on “Simple Main Effects.”
2. Move AgeGroup to the “Simple Effect Factor” box.
3. Move Gender to the “Moderator Factor 1” box.

This will test to see if there are differences in our second factor, age group, at each level of the first factor (women, men). Our results are copied below.

To interpret simple effects, we will focus on the p values for each comparison.

• The p value on top (p = .21), tells us that when we compare participant attitudes toward women in politics between young, middle-aged, and older men, there is no statistical difference because p is greater than .05.
• The p value on the bottom (p = .009), tells us that when we compare participant attitudes toward women in politics between young, middle-aged, and older women, there is a statistical difference because p is less than than .05.

We can repeat these steps to run simple main effects to see if there are differences in our first factor, gender, at each level of the second factor (younger, middle-aged, older). We will follow the same instructions as before but switch the boxes we move Gender and AgeGroup into.

1. In the ANOVA analysis window, click on “Simple Main Effects.”
2. Move Gender to the “Simple Effect Factor” box.
3. Move AgeGroup to the “Moderator Factor 1” box.

The results of the additional simple main effects analyses are copied below.

To interpret simple effects, we will focus on the p values for each comparison.

• The p value on top (p < .001), tells us that when we compare attitudes toward women in politics for women and men who are 30-49 years old, there is a significant difference because p is less than .05.
• The p value in the middle (p < .001), tells us that when we compare attitudes toward women in politics for women and men who are 50-64 years old, there is a significant difference because p is less than .05.
• The p value on the bottom (p < .001), tells us that when we compare attitudes toward women in politics for women and men who are 65 years and older, there is a significant difference because p is less than .05.

To report the results of our simple main effects analyses, we could write:

After conducting analyses of simple main effects, we found a significant difference in attitudes toward women in politics between women and men across in the younger adult (F(1) = 63.14, p < .001), middle-aged adult (F(1) = 26.75, p < .001), and older adult age groups (F(1) = 12.94, p < .001). Across all age groups, women had significantly more favorable views toward women in politics compared to men. Further, there was not a simple main effect of age for men, F(2) = 1.54, p > .05. There were no differences in attitudes toward women in politics across age groups for men. However, there was a simple main effect of age for women, F(2) = 4.68, p < .01. Younger women held more favorable attitudes toward women in politics than the middle-aged and older age groups.

Interpreting the Practical Significance of Two-Way ANOVAs

Now that we have interpreted the statistical significance, we will look at the practical significance by looking at our effect size. A significant p value tells us that there is a significant interaction between gender and age on attitudes toward women in politics, but the effect size tells us how much of our dependent variable we are explaining with these factors.
Reminder: Omega Squared Effect Sizes (Cohen, 1988; Kirk, 1996)

We obtained an omega-squared value of .002 for our interaction, this is less than a small effect. There is a statistical interaction between gender and age on attitudes toward women in politics, but the variance explained by that interaction is very small. Meaning, the interaction between gender and age only explains .2% of the variance in participant attitudes toward women in politics. If we look at our main effect for gender, however, the omega-squared value was .02. This means that the variance in participant attitudes toward women in politics explained by gender alone was 2.3%, still a small amount, but larger than the interaction.

Reporting in APA Format

What do you need to report in your results?

Putting it All Together

Now, let’s write up the results of our test combining everything we’ve done so far, including the test of assumptions, and the results of the two-way ANOVA.

A two-way ANOVA was conducted to determine if there was an interaction between gender and age on attitudes toward women in politics. Attitudes toward women in politics was normally distributed within all six cells, with skewness and kurtosis statistics between -2 and +2. There were five outliers in the 65+, women group; 19 outliers in the 50-64-year-old, women group; and 21 outliers in the 30-49-year-old, women group as assessed by the inspection of a boxplot. There was not homogeneity of variances, as assessed by the Levene’s test for equality of variances (p < .05); however, because there was an approximately equal number of participants in each cell, the two-way ANOVA is considered robust to this violation (Maxwell & Delaney, 2004).

There was a statistically significant interaction between gender and age on attitudes toward women in politics, F(2, 3873) = 4.56, p < .05, ω² = .002. There was also a main effect of gender on attitudes toward women in politics, F(1, 3873) = 92.39, p < .001, ω² = .02. There was not a main effect of age group, F(2, 3873) = 1.71, p > .05, ω² = .00. After conducting analyses of simple main effects, we found a significant difference in attitudes toward women in politics between women and men across in the younger adult (F(1) = 63.14, p < .001), middle-aged adult (F(1) = 26.75, p < .001), and older adult age groups (F(1) = 12.94, p < .001). Across all age groups, women had significantly more favorable views toward women in politics compared to men. Further, there was not a simple main effect of age for men, F(2) = 1.54, p > .05. There were no differences in attitudes toward women in politics across age groups for men. However, there was a simple main effect of age for women, F(2) = 4.68, p < .01. Younger women held more favorable attitudes toward women in politics than the middle-aged and older age groups.

Real World Meaning

Remember, when we interpret the real world meaning of a study, we want to take out any statistical jargon and describe the results of the study in a way that is easy to understand by people who do not know anything about statistics. Think about how you would describe the results to your roommate, sibling, parent, or neighbor. For example, if I was telling my neighbor the real world meaning of our results, I would say:

When researchers studied the impact of gender and age on attitudes toward women in politics they found a few things: (1) men have less favorable attitudes toward women in politics across all age groups, (2) there are differences in attitudes toward women in politics between young, middle-aged, and older women – but not between different ages for men, and (3) younger women have the most favorable attitudes toward women in politics. Younger women had more favorable attitudes toward women in politics compared to all of the men in the study, and more favorable attitudes compared to the middle-aged and older women as well.

References

Parker, K., Horowitz, J., & Pumphrey, J. (2018). Women and leadership 2018. Pew Research Center. Retrieved from https://www.pewresearch.org/social-trends/wp-content/uploads/sites/3/2018/09/Gender-and-leadership-for-PDF_updated-10.1.pdf

Authors

This guide was written and created by Ruth V. Walker, PhD, Ashlyn A. Moraine, and Erin N. Prince.

Acknowledgements

We would like to thank the Pew Research Center for generously making their data available for secondary analysis. Additionally, we appreciate Dr. Kristen Black’s edits and suggestions during the creation of this guide.

Copyright

CC BY-NC-ND: This license allows reusers to copy and distribute the material in any medium or format in unadapted form only, for noncommercial purposes only, and only so long as attribution is given to the creator.