I have to agree with Matt. It appears you may be over-generalizing from the data. Definitely by Game 4, there are no differences between time spent playing the games. I think one big takeaway would be Theme hardly matters. (by game 4, theme doesn't seem to be impacting time spent processing - however, it might actually still be hindering performance).
In Table 1, the "rules are difficult to understand" is insignificant across all conditions - at least with a cell size of 100. Yes, there is a significant difference in Table 2 between abstract and adventures, but I wouldn't find this one difference a compelling finding - after all, there are 33 pairwise comparisons - I'm not surprised that one is significant...
I think it is an interesting study. One question that intrigues me is since each game was fixed, we know what the optimal series of choices was for each game (either based upon card outcome or by expected values updated after each card is revealed). When participants performed sub-optimally in each condition, we could check if it was because whether they took too much or two little risk in their decisions to stay or return to base camp/leave the building. An interesting analysis would be to see if players acted more "riskily" in one condition over another.
Hi Eric! As you indicate, the Likert scale items do not show any real significant differences. There are significant differences in the behavioral data; in terms of effect size they tend to be small to medium, but the fact that there are any, particularly at the beginning, shows that theme does have an impact--enough to perhaps affect people's initial enjoyment of the game, to where they might bounce off of it after the first play or two. There's a pre-print of this paper here: https://doi.org/10.31234/osf.io/syjm9 (this takes you to an ArXiv site) that will show you what's truly significant (mostly ANOVAs)
Oh, and thanks for the idea of the optimal/sub-optimal comparison. I'll have to think more about how best to do it, but there may be something there. Thanks!
@geoff it's nice to see any data on these topics, and this is definitely a fascinating one.
Forgive me if my statistics are rusty, but +/- 1 SE shows that the true population value for these measures has a ~68.2% chance of falling within the error margins right? Do you know what these look like at a 95% confidence interval?
Hi Matt! You're right, some of these differences are non-significant, but there are some that fall into that p < .05 level. A pre-print of this paper can be here: https://doi.org/10.31234/osf.io/syjm9 (this takes you to an ArXiv site). This will give you more insight into what significant and what's not, along with effect sizes. Thanks for your comments!
It would be interesting to look at the variance of the scores. did the "firefighters" took more risks and more of them finished the game with 0 points, lowering tha average? Or did they play more safe and rescued fewer "points", trying to at lease save some people from the fire?
I see a difference between "If you push your luck too much you die, and loose the treasure you collected so far" and "you die, and so do all the people you rescued so far".
Hi all! This is Stephen Blessing, the person who did the experiment. Special shout-out to my undergrad assistant who tested most of the participants, Elena Sarkosky. She also did the additional art we created for the variants and the backdrop (I didn't know when she came on board she had some artistic talents!).
If you have questions, feel free to post here obviously, or send me an email at sblessing at ut.edu. A pre-print of the paper can also be found here: https://doi.org/10.31234/osf.io/syjm9 (this takes you to an ArXiv site; this version only has the first experiment). This will find a journal home someday, but it kind of falls between the cracks of topics that cog psych journals find interesting (where I mostly publish).
I have to agree with Matt. It appears you may be over-generalizing from the data. Definitely by Game 4, there are no differences between time spent playing the games. I think one big takeaway would be Theme hardly matters. (by game 4, theme doesn't seem to be impacting time spent processing - however, it might actually still be hindering performance).
In Table 1, the "rules are difficult to understand" is insignificant across all conditions - at least with a cell size of 100. Yes, there is a significant difference in Table 2 between abstract and adventures, but I wouldn't find this one difference a compelling finding - after all, there are 33 pairwise comparisons - I'm not surprised that one is significant...
I think it is an interesting study. One question that intrigues me is since each game was fixed, we know what the optimal series of choices was for each game (either based upon card outcome or by expected values updated after each card is revealed). When participants performed sub-optimally in each condition, we could check if it was because whether they took too much or two little risk in their decisions to stay or return to base camp/leave the building. An interesting analysis would be to see if players acted more "riskily" in one condition over another.
Thanks for the feedback! I'm going to see if I can get access to the raw data so more calcs like this can be done.
Hi Eric! As you indicate, the Likert scale items do not show any real significant differences. There are significant differences in the behavioral data; in terms of effect size they tend to be small to medium, but the fact that there are any, particularly at the beginning, shows that theme does have an impact--enough to perhaps affect people's initial enjoyment of the game, to where they might bounce off of it after the first play or two. There's a pre-print of this paper here: https://doi.org/10.31234/osf.io/syjm9 (this takes you to an ArXiv site) that will show you what's truly significant (mostly ANOVAs)
Oh, and thanks for the idea of the optimal/sub-optimal comparison. I'll have to think more about how best to do it, but there may be something there. Thanks!
@geoff it's nice to see any data on these topics, and this is definitely a fascinating one.
Forgive me if my statistics are rusty, but +/- 1 SE shows that the true population value for these measures has a ~68.2% chance of falling within the error margins right? Do you know what these look like at a 95% confidence interval?
Thanks for the feedback! I'm going to see if I can get access to the raw data so more calcs like this can be done.
Hi Matt! You're right, some of these differences are non-significant, but there are some that fall into that p < .05 level. A pre-print of this paper can be here: https://doi.org/10.31234/osf.io/syjm9 (this takes you to an ArXiv site). This will give you more insight into what significant and what's not, along with effect sizes. Thanks for your comments!
It would be interesting to look at the variance of the scores. did the "firefighters" took more risks and more of them finished the game with 0 points, lowering tha average? Or did they play more safe and rescued fewer "points", trying to at lease save some people from the fire?
I see a difference between "If you push your luck too much you die, and loose the treasure you collected so far" and "you die, and so do all the people you rescued so far".
Hi all! This is Stephen Blessing, the person who did the experiment. Special shout-out to my undergrad assistant who tested most of the participants, Elena Sarkosky. She also did the additional art we created for the variants and the backdrop (I didn't know when she came on board she had some artistic talents!).
If you have questions, feel free to post here obviously, or send me an email at sblessing at ut.edu. A pre-print of the paper can also be found here: https://doi.org/10.31234/osf.io/syjm9 (this takes you to an ArXiv site; this version only has the first experiment). This will find a journal home someday, but it kind of falls between the cracks of topics that cog psych journals find interesting (where I mostly publish).
Thanks for everyone's interest!
My takeaway - themes matter less than most people think, and we should make more abstract games. :)