An interview
We recently talked with the author of EJPs first Registered Report, Colin Vize, whose article titled, “Merging structural and process-related approaches to the study of agreeableness: A preregistered replication and extension attempt”, recently appeared in the March/April 2020 issue of EJP. Colin is a clinical psychology PhD candidate at Purdue University.
Read on to learn more about Colin’s Registered Report!
Q: Hello Colin! Can you tell us a bit about yourself and how you became involved in personality psychology?
Sure. My training is in clinical psychology and initially, my interest in personality psychology was squarely focused on psychopathy. Since being in grad school, my interests have definitely expanded – which is probably typical for a lot of folks. When I got into grad school I had a narrow idea of the type of work that I wanted to do, but things have changed quite a bit over the years.
I think how I came to be interested in psychopathy was kind of a long, winding road, but I had started with a lay interest in psychopathy. It's pretty popular in pop culture contexts and I wanted to understand it more. I took some classes in undergrad, and then the more I read, the more I realized how much I didn't know, so I wanted to learn more. And then, in grad school I’ve spent a lot of time working with my advisor, Don Lynam at Purdue University, looking at psychopathy. My interests have broadened into popular models of personality and personality research more generally. So, it's definitely been a long road.
Q: Can you tell us about the process of drafting the Registered Report (RR)?
Speaking of long, winding roads! A lot of the work I was doing before the RR content-wise was focused on antagonism, or low agreeableness, and was descriptive in nature. A lot of it was cross-sectional, correlational stuff – meta-analyses and things like that. And a lot of that implicated low agreeableness as the most robust correlate of a variety of antisocial behaviors, relative to other personality traits. That is, it tended to be the most important in terms of accounting for variance in clinically relevant outcomes such as aggression and antisocial behavior.
I really enjoyed that work and it was really interesting, but it didn’t tell us a lot about how agreeableness gives rise or increases the probability of people acting in antisocial ways. The Registered Report attempted to hone in on what processes of agreeableness may tell us a little bit more about how low agreeableness increases the chance of people acting aggressively or in anti-social ways.
There's a small group of people who do research using the Five Factor Model and who have process-based accounts of the different domains of the Five Factor Model, including agreeableness. The RR project was about picking some of the behavioral tasks that are thought to be indicators of agreeableness processes and seeing whether we could replicate the initial results and extend them to empirically identified facets of agreeableness.
I also mentioned this in my Twitter thread about the Registered Report, but I think our goals for the Registered Report were quite a bit more modest than some large-scale replication projects. I think it makes most sense to think of it as sort of a measurement validation-type paper, asking the question: "Do these operationalizations, these tasks, reliably index processes of agreeableness?"
So that was all the motivation for this project. The process itself was initially very taxing. It was a lot of work on the front end. I had never done it before, and I think the Stage 1 RR manuscript went through 3 or 4 rounds of revision. Ultimately however I think it was super helpful and pretty illuminating for me. The reviewers we had were well-versed in RRs, so it was helpful to see how they thought about the necessary steps that needed to be included in a RR, how to think through all of the analytical decisions that go into a paper like this, et cetera. Relative to all the other work that I have done, it probably has had the biggest impact in terms of how I do research and how I review papers. But that being said, it was a lot of work on the front end.
However, the back end felt so much easier! Everything is set and the code is written already, so the turnaround after getting the data is much faster; you just plug in the data, you have your analysis scripts ready to go. So it was a really interesting process overall. I also think it was more collaborative, in terms of working with reviewers, working with your coauthors, and working with Dr. Mitja Back at EJP.
It was also nice knowing that your job was to think pretty deeply about your project and that as long as you did that and you followed your pre-registered plan, your results were going to be published no matter what.
Q: How did you come up with the idea of doing the RR?
I guess I was thinking that given my research program in grad school to that point, the next step seemed to be to better understand how agreeableness gives rise or increases the chances of people behaving in antisocial ways. But before jumping into that, I had to assess how robust and reliable the measures are that we have used so far to measure some of the processes of agreeableness.
When I learned about RR, it seemed like a natural first step. RRs are a powerful tool that we have, so I wanted to take it and apply it to my own research to see if we could replicate and extend previous work by looking at facets of agreeableness.
I also wanted to add the Registered Report as a tool in my toolbelt. I, like a lot of folks in psychology, have been following the Open Science movement really closely. As I move into being an early career researcher, I want to be able to demonstrate that I have an understanding of why doing an RR and engaging in Open Science practices more generally is important, and how it can be used to improve the quality of research.
Q: You mention the RR being a tool in your tool belt. Would it also be a tool that you consider using again?
I think so. Since doing this RR, I've also done a few pre-registrations for a variety of different studies and I think there's different levels of rigor one can bring to pre-registration, RRs, and things like that. I would definitely be open to doing another RR, but I also think it's a tool that – like most other tools – is best used for specific questions where it really is going to strengthen the research. So I think it can be context-dependent.
Q: You already mentioned the motivation of the study. Can you tell us more about the study itself?
In terms of the design, we picked two behavioral tasks that are thought to index processes of agreeableness at different stages of processing. One was a task that tried to operationalize situation selection. This was a picture viewing task in which participants viewed one hundred different pictures from the International Affective Picture System. How long they viewed pictures was the primary outcome for that paradigm. The second task that we looked at was a spatial attention paradigm. This was an extension of the Posner cueing paradigm. Pro-social or anti-social cue words were first shown. The pro-social and anti-social words were shown to the left or right of a fixation cross. Participants had to identify the cues as either pro-social or anti-social and then respond to a target that appeared either on the same side or on the opposite side of the cue word.
For both of these paradigms, the original results found interactions between agreeableness and features of the tasks. For the picture viewing task, it was found that picture valence – whether the picture was positive or negative – interacted with agreeableness, such that people who were low on agreeableness viewed negatively-valenced pictures for longer. People high on agreeableness didn't show such a preference. In the spatial attention task, it took longer for people low on agreeableness to disengage from anti-social cue words and identify the target.
To attempt to replicate these findings, we set up the tasks for an online sample in addition to some other self-report measures. We had two measures of agreeableness: the Big Five Markers and the International Personality Item Pool-NEO (IPIP-NEO) agreeableness scale. The Big Five Markers was the measure used in the original studies and the IPIP was the measure that we wanted to use to examine facet relations. We also had a lot of self-report measures on anti-social behavior as we wanted to map those on to some of the behavioral tasks that we had included.
We weren't able to replicate anything with the IPIP agreeableness scale, which was the primary goal for our analyses. We had additional multilevel models that used the Big Five Markers and the exact same parameters as the original models. For these models, we replicated the sign of the interaction in the picture viewing task, but the nature of the interaction wasn't quite the same. That is, we found that both people low on agreeableness and high on agreeableness showed a preference for negatively-valenced pictures. In addition, the magnitude of the effect was quite a bit smaller.
There are some interesting ways to talk about what it means to replicate and how you make sense of all the different types of results– and we talk about this quite a bit in the paper. For instance, what counts as support for a theory or a hypothesis? But by and large I think it’s fair to say that we weren't able to replicate any of our focal hypotheses.
Some of the implications of our findings are that perhaps these behavioral tasks might be difficult ways to assess processes of agreeableness. But of course there are many other tasks – we only focused on two. So I think we need to be careful in generalizing to other behavioral tasks that are thought to be measures of agreeableness.
Q: What's next for you?
I want to take a step back to a broader conceptualization of agreeableness-related processes, as this study was very specific and focused on tasks that could be considered t assess fine-grained processes. For some of the work that I'm doing now, I'm working with Aiden Wright at the University of Pittsburgh. We are using EMA methods as a different approach to studying processes of agreeableness. In the RR paper, we mention that if we are going to talk about processes of agreeableness, these processes are likely to be operating in an interpersonal context. So using EMA, we want to look at interpersonal interactions and at what might be happening "in the moment". So for instance, we’re interested in how people report in the moment how they see other people and how they perceive how other people are acting towards them. So I think it's more of a macro-level approach to processes of agreeableness, compared to something like spatial processing of antisocial stimuli.
Q: Taking your experience with this RR but also your broader experience in the field into account, do you have any tips or advice that you would like to share with other young researchers?
I think I've thought about this question the most of all the questions that you sent along before the interview. I always struggle to come up with advice for younger researchers that is both applicable to most people but also not obvious.
I received some advice when I was just getting started in personality research that I found helpful at the time – and perhaps it will also be helpful for other folks. When I started grad school, I knew surprisingly little about things like the Five-factor Model and personality research more generally. I didn't work in a big lab in undergrad and I didn't have a lot of exposure to personality research. When I got to grad school, my advisor was very reassuring by letting me know that a lot of the content of personality psychology and personality theory I’d will pick up along the way. I didn’t need to be an expert right off the bat. You have a lot of time to be around very bright people and talk with them. It is really surprising how quickly you soak up information – and by the end you realize that you've learned a lot and can talk quite a bit about personality and personality theory.
Another piece of advice that we hear quite a bit – and therefore may be a bit obvious – has to do with balancing professional and personal needs. It still may be worth repeating. I love doing what I do and I feel privileged to be able to do it. I’ve also found that spending time with friends outside of academia and more generally taking care of myself and making sure that my needs are getting met – whatever they are – is important. So I’d recommend that outside of professional development and intellectual needs, you are also checking that other personal needs are being met. That helped me enjoy grad school and being a young academic even more.