Collections²: Using “Crowdsourcing” within Psychological Research

In contrast to many special issues in traditional journals, the common theme of the articles that will be included in the Nexus is the methodological approach rather than the substantive topic. Specifically, each empirical paper will involve several researchers who each collect data at independent sites and who aggregate all of the collected data into a common analysis (most likely a meta-analysis). For the current Nexus we call each multi-site study a “Collection²” (pronounced merely as “collection” but denoted as a type of crowdsourced research project by the capital C and the exponent). The name “Collection²” succinctly describes the methodological approach of these projects because there is a collection of researchers who each collect data at their individual site (i.e., these are collections of collections of data). The current Nexus also will be open to, for example, theoretical critiques of crowdsourced research, commentary articles on the promise or benefits of crowdsourced research, re-analyses of already-completed Collections², and meta-science articles that are relevant to crowdsourced research.

Here is a non-exhaustive list of the types of Collections² that are possible for inclusion in the current Nexus and a hypothetical example of each.

1. Collections² with Identical Operationalizations: A project wherein several researchers simultaneously conduct operational replications of a previously-published effect or of a novel (i.e., not previously-published) effect. These projects can test one effect (such as some RRRs) or can test several effects within the data collection process (such as the ManyLabs projects). Because the operationalization of the key variables is the same across labs, this design is excellent for estimating the magnitude and precision of an observed effect with a specific operationalization. Example: Researchers at five different data collection sites test the “ego depletion” hypothesis using the same method and same variable operationalization at each site. A random-effects meta-analysis is used to aggregate the results.

2. Collections² with Multiple Operationalizations: A project wherein there is a common hypothesis that will be simultaneously tested at several different sites, but there are several different operationalizations of how the effect will be tested. The to-be-tested hypothesis can either be one that has been previously-published or not. These projects would test the conceptual replicability of an effect and whether the effect generalizes across different operationalizations of the key variables. Example: In a test of adaptive memory, ten researchers each test the hypothesis that individuals are better able to recall information that is processed in terms of its survival relevance. After agreeing on a common study design to facilitate the planned meta-analysis, each research team is allowed to test the focal hypothesis in whatever manner they deem appropriate. The overall project results in 10 different operationalizations of the focal hypothesis. The resulting meta-analysis estimates the population effect size and examines whether the particular operationalizations affected the estimated magnitude of the effect.

3. Construct-themed Collections²: Projects where researchers are interested in a common construct and several researchers collect data on several outcomes associated with the target construct. Example: Five individual researchers are each interested in the construct of Subjective Well-Being (SWB); however, each researcher is interested in a slightly different hypothesis involving that construct. They conduct a study that includes a common measure of SWB as well as several other measures that were chosen based on the individual researchers’ interests. Care is taken so the included effects would not interfere with one another when combined into a common data collection procedure. The researchers collect a sample from each locally-available subject pool, combine the data, and jointly publish a SWB-themed Collection². The resulting publication contains a series of meta-analyses estimating the relationship between SWB and the several other constructs.

4. Population-themed Collections²: Projects where contributing researchers have a common interest in a specific population. This sort of a collaboration can test one or more hypotheses and would be ideal for researchers who study hard-to-recruit populations and want to maximize these participants’ time. Example: Five researchers have an interest in students with disabilities; however, each researcher has a slightly different interest in hypotheses involving that population. Because students with disabilities typically have a low prevalence in each locally-available subject pool, gathering large and representative samples is difficult. The researchers collect a sample from each locally-available subject pool, combine the data, and jointly publish a students-with-disabilities-themed Collection². The resulting publication contains a series of meta-analyses of several effects involving students with disabilities.

5. Other Types of Collections²: Because of the diversity of the projects that can conceivably fall under the broad umbrella of crowdsourced research, the line-drawing of which articles can be included is difficult to do with certainty (e.g., a minimum number of labs needed to be considered “multi”-site, etc.). Researchers who have an idea for a possible Collection² and are unsure whether it is appropriate for this Nexus are encouraged to contact us to discuss your idea.

Unlike special issues in traditional journals, the Nexus format allows articles to be submitted for publication as they are completed. In short, authors do not have to wait for their projects to be evaluated or published at the same time as other projects (i.e., there is no submission deadline or “publication date” of the special issue). We believe this open-endedness in the timeline will be beneficial because we expect variability in how long different projects will take to complete. Importantly, there is no reason for researchers to opt out of these projects because they do not believe they would be completed in time for inclusion in the Nexus.

The Author Processing Charges (APC) and APC waiver process for Collabra: Psychology also apply to this Nexus. This Nexus is open to all areas of psychological research. In fact, we hope to see submissions that include diversity in the areas of psychology, the research questions addressed, the methods, etc. Randy McCarthy and Chris Chartier will be the lead editors for this Nexus. We will serve as the point of contact for general questions about this Nexus. However, we may enlist ad-hoc editors to provide domain-specific knowledge for some proposals outside our areas of expertise. These ad-hoc editors would serve as the point of contact for specific projects they are handling.

Each Collection² will have a researcher who is designated as the corresponding researcher. Corresponding researchers will be responsible for submitting the articles through the Collabra: Psychology submission portal, will be the point-of-contact between Collabra: Psychology and all of the contributing researchers within each Collection², and will take a leadership role in coordinating their Collection². There are two ways to lead a Collection²: Lead a non-Registered Reports Collection² or lead a Registered Reports Collection².

Some Collections² that are eligible for inclusion in the Nexus may be in progress at the time of this announcement. Or, for whatever reason, authors may choose to design their study, collect data, and submit a manuscript to the Nexus as a traditional, post-data-collection submission (i.e., non-Registered Reports). These submissions will be given full consideration for inclusion in the Nexus and the submitted manuscript will be evaluated in a traditional peer-review process. In this case, the corresponding researcher would serve the same role as a corresponding researcher in the traditional publication process.

Some Collections² may choose to be submitted as a Registered Report. A description of the general Registered Reports track at Collabra: Psychology can be found here: https://docs.google.com/document/d/1eeXPEC_oc4OYlywC9pRt6seL2xazlT58jAtRsCuEdCI/edit. In a notable departure from the 2-stage Registered Reports process described in this document, the Registered Reports format for this Nexus will involve a 3-stage process: A pre-data Collection² proposal stage, a recruiting and registering contributing labs stage, and a post-data Collection² manuscript stage. Thus, the first and third steps will follow Collabra: Psychology’s general Registered Reports process and the second step will be unique to this Nexus. Figure 1 shows the Registered Reports format for this Nexus.

An abbreviated description of this process is provided below with an emphasis on points where the Registered Reports process for the current Nexus departs from Collabra: Psychology’s general Registered Reports process. Authors should consult the Registered Reports process for Collabra: Psychology for more detail.

In Stage 1, the corresponding researcher will submit a proposed crowdsourced research project. Based on the Stage 1 peer-review process, an in-principle acceptance (IPA) may be extended. This IPA will give the researcher permission to move onto Stage 2.

What should be included in the Stage 1 proposal? Broadly, proposed projects will be evaluated like any other study. Namely, what is the research question? How is it (theoretically, practically) relevant? And is the proposed design of the study appropriate for addressing the research question? Aside from these broader issues, there are specifics that must be included in each proposed Collection². Proposals will include an Introduction section, a Planned Methods section, and a Planned Analyses section. Each of these is described in detail below.

Introduction section for a proposed Collection². Just like any empirical study, the Introduction section should introduce readers to the relevant background necessary to evaluate the proposed study. Additionally, Collections² typically involve the use of a lot of research resources (e.g., participants, time, etc.); therefore, proposing authors must make a compelling justification why their proposed Collection² would best be addressed using a “crowdsourced” methodological approach as opposed to a traditional single-site approach. For example, it may be difficult to justify Collections² to study effects that are sufficiently established (e.g., a common Stroop task) or to study effects that are extremely speculative because those projects may not be a wise use of the collective research resources.

Planned Methods section for a proposed Collection². In addition to describing a method that addresses the research question, the Planned Methods for Collections² must address the following points:

1. What are the data collection procedures that will be followed at each contributing data collection site? This part will look much like a traditional Planned Methods section. Will each contributor follow the exact same procedures and use the same variable operationalizations? If not, how will the procedures for each individual data collection site be selected (e.g., each individual collaborator can choose how to test the stated hypothesis, the Collection² lead researcher will randomly assign a procedure to a data collection site, each collaborator will select the procedures that another collaborator must follow, etc.)?

2. How many contributing data collection sites will participate? And how many participants at each lab will participate? How were these numbers determined?

3. What is the process for determining authorship on the final manuscript? Keep in mind that not all contributing labs need to be identified prior to submitting a proposal. Thus, it is critical to outline the criteria that will be applied when determining who will be an author on the final manuscript to prevent misunderstandings or disputes around authorship later. These criteria should include who is eligible for being an author and how authorship order for the final manuscript will be determined.

4. What is the proposed timeline for the project? How long is recruitment of contributing labs expected to take? Once the labs are recruited, how long before data collection could occur? Once contributing labs are identified, how long would data collection and data analysis take? How long until a final manuscript would be submitted? We understand that some of these steps may be difficult to know in advance and that some may change. However, please provide your best estimate.

5. Does the project require funding? If so, how will funding-related issues be handled?

6. Will you offer other contributing researchers time or space in the data collection procedure as a recruitment tool (see section on Sticks and Carrots below)? For example, to recruit other contributors you may want to offer them time at the end of a survey where they can add something relevant to their personal research interests. You do not (or probably cannot) know the specifics of what these additional measures may be at the time of the proposal, but we believe it is important to be explicit if this is something you will want to offer so the peer-reviewers can consider the implications for the project’s primary focus. If time and space is planned to be used as a recruitment tool, will the analyses of these additional measures be included in the final manuscript?

7. What are the criteria for contributing labs? Can undergraduate students or graduate students be contributing researchers? Can students only contribute if they are supervised by an advisor? Is there special training, expertise, or equipment that contributing labs need? Do contributing labs need specific software?

8. Any other topics that are relevant to the proposed project. Include any information the peer-reviewers would need to evaluate the proposed project.

Planned Analyses section for a proposed Collection². The Planned Analyses for Collections² must address the following points:

1. Are you using manipulation/attention checks or positive controls? How will those be analyzed?

2. Are there any exclusion criteria (e.g., participants must answer all items, participants must pass all manipulation checks, participants can only have 5% errors, etc.)? If so, how will those exclusion criteria be applied?

3. What is your focal hypothesis test? Which results would support your hypothesis and which results would be considered unsupportive of the hypothesis? This also needs to specify the analyses that will be used to aggregate or synthesize all of the data from the different labs (e.g., a random-effects meta-analysis).

4. What is your plan for sharing data? If you cannot share data, please specify why you cannot (e.g., there would be re-identification concerns, etc.).

In Stage 2, the corresponding researcher will identify all of the contributing labs for their Collection². Prior to data collection, the lead researcher must provide a list of contributing labs and a confirmation that each contributing lab understands how authorship will be determined. Once the contributing labs and agreement on the authorship criteria are approved by the editors, a date-and time-stamped document will be posted on the Open Science Framework. The project will then be approved to begin data collection.

Recruiting contributors. Proposing authors may choose to recruit contributing labs on their own. If proposing authors want assistance with recruiting labs, the editors can assist authors with finding contributing labs by advertising on social media, using a mailing list of labs who have expressed interest in contributing to crowdsourced research, etc. Collections² with an IPA also can post their project to StudySwap, which is an online platform for researchers to find potential collaborators. The editors can assist researchers in using the StudySwap platform. Our goal is to help make each Collection² a success and we understand that getting contributing labs is an essential component of making these projects successful.

Sticks and Carrots. We believe that identifying contributing labs may be perceived as a challenge to these crowdsourced research projects. We hope this perception is not too much of a deterrent for researchers who want to lead a Collection². It is worth noting there are several things that can be done to encourage other researchers to contribute to a proposed Collection².

First, for authors who choose the Registered Reports route, an IPA is likely a great way to assuage researchers’ hesitations about contributing to a Collection². Once an IPA is obtained, the lead researcher can approach potential contributors and assure them that a successfully-executed project will be accepted for publication. Second, some researchers may offer time or space in the data collection procedure to use as inducements for collaborators (as long as this approach is approved in the initial review process). For example, suppose a primary research project will take 40 minutes and the goal is to get five total labs to contribute to a project. The lead researcher may propose a one-hour data collection procedure where the first 40 minutes will be the primary project and four 5-minute chunks of time will be offered to the other four contributors. Finally, other recruitment tools (e.g., funding, etc.) may be used so long as it is approved during the Stage 1 review process.

In Stage 3, the corresponding researcher will submit a post-data-collection manuscript. This second round of reviews will not focus on the observed results. Instead, these reviews will focus on whether the proposed methods were properly executed, the planned analyses were properly conducted, the results were properly interpreted, and the proposed data sharing plan was followed. Additionally, authors need to include a section detailing any deviations from the proposed study. Things happen during data collection that are unforeseeable. That is fine and cannot be completely avoided. The important thing is these deviations are properly documented and transparently communicated with readers.

When a successfully-completed manuscript is ready to be accepted, authors will post the accepted IPA, the list of contributing labs, the pre-data authorship inclusion criteria (showing that each contributing lab understood the authorship criteria), and any shareable stimuli and data to an online repository (e.g., the Open Science Framework). This will transparently allow readers to compare the final manuscript to what was planned and approved. Links to these materials must be included in the final manuscript.

Notably, if a Collection² is a replication of a previously-published article or is closely-aligned with an individual researcher who was not part of the project, we may solicit commentaries from original authors to be included in the Nexus. We will consider these invited commentaries on a case-by-case basis, but will be transparent with all of the parties involved.

This Nexus will provide many opportunities for individuals to contribute to crowdsourced research. Please consider joining a Collaboration² as a contributor or mentor a student who wants to contribute to an appropriate Collaboratio². Follow StudySwap on Twitter (@Study_Swap), like StudySwap on facebook, or contact us to join our mailing list to receive updates.

The Nexus also will have other ways that authors can contribute. If you are interested in writing an opinion piece or commentary on crowdsourced research in general, or of a specific Collection², please contact us with your idea. Also, given our encouragement to openly share data, we hope this Nexus is a boon to meta-science. Researchers also are free to submit re-analyses of Collections² to the Nexus. If you are interested in submitting a re-analysis, please contact us to discuss your idea prior to submission.

And, of course, an invaluable way to help make this Nexus a success is by getting the word out.

We are very excited for the possibilities of this Nexus. We hope this Nexus will generate a lot of informative data and demonstrate how crowdsourced research can be a useful methodological tool for the future of psychological science.

The authors have no competing interests to declare.

Drafting the article or revising it critically for important intellectual content: RM/CC

Final approval of the version to be published: RM/CC

The author(s) of this paper chose the Open Review option, and the peer review comments are available at:http://doi.org/10.1525/collabra.107.pr