Assessing the Replicability of Cognitive Psychology During Remote Experiential Learning via Mobile Phone Technology

In terms of syllabus design considerations, a number of features were key. First, students had flexible access (Little & Francis, 2005) to the Flex Labs in that they could interact with Presentation Mobile at any time during the week. Second, in the majority of cases, the Flex Lab delivered on Week x was directly related to the course materials to be discussed for Week x+1, thereby facilitating distributed learning (Litman & Davachi, 2008). For example, a Flex Lab examining the relationship between hemispheric specialization and emotion was completed the week before course materials on Cognitive Neuroscience. Third, participation marks (1% of course grade) were allocated to each Flex Lab, with a bonus 1% for engagement with all 9 Flex Labs, thereby providing external motivation for course participation (Gallo & Rinaldo, 2011). Targets for replication were largely driven by two pragmatic concerns: a) the presence of existing materials used in previous iterations of in-person versions of the Cognitive Psychology course, and, b) the pressures in converting these materials to remote experimental packages such that they were ready on time for our first fully on-line semester of Fall 2021 (used as a pilot semester) prior to more formal data collection in Winter 2021.

Each of the resultant 8 Flex Labs data sets across three semesters of on-line teaching are now summarized with respect to: their relationship with the course materials, the methodology used, the average effect sizes generated either in support or in refute of the original observation across three semesters, and, a comparison with the original effect size (when available). Table 1 provides a summary of effects. Fixed effects models were used since it was assumed the true effect size should be the same across all three points of data collection (Winter 21, Fall 21, Winter 22). Random effect models are preferred when there is heterogeneity across data collection (Barili et al., 2018). Experimental packages and summary data freely available via the Open Science Framework (https://osf.io/kv5qp/). The Flex Lab data illustrate a number of points related to the empirical health of Cognitive Psychology in the context of the replication crisis, tackled in the General Discussion.

Within the context of Cognitive Neuroscience, Levy, Heller, Banich & Burton (1983) discuss the potential lateralization of emotional processing, using a chimeric face procedure. Chimeric faces are created by vertically cutting and pasting two versions of the same face: one with and one without emotion. As such, two copies of each face exist, one where the left side is neutral and right side emotional, and, where the right side is emotional and the left side neutral. Levy et al (1983; see also Rueckert & Naybar, 2008) inferred a preference for emotional processing in the right hemisphere by revealing a left-sided bias when participants were asked to choose between two copies of chimeric faces.

On each of 36 trials, a lateralized pair of chimeric faces taken from Levy et al. (1983) were presented. Each faces juxtaposed a neutral side and happy (smiling) side. On 18 trials, when emotion appeared on the left-side of the chimeric face, this face was presented on the left. On 18 trials, when emotion appeared on the left-side of the chimeric face, this face was presented on the right. Trial order was individually randomized for each participant. Responding was self-paced, with participants pressing either the left or right side of their device to “select the face that you think looks the happiest.” After 36 trials, participants were indicate whether they were left- or right-handed. The degree of bias was calculated from the formula for Laterality Quotient (LQ; Rueckert & Naybar, 2008), which was [[right responses – left responses] / 36]. As shown in Figure 2a, across all three semesters, a small effect (Cohen, 1988) was consistently observed in support of this hypothesis, resulting in an average effect size estimate of 0.262 [CI -/+ 95%: 0.177 – 0.347; n = 551]. This is relative to the large effect originally reported by Levy et al. (1983) of d = 0.689 [CI -/+ 95%: 0.480 – 0.894; n = 111].

For the topic of Perception, the visual search paradigm and feature integration theory (FIT; Treisman & Gelade, 1983) were introduced. As summarized by Wolfe (2001), asymmetries exist during visual search performance, in that it is easier to detect unknown targets defined by the presence rather than absence of a feature. For example, finding a Q amongst a series of Os is easier than finding an O amongst Qs, since the Q contains an additional diagonal stroke.

On each of 24 trials, participants were presented with a visual search display, containing 20 spatially diffuse elements. 19 of those elements were identical (distractors) and 1 was unique (target). The stimuli used included O vs. Q, forwards vs. backwards letters, O vs. C, rotated vs. non-rotated letters, and, diagonal vs. vertical lines. For 12 feature present trials, one element was assigned target and the other element was assigned distractor (e.g., 1 Q, 19 Os). For the other 12 feature absent trials, the assignment of element to target and distractor was switched (e.g., 1 O, 19 Qs). Trial order was individually randomized for each participant. Responding was self-paced, with participants asked to “as fast as possible, press the odd one out.” A robust, large effect size was consistently observed in support of the faster detection of a single target defined by the presence rather than absence of a feature (average effect size estimate = 0.887 [CI -/+ 95%: 0.799 – 0.976]; n = 668; see Figure 2b).

Andrade (2010) reports a novel finding of dual-task benefit rather than dual-task decrement, in the context of Attention. Specifically, participants were asked to monitor a telephone message and to note the names of individuals who would either definitely or probably be coming to a hypothetical party. In contrast to a single task group, participants in another group (dual task group) were also asked to ‘doodle’ (fill in shapes) as they were listening to the message. Not only were the doodle group observed to have better intentional memory of the party-goers (Andrade, 2010, n = 40; Hedges g = 0.826 [CI -/+ 95%: 0.094, 1.377]), but in a surprise memory test, the doodle group also performed better at incidentally remembering locations that were mentioned during the message (ibid; Hedges g = 0.416 [CI -/+ 95%: 0.205, 1.049]).

A variant of the original Andrade (2010) message was recorded, specifically replacing UK city locations with major Canadian cities. Participants were allocated to either the non-doodle (single-task) or doodle (dual-task) group based on the first letter of their last name (A-M, or, N-Z, respectively). Participants in the dual-task group were further instructed to download a version of the Doodle sheet. Both groups were then directed to https://www.youtube.com/watch?v=_YUbZ_Qwp4c. The video contained instructions based on Andrade (2010), and an audio reading of the modified phone message (approximately 2m30s) during which time, participants were encouraged to note who would likely be coming to the party. An incidental memory prompt then appeared for approximately 1m, asking participants to write down any city names they could remember. All individuals were then directed to Presentation Mobile. This script prompted them to record their group allocation, followed by the presentation of 8 names and 8 places. For each of the 16 trials, they were asked to indicate whether they had remembered each specific item. Trial order within each category was individually randomized and responding was self-paced. Our data reliably produced data contrary to the observation that intentional memory was facilitated by concurrent doodling, as indexed by a negative average effect size = -0.340 [CI -/+ 95%: -0.497 – -0.182] (n = 653; see Figure 2c).

The role of schema in the topic of Everyday Memory was explored using the study by Pichert & Anderson (1976). Here, participants were provided with a perspective from which to imagine a scenario regarding the details of a house: either imagine the passage as a buyer or burglar of the house. When tested on buyer-related and burglar-related items contained within the passage, the influence of perspective on memory was demonstrated in that perspective-consistent information (e.g., buyer – leaking roof) was better remembered than perspective-inconsistent information (e.g., burglar – leaking roof). Furthermore, when participants were asked to switch rather than maintain perspectives, this ‘unlocked’ new information about the passage unavailable at the time of initial retrieval.

Participants were assigned to 1 of 4 groups on the basis of the first letter of their last name (A-D, Buyer-Buyer; E-K, Buyer-Burglar; L-P Burglar-Buyer; Q-Z, Burglar-Burglar). Groups were then directed to watch a tailored YouTube video, where they were instructed to listen to an audio version of the Pichert & Anderson (1976) narrative (approximately 2 minutes) from a particular perspective (Buyer or Burglar), eg, Group A: https://www.youtube.com/watch?v=pASKfnJ00O4. Participants then had two minutes to “write down as much of the exact story as you can.” This was followed by another two minute tailored prompt that either asked participants to continue to recall the story from the same perspective (Buyer – Buyer, Burglar – Burglar), or, to switch into a different perspective (Buyer – Burglar, Burglar – Buyer). All individuals were then directed to Presentation Mobile and prompted to record their group allocation. This was followed by the presentation of 12 Buyer relevant items and 12 Burglar relevant items in a random order, with participants responding as to whether they had remembered them. Trial order within each category was individually randomized and responding was self-paced. To finish the study, participants were invited to completed both the Vividness of Visual Imagery Questionnaire (VVIQ; Marks, 1973) and the Object-Spatial Imagery Questionnaire (OSIQ; Blajenkova et al., 2006). In testing the hypothesis that switching perspectives increased the recall of initially irrelevant information, the data across the three semesters of PSYCH258 were highly variable (see Figure 2d; n = 619), resulting in an ambiguous average effect size = -0.016 [CI -/+ 95%: -0.188 – 0.155]. The reasons for this variability will be discussed later.

A perennial question regarding Long-Term Memory is its limit: exactly what level of detail is retained within memory, and, for how long? Brady et al. (2008) provided provocative data suggesting that high fidelity details regarding visual objects are retained within long-term memory, even when those objects are only viewed once. Specifically, 2896 objects were presented across a 5-6 hour period, in which participants were required to count the number of object repetitions that occurred. Following that lengthy period of encoding, participants were shown a further 300 pairs of objects, with one object in the pair being previously seen. Brady et al. (2008) were able to assess the level of detail retained within long-term memory by manipulating the degree of similarity between old and new objects. In novel pairs, the new object was completely new (e.g., a chocolate cake versus a toy train). In exemplar pairs, the new object was a different version of the object (e.g., two chocolate cakes). In state pairs, the new object was the old object in a different configuration (e.g., the same chocolate cake but with a slice taken out). Memory performance for novel, exemplar and state pairs was 93%, 88% and 87%, respectively.

Data collection took part across two separate weeks of the course. For the first week, participants were shown a sample of 144 images from Brady et al. (2008) at a rate of 1 every 3 seconds. 120 images were shown only once, whereas 12 images were shown twice (2 x 12 = 24). During this initial phase of the study, participants were explicitly asked to count the number of image repetition they think they saw. To end their contribution, they selected a range that contained the number of repetition they thought they saw (0-2, 3-5, 6-8, 9-11, 12-14, 16-17). These repeated objects played no further role in the study.

For the second week of data collection, participants had to indicate whether they participated in the first week. Regardless of their response, participants were then shown 120 pairs of images. In each pair, there was an image they had seen during the first week along with a new image. This new image was a novel object, a different exemplar of the same object category, or, the same object in a different state. There were 40 trials of each of the 3 kind of pairs. Responding was self-paced and image pairs were presented in individual random order. To move on to the next trial, participants selected the side of the screen containing the old object (and were encouraged to guess). Although a straightforward replication of Brady et al. (2008) was impossible given the constraints of teaching, re-testing 120 objects across a two week interval revealed a small reliable effect in favour of better than chance recognition of old objects in the context of state pairs: average effect size estimate = 0.314 [CI -/+ 95%: 0.227 – 0.400; n = 543; see Figure 2e].

Key practicalities arising from the course discussion of Semantic Memory are the ways in which memory can be better remembered. Specifically in the context of education, it is a pertinent question what kind of feedback enables memory improvement. Pashler, Cepeda, Wixted & Rohrer (2005) examined three different classes of feedback over a variety of retention periods. In response to learning a novel Lugandan dialect, they compared no feedback, with the feedback of being ‘correct’ or ‘incorrect’, with the provision of the correct answer. They found that performance following ‘correct’ / ‘incorrect’ messaging was actually worse than no feedback, but that providing the correct answer had a benefit over no feedback. Intriguingly, the authors report that the “…basic pattern of results described above was confirmed in another, quite similar, online experiment that we carried out teaching subjects obscure facts rather than foreign language vocabulary, and using a within-subject instead of a between-subjects manipulation of the type of feedback.” (Pashler et al., 2005, p. 6). Inspired by this comment, I asked whether providing the feedback labels of ‘correct’ and ‘incorrect’ would have similarly detrimental effects, relative to a no feedback condition in the context of updating trivia knowledge.

60 trivia questions were identified from a variety of on-line source encompassing topics including Science & Nature, Religion & Mythology, Food & Drink, Art & Literature. Participants were informed that they were invited to answer 120 trivia questions: 60 at Stage 1 and 60 at Stage 2. For each Stage 1 trial, participants were shown a question with two responses: one on the left and one on the right. After pressing which response they thought was correct, they were provided with no feedback, the feedback of ‘correct’ or ‘incorrect’, or, provided with the correct answer. 20 questions were allocated to each of these 3 feedback categories, with trial order individually randomized. At the beginning of Stage 2, participants were informed that they were invited to answer the same 60 questions again, and that in all cases feedback would be delivered in the form of ‘correct’ or ‘incorrect’. We reliably produced a large effect size contrary to this claim (-0.951 [CI -/+ 95%: -1.072 – -0.831]; n = 572; see Figure 2f).

In our discussion of Decision Making, the class was introduced to how human decisions can be sub-optimal. To demonstrate this concept, the three-response zero-sum game of Rock, Paper, Scissors was used. Here, the only way to guarantee not being beaten in the game is to play according to mixed-strategy. This is where individual items are played with equal probability, and there is no association between the previous outcome and current response (Dyson et al., 2016). However, operant conditioning principles such as lose-shift continue to influence game performance in sub-optimal ways. Specifically, Dyson et al. (2016) noted that the probability of changing responses is more likely after losing relative to winning a RPS round (ie, lose-shift > win-shift; (Hedges g = 0.788 [CI -/+ 95%: 0.379, 1.228]).

90 trials of Rock, Paper, Scissors were delivered over the Presentation Mobile platform. Following instructions, participants tapped a left, middle or right location to indicate a Rock, Paper or Scissors response. Following a 500 ms pause, both computer (left) and participant (right) response were shown for 1000 ms (using stimuli adapted from Forder & Dyson, 2016). Following another 500 ms pause, trial outcome was reported as ‘WIN (+1)’, ‘LOSE (-1)’ or ‘DRAW (0)’ for 1000 ms. Scores were updated during a final 500 ms interval, and another response was requested. The computerized opponent played 30 Rock, 30 Paper, 30 Scissors responses in an individually randomized order. A small effect size consistent with larger lose-shift relative to win-shift proportions was also observed across the three semesters of PSYCH258 (0.295 [CI -/+ 95%: 0.209 – 0.382]; n = 560; see Figure 2g).

A final, classic Cognitive Psychology finding relates to Visual Imagery and behavioural data consistent with the notion of mental rotation. From Cooper & Shepard (1973), participants were asked to categorize shapes with varying degrees of rotation, as either forwards-facing or backwards-facing letter Rs. Of principle concern was whether the degree of effort involved in physical rotation was akin to the degree of effort involved in mental rotation. If this was the case, then the further a shape was rotated away from its canonical position (i.e. 0 degrees), the longer it should take to rotate back in order to confirm letter categorization.

12 images were shown via Presentation Mobile: a forward-facing R or backwards-facing R (categorization x 2), rotated by 0, 60, 120, 180, 240, 300 degrees (rotation x 6). Each image was shown 15 times in a random order, resulting in 180 trials in total. Participants were asked to be both fast and accurate, pressing the left side of their screen if the shape was a forwards-facing R, and, the right side of their screen if the shape was a backwards-facing R. Feedback was not provided, and a 500 ms pause was introduced between each stimulus presentation. Only correct RTs as a function of rotation were considered in analysis. Reliably large effect sizes were shown across Winter 2021, Fall 2021 and Winter 2022 samples (average effect size = 1.039 [CI -/+ 95%: 1.009 – 1.070]; n = 518; see Figure 2h).

The recent requirement to move both teaching and research from in-person to on-line presented an opportunity to conduct an environmental sweep on the empirical health of Cognitive Psychology. In the context of an undergraduate course, 8 empirical studies were selected- each representing a classic or contemporary observation directly linked to course materials- and an on-line version of each study (Flex Lab) was implemented via Presentation Mobile. Running across three semesters, the average sample size for each study was 585- well exceeding the recommendation that attempted replications should have a sample size 2.5 times the original (Simonsohn, 2013), when such information was available.

There were two Flex Labs (Faces, RPS) where an estimate of the original effect size was available and the current data provided support for the original hypothesis. The current effect size was at least half the size of the original effect size (Faces, Levy et al. (1983) d = 0.689 versus current d = 0.262; RPS, Dyson et al. (2016) g = 0.788 versus current g = 0.295). Such observations are consistent with the likelihood that published effect sizes are an overestimate of true effect size, since- historically- studies finding non-significant results are not published to the same degree (Nuijten et al., 2015; Schäfer & Schwarz, 2019; see also Serra-Garcia & Gneezy, 2021). Furthermore, two additional Flex Labs (Search, Rotate) may seem relatively uncontroversial in terms of providing both robust and large effect sizes in support of speeded reaction time for the detection of unknown targets defined by feature presence rather than absence (Search; after Wolfe, 2001), and, slowed reaction time as the degree of mental rotation increases (Rotate; after Shepard & Metzler, 1971). The observation of such robust and large effect sizes might lead us to place some confidence in the empirical health of certain key (albeit selective) findings within Cognitive Psychology (Open Science Collaboration, 2015; see their Table 1). A cautionary note is warranted however in terms of the absence of accuracy data for the Search lab. Similarly, in the case of Rotate, the extent to which speed is sacrificed for accuracy during mental rotation remains an open question (eg, Liesefeld et al., 2015).

There were also two cases (Doodle, Trivia) where current data reliably provided evidence refuting the original hypothesis. For the Doodle Flex Lab, we consistently found evidence for a dual-task decrement rather than dual-task benefit with respect to the auditory memory of a telephone message and the visuo-motor act of doodling (after Andrade, 2010). This discrepancy highlights the difficulties in reconciling attempts at replication that yield contrary observations in terms of the inclusion or exclusion of auxiliary assumptions. Hudson (2021) provides the example where a researcher erroneously assumes laboratory temperature is an irrelevant aspect of a replication design, and then fails to find evidence in favour of the original hypothesis as a result of setting laboratory ambient temperature at -25C (see also Feest, 2019, for a similar example). Undoubtedly, there were similar- albeit less extreme- assumptions made in re-implementing Andrade (2010). First, the original study (conducted in the UK) contained both putatively Caucasian names (eg, Claire, William) and UK towns (eg, Colchester, Peterborough) as the target of monitored and incidental recall, respectively. Since the current version was completed in Canada, it was considered important to change UK towns to Canadian cities (eg., Montreal, Vancouver). Second, and because of the first, the audio message provided in the original Appendix was re-recorded. Third, participants in the original Doodle study were asked to complete this study directly after completing another, as a way to increase boredom “by testing people who were already thinking about going home.” (Andrade, 2010, p. 101). This is in direct contrast to the current study, where it could be argued individuals were actively seeking participation. Thus, if boredom is a key mechanism driving the effect, I have failed to adequate represent the original study. Fourth, in the original study participants were paid for participation (as opposed to given course grade). In sum, a failure to replicate may always be attributed to changes in auxiliary assumptions. A central question for all future research is whether in-person rather than on-line data collection is an essential auxiliary assumption for replication. The interested reader is also directed to Murre’s (2021) recent failure to similarly replicate Godden & Baddeley’s (1975) ‘classic’ finding of context-dependent memory between being on land and underwater. To Murre’s (2021) credit, a number of auxiliary assumptions (of unknown importance) are noted in the attempted replication: the use of an indoor pool versus open water, the amateur or professional status of the divers, testing divers in a single day relative to across 4 days, to name but three.

In the case of the Trivia Flex Lab, we again found evidence contrary to the position of Pashler et al. (2005) that the delivery of ‘correct / incorrect’ feedback was detrimental to the retention of trivia information relative to the provision of no feedback. Here, however, the degree to which the inclusion or exclusion of auxiliary assumptions may determine the direction of the effect cannot be properly assessed due to an absence of detail in the original study itself. The footnote reported in the Results section above (Pashler et al., 2005, p. 6) is the extent of detail related to the original study. Hence, exact (or, direct) replications are never possible for studies for which original details cannot be retrieved. As such, replications are immediately restricted to the category of experimental or conceptual replications (see Hudson, 2021). As such, we offer our Trivia package (and all other packages) via the Open Science Framework to enable a direct replication of this work, itself reflecting a conceptual replication of the original Pashler et al. (2005) study.

A pragmatic note on some difficulties associated with implementing on-line versions of paradigms. It should be noted that for the Object Flex Lab, neither a direct nor a conceptual replication of the Brady et al (2008) work was possible due to the significant time demands of the original study (approximately 6 hours). Hence, we are left with a different demonstration that- over a longer time period with a smaller sample of objects- individuals are able to remember specific object states at a level greater than chance. This is opposed to (but does not necessarily contradict) the original conclusion of the Brady et al (2008) study, where long-term memory for novel, exemplar and state objects was close to ceiling. Finally, in the case of the Winter 2021 and Fall 2021 House lab data, it was apparent that the script failed to accurately record group allocation, due to errors in touch screen allocation. Thus, the script for the House Flex Lab was re-written for the Winter 2022, rendering the comparison between semesters ambiguous.

In summary, this project attempted to meet the challenges of the configuration of both Cognitive Psychology teaching and research to on-line formats. There are numerous ways in which Open Science and reproducibility or replicability can be integrated into higher education, yet all share the ultimate goals of positively affecting students’ engagement, attitudes toward science and scientific literacy (see Pownall et al., 2023, for a review). For example, the Collaborative Replications and Education Project (CREP) takes a student-led rather than instructor-led approach to the preparation of attempted replication materials (see Wagge et al., 2019, for an example). The current solution of how to provide experiential, replication crisis learning during remote and asynchronous delivery was provided by the development of portable experimental packages written for mobile phone via the Presentation Mobile app. These Flex Labs represented attempted replications of classic and contemporary findings within the literature, which also explicitly connected with course material. The resultant data not only further allowed students to question the degree to which replication is possible within empirical science but also potentially contribute to the Cognitive Psychology literature in highlighting both robust (eg., Rotation; cf. Shepard & Metzler, 1971) and potentially ambiguous (Trivia; cf. Pashler et al., 2005) previous findings. The Flex Lab program provides a framework for supporting future replication attempts utilizing large on-line samples both in and beyond Cognitive Psychology, with the precise operationalization of the study preserved by the open access availability of study packages (https://osf.io/kv5qp/).

B.J.D. designed the studies, analysed the data and wrote the manuscript.

The authors would like to thank Dr Peggy St Jacques (University of Alberta, Canada), Dr Julia Spaniol (Toronto Metropolitan University, Canada) and Dr Graham Hole (Sussex University, UK) for providing some the materials related to the studies, and, Ver-Se Denga for providing the trivia questions. The author would also like to acknowledge the Teaching Assistants for PSYCH258: Yafei Qi & Eamin Zahan Heanoy (Fall 2020), Michelle Tomazck & Arturo Perez (Winter 2021), Sijie Ling and Eamin Zahan Heanoy (Fall 2021), and, Eunchan Na and Yajing Zhang (Winter 2022). Part of the data were presented at the 2021 and 2022 Festival of Teaching and Learning, University of Alberta, Canada.

There was no funding.

The author declares no competing interests.

Materials for all experiments are available at https://osf.io/kv5qp.

Summary data for all experiments are available at https://osf.io/kv5qp.