Rewarding More Is Better for Soliciting Help, Yet More So for Cash Than for Goods: Revisiting and Reframing the Tale of Two Markets With Replications and Extensions of Heyman and Ariely (2004)

We conducted two parallel well-powered pre-registered replications of Study 1 in Heyman & Ariely (2004), with extensions. According to LeBel et al.’s (2018) criteria, our studies were classified as very close replications. We pre-registered hypotheses and analytic plans of Study 1 (https://osf.io/h9wus/) and Study 2 (https://osf.io/5j7fg/), and we provided data, study materials, and results with analysis code at https://osf.io/y9p7u/.

Due to erroneous statistics reported in the original article (Bauer & Ariely, 2021), we could not rely on the original effects for our power analyses. We, therefore, sought to recruit the maximum number of participants that was possible with the budgetary constraints of our project, which would far exceed even the most conservative effect estimates. For Study 1, we recruited a total of 2203 American participants from the United States via Amazon Mechanical Turk. According to Simonsohn’s (2015) suggestion for simpler designs, our sample size was well beyond 2.5 times larger than the original sample size. In Study 2, we employed an adjusted within-subject design that is better powered and recruited a total of 999 British participants from Prolific Academic. We summarize the key demographic information of the samples in Table 1.

Our studies followed a 3 (payment form: cash vs. candy vs. monetized candy) x 2 (payment level: small vs. medium) design. In Study 1, we employed a between-subject design, similar to the original study. In Study 2, we made adjustments to a within-subject design. In addition to the six conditions, we had two control conditions, therefore, eight experimental conditions in total.

In the original study, however, they employed the same 3 x 2 between-subject design with one control condition where no payment was introduced. In addition to the original seven conditions, we introduced a new control condition as an extension; in the original study, there was a control condition where a helper would not be paid at all. Nevertheless, the authors did not clearly report whether they explicitly instructed participants that no payment would be given for helping. We sought to address the ambiguity in the original study and, thus, had two control conditions: nonpayment-without-mention and nonpayment-with-mention conditions. In the former condition, we did not mention payment at all. By contrast, in the latter, we told participants that no payment would be provided. The explicit mention of the absence of monetary compensations would prime the money market relationship. In the original study, it was hypothesized that their control condition would induce the social market relationship and that the expected willingness to help would be similar in the control and candy conditions (social market relationship). Thus, it seems that they did not explicitly mention nonpayment, and we decided to use the nonpayment-without-mention condition as a reference group for hypothesis testing. We summarized the operationalized hypotheses in Table 2.

Following Heyman & Ariely (2004), we constructed eight scenarios in which a person was seeking someone to help load a sofa into a van. In the scenarios, we manipulated the payment form and level (see Table 3), and we used them for both studies.

In Study 1, we introduced a new dependent variable, joy, as an extension; previous studies have demonstrated that altruistic helping leads to satisfaction and happiness (Dunn et al., 2008, 2014; Weinstein & Ryan, 2010; Yamaguchi et al., 2016). Heyman & Ariely (2004) originally argued that altruistic motives would underlie helping in the social market relationship, and we predicted that the level of joy individuals in the social market relationship would experience while helping others would not depend on the amount of reward they receive (H3b). Contrastingly, the level of reward would influence the level of joy for those in the money market relationship (H3a). In addition, given that altruistic behavior (i.e., helping) leads to satisfaction, it can be predicted that people expect helpers to experience more joy in the social market condition than in the money market condition (H4). To clarify H3 and H4, we would like to note that we were concerned about the simple effect of payment level in each payment form condition for H3, and we focused on the main effect of the payment form for H4. The inclusion of the new variable, joy, allowed us to test Heyman and Ariely’s (2004) claim, using direct emotions assessment. See Table 2 for pre-registered hypotheses.

After giving consent, participants were randomly assigned to one of the eight conditions and asked to read a corresponding scenario. Then, they rated how likely others would help the person in the scenario - “How likely is the average person to help load the sofa into the van in return for …” (1 = Not likely at all, 11 = Will help for sure). Next, we asked them to indicate how likely a person who helped and did not help in the given scenario would experience joy and regret - “if a person were to decide [not to help/to help] in that scenario, to what extent do you think that person would experience [regret over not helping /joy over helping],” respectively (1 = Not at all, 11 = Extremely likely). We measured regret for exploratory purposes, and auxiliary analyses on regret can be found on the OSF project page.

Based on LeBel et al.’s (2018) criteria for evaluation of replications (see “Lebel’s criteria for evaluation of replications” in Supplementary), we compared effect sizes from our hypothesis testing with those in the original study (see Tables 4 and 5). Since Heyman & Ariely (2004) did not report effect sizes, we calculated those using reported F statistics, cell means, and standard errors (see “Effect size calculation” in Supplementary). Following our pre-registration, no data exclusion was performed. Unless explicitly mentioned in the manuscript, we did not deviate from pre-registered analytic plans. We used JAMOVI (version 1.6.3), and R (version 3.6.3) for statistical analyses. We report a 95% CI for Cohen’s d and a 90% CI for $ηp2$⁠. Because the latter can only be positive, a 90% CI would be equivalent to a 95% CI for Cohen’s d. Our preregistered analyses were produced using JAMOVI, and we used R codes to compute effect sizes and their CIs that JAMOVI did not calculate (e.g., post hoc comparisons using estimated marginal means). We also used an online effect size calculator where appropriate: https://effect-size-calculator.herokuapp.com/.

We first conducted a 3 (payment form: cash vs. candy vs. monetized candy) x 2 (payment level: low vs. medium) between-subject ANOVA on the expected willingness to help (see Figure 1 for descriptive statistics). It yielded support for a very weak main effect of the payment form F(2, 1643) = 3.33, p = .04, $ηp2$ = .004, 90% CI [.0001, .01]. The main effect of the payment level was large, F(1, 1643) = 189.53, p .001, $ηp2$ = .10, 90% CI [.08, .13]. Moreover, there was support for an interaction, F(2, 1643) = 31.96, p .001, $ηp2$ = .04, 90% CI [.02, .05], supporting H1. We conducted planned pairwise comparisons using estimated marginal means to directly address H1a and H1b (see Tables 4 and 5). Reported p-values were adjusted with the Tukey method.

We found support for H1a, revealing that the expected willingness to help in the medium-cash condition (M = 7.15, SD = 2.60, N = 274) was higher than that in the low-cash condition (M = 3.63, SD = 3.02, N = 273), and the replication effect size was larger than the original one (see 4 and 5). By contrast, whereas Heyman and Ariely’s (2004) findings did not observe a difference between the low-candy (M = 5.14, SD = 2.74, N = 275) and medium-candy conditions (M = 6.31, SD = 2.66, N = 276), the CI for the effect size did not include zero in the present study, and we cannot conclude the absence of an effect (see Tables 4 and 5). Thus, we did not find support for H1b. H1b is a null hypothesis, and we therefore decided to conduct a non-preregistered Bayesian analysis to supplement the hypothesis testing. We report the results of the analysis at the end of the section. The effect of the payment size was larger in the cash condition than in the candy condition (see Tables 4 and 5), and the results suggest that participants in the cash condition (i.e., the money market relationship) were more sensitive to the change in the payment level than those in the candy condition (i.e., the social market relationship), thus, supporting H1.

To test H1c, we carried out a one-way between-subject Welch’s ANOVA using the following four conditions: payment-low-cash vs. payment-medium-cash vs. nonpayment-with-notice vs. nonpayment-without-mention. In the pre-registration, we did not explicitly mention whether we would use a conventional Fisher’s or Welch’s ANOVA. However, given that Levene’s test indicated that the assumption of equal variance was violated (F(3, 1097) = 3.77, p = .01), we opted for Welch’s ANOVA as it would correct degrees of freedom for the violation. The analysis revealed a large main effect, F(3, 608.17) = 72.88, p .001, $η2$= .26, 90% CI [.22, .31]. A post hoc comparison showed support for the differences between the nonpayment-without-mention (M = 6.15, SD = 2.55, N = 275) and low-cash (M = 3.63, SD = 3.02, N = 273) conditions (see Tables 4 and 5), with those in the former condition estimating the likelihood of helping higher than those in the latter. Thus, we found support for H1c and replicated the original finding with a larger effect size.

To address H2, we looked at the post hoc comparisons from the 3 x 2 ANOVA and found that the expected willingness to help in the medium monetized candy condition (M = 5.86, SD = 3.00, N = 274) was higher than that in the low monetized candy condition (M = 4.75, SD = 3.02, N = 277, see Tables 4 and 5). However, the effect size was much smaller than the original one, and the effect size of the payment level in the monetized candy condition (d = -0.37, 95% CI = [-0.54, -0.20]) was more similar to that in the candy condition (d = -0.43, 95% CI = [-0.60, -0,26]) than the cash condition (d = -1.25, 95% CI = [-1.43, -1.07]). Thus, we failed to find support for H2, where Heyman & Ariely (2004) predicted that monetized goods would prime the money market relationship, and the expected willingness to help in the cash and monetized goods condition would be similar.

H1b was a null hypothesis, predicting that the expected willingness to help in the low-candy condition would not be different from that in the medium candy condition. In addition to the pre-registered conventional hypothesis testing, we carried out an exploratory Bayesian t-test to directly test H1b. The null hypothesis was that there was no difference in the expected willingness to help between the small-candy and medium candy conditions, and the two-sided alternative hypothesis postulated that the expected willingness to help in the two candy conditions was different. The Bayes factor indicated that the data was in favor of the alternative hypothesis, B₁₀ = 20841.04, yielding strong evidence against Heyman and Ariely’s (2004) original finding that the level of reward in the candy conditions did not affect the expected willingness to help.

For our extension (H3-H4), we conducted a 3 (payment form: cash vs. candy vs. monetized candy) x 2 (payment level: low vs medium) between-subject ANOVA on joy (see Figure 2 for descriptive statistics). We found support for main effects: payment form: F(2, 1643) = 10.32, p .001, $ηp2$ = .01, 95% CI [.005, .02]; payment level: F(1, 1643) = 28.10, p .001, $ηp2$ = .02, 95% CI [.01, .03]. However, the 95% CI for the interaction effect included zero, F(2, 1643) = 2.15, p = .12, $ηp2$ = .003, 95% CI [.00, .01]. Thus, we did not find support for H3.

We conducted pre-registered planned comparisons to test H3a, H3b, and H4 (see Table 4). We found that in the cash condition, joy was higher when the payment level was medium (M = 7.00, SD = 2.46, N = 274) compared to when it was low (M = 5.92, SD = 2.92, N = 273, see Table 4). In comparison, the effect of the payment level was much weaker in the candy conditions (low-candy condition: M = 6.94, SD = 2.69, N = 275; medium-candy condition: M = 7.44, SD = 2.36, N = 276), and in the monetized candy conditions (low-monetized candy condition: M = 6.54, SD = 2.81, N = 277; medium-monetized candy condition: M = 7.05, SD = 2.77, N = 274; see Table 4). Consistent with H4, we found that perceived joy was higher in the candy condition than in the cash and monetized candy conditions.

Since H3b was a null hypothesis, we conducted a non-preregistered Bayesian analysis to complement the t-test. The null hypothesis was that there was no difference in expected joy between the small-candy and medium-candy conditions, and the two-sided alternative hypothesis was that expected joy in the two conditions was different. The Bayes factor was B₁₀ = 1.25. Thus, the analysis provided anecdotal evidence for the null hypothesis.

Finally, as an exploratory extension (not pre-registered), we compared the nonpayment-without-mention condition with the nonpayment-with-mention condition; we did not find support for differences in the expected willingness to help between the control conditions (nonpayment-without-mention: M = 6.15, SD = 2.55, N = 275; nonpayment-with-mention: M = 5.78, SD = 2.70, N = 279), t(551.07) = -1.68, p = .09, d = -0.14, 95%CI [-0,31, 0.02].

We report results of other pre-registered analyses in the supplementary material (see Supplementary Results for Study 1 in supplementary material).

We found support for H1; the effect of the payment level was different across the money and social market relationships. However, whereas Heyman & Ariely (2004) found that individuals in the social market relationship were insensitive to the payment level, our results suggested that they were indeed sensitive in the social market relationship but less so compared to those in the money market relationship. In addition, we successfully replicated H1c with a larger effect size and revealed that the expected willingness to help in the low cash payment condition was lower than the nonpayment control condition, suggesting that the low level of monetary incentive would be counterproductive. However, contrary to Heyman & Ariely (2004), the monetized candy did not induce the money market relationship, as the effect of the payment level in the monetized candy condition was more similar to that in the candy condition than in the cash condition.

As an extension, we measured to what extent participants thought a helper in the scenario would experience joy. As expected, the level of payment did not influence joy in the social market relationship (i.e., the candy condition), where the willingness to help others is primarily driven by internal, altruistic motivations. We predicted that the level of payment should influence joy in the money market relationship (i.e., the cash and monetized candy conditions). We found support for an effect for joy in the cash condition, yet failed to find support for an effect in the monetized candy condition. Thus, this casts doubt on Heyman and Ariely’s (2004) argument that monetized goods would prime the money market relationship. Our results, overall, suggest that monetized goods would fall under the social market relationship.

We conducted Study 2 using a within-subject design; Charness et al. (2012) pointed to the importance of a choice of experimental design (between vs. within) in various decision-making tasks, showing that while some effects and decision-making processes were insensitive to experimental design, others were sensitive. Thus, a comparison of results from between- and within-subject design experiments is a sensible step that can further shed light on the robustness and generalizability of the findings.

After giving consent, participants were first presented with a vignette of the nonpayment-without-mention condition (i.e., the control condition) and indicated the likelihood that others would help in the scenario. This design was meant to ensure the control condition is not affected by carryover effects from the other conditions. Then, they were shown the other seven experimental scenarios in a randomized order and answered the dependent measure for each scenario. We did not include joy and regret but otherwise employed the same measures and experimental instructions as in Study 1.

In the pre-registration manuscript, we proposed to use a 7-point scale to measure the expected willingness to help, while Study 1 and the original study used an 11-point scale (“replication extension main manuscript – Heyman Ariely, 2004 – Group B.docx”, page 19). However, in the pre-registered study materials, we planned to use an 11-point scale, and we did conduct the study with the 11-point scale. This was an oversight misalignment between the registered manuscript and the registered survey materials.

Unless explicitly mentioned, we followed the pre-registered analytic plans. We first carried out a 3 (payment form: cash vs. candy vs. monetized candy) x 2 (payment level: low vs. medium) within-subject ANOVA on the expected willingness to help (see Figure 1 for descriptive statistics). Mauchly’s test revealed that the assumption of sphericity was violated for the main effect of the payment form and the interaction term, and we employed Greenhouse-Geisser corrected degrees of freedom. We found a large effect for the payment form, F(1.76, 1752.33) = 161.56, p .001, $ηp2$ = .14, 90% CI [.12, .16]. The main effect of the payment level was also large, F(1, 998) = 1679.32, p .001, $ηp2$ = .63, 90% CI [.60, .65]. Finally, we found an interaction effect, F(1.90, 1900.34) = 428.58, p .001, $ηp2$ = .30, 90% CI [.27, .33], supporting H1.

We then carried out planned pairwise comparisons to directly address hypotheses (see Table 4). For Study 1, following our preregistered plan, we conducted ANOVAs with post hoc comparisons using estimated marginal means with the Tukey correction. By contrast, for Study 2, we preregistered that we would run simple paired t-tests using raw means, and that p-values would not be adjusted. As we did not include any covariates in ANOVAs in Study 1 and the number of participants in each cell did not substantially vary, the use of different types of means would not be a problem. Thus, except for the presence of p-value adjustment, these different analytic strategies yielded compatible results. For replication evaluation, we focused on effect sizes and their CIs rather than p values, and we did not deviate from the pre-registered analytic strategies.

First, we found support for H1a; the expected willingness to help was higher in the medium-cash condition (M = 7.98, SD = 2.17) compared to the low-cash condition (M = 5.35, SD = 2.54). Moreover, the effect size was bigger than the original one (see Table 4). However, we found that the increase in the payment level in the candy condition also resulted in higher expected willingness to help (low-candy: M = 5.63, SD = 2.41; medium-candy: M = 6.77, SD = 2.24), conflicting with H1b (see Table 4). Regarding H1c, whereas Heyman & Ariely (2004) demonstrated that the expected willingness to help in the control condition was higher than that in the low monetary payment condition, we found support for an effect in the opposite direction (nonpayment-without-mention: M = 5.15, SD = 2.35; low-cash: M = 5.35, SD = 2.54). Thus, H1c was not supported (see Table 4).

We further conducted a pairwise comparison to address H2 and found that as in the cash condition, the expected willingness to help in the medium-monetized candy condition (M = 6.68, SD = 2.36) was higher than that in the low-monetized candy condition (M = 5.43, SD = 2.42, see Table 4), consistent with Heyman & Ariely (2004). However, as in Study 1, the effect size of the payment level in the monetized candy condition (d = -0.80, 95% CI = [-0.89, -0.71]) was more similar to that in the candy condition (d = -0.87, 95% CI = [-0.96, -0.78]), rather than the cash condition (d = -1.30, 95% CI = [-1.39, -1.20]). Overall, these results did not support H2.

Finally, we compared the expected willingness to help in the nonpayment-without-mention condition (M = 5.15, SD = 2.35) with that in the nonpayment-with-mention-condition (M = 5.47, SD = 2.36). We conducted a paired sample t-test and revealed that the expected willingness to help was higher in the latter condition than in the former, t(998) = 7.73, p .001, d = 0.25, 95% CI [0.24, 0.40]. We report results from other pre-registered analyses in the supplementary material (see Supplementary).

Overall, our results suggested that the expected willingness to help was higher when the payment level was medium compared to when it was low, regardless of the payment form. Our results are consistent with Heyman and Ariely’s (2004) core argument that the effect of the payment level would vary depending on the payment form. We replicated the effect of the payment level in the cash condition (H1a) with larger effect size and found that the expected willingness to help was higher in the low cash condition than in the medium cash condition. However, we did not find support for H1b; Heyman & Ariely (2004) found that people were insensitive to the payment level in the candy condition, yet our results showed that participants were sensitive to payment but less so in the candy condition than in the cash condition. Moreover, while we replicated the effect of the payment level in the monetized-candy condition, the effect size was similar to that in the candy condition, failing to support H2. We also failed to find support for H1c. In the original study, they found that the expected willingness to help was higher in the control condition compared to the low cash condition. Nevertheless, our replication revealed that it was the opposite; the expected willingness to help was higher in the low cash condition than in the nonpayment-without-mention condition.

In Studies 1 and 2 we employed different experimental designs, and these yielded mostly converging results. Yet a discrepancy emerged in the control conditions; the expected willingness to help in the control conditions was lowest in the present study, whereas it was high in the original study. One possible explanation may be our choice of the experimental design; in our Study 2 using a within-subject design, participants were first presented with the nonpayment-without-mention condition and then shown the remaining seven scenarios in a randomized order. Presumably, participants used their judgment in the control condition as a baseline; they perceived low-level payments of any kind as being more attractive than nonpayment, and this might have inflated the expected willingness to help in the experimental conditions relative to that in the control conditions. Participants in Study 2 perceived even small compensations as more attractive than receiving nothing. This might be explained by individuals’ stronger sensitivity to the size of the payment in joint evaluation mode (i.e., the within-subject design) compared to single evaluation mode (i.e., between-subject design) (Anvari et al., 2021; Hsee & Zhang, 2010).

Alternatively, it is possible that the results of Study 2 were affected by demand effects. Participants saw the least attractive option first (nonpayment-without-mention condition) and were then presented with more attractive scenarios in which rewards were given. This might have led to participants guessing the study’s goal (i.e., whether rewards would increase people’s motivation to help others), and then to their responding in a way that would help achieve the desired outcome. Overall, the discrepancy between Studies 1 and 2 regarding H1c could be attributed to our choice of experimental design and our chosen order of display in Study 2.

Heyman & Ariely (2004) claimed that the effect of the payment level would depend on the payment form. More specifically, they found that individuals expected others to be more willing to help in the money market relationship, but not in the social market relationship. In the two well-powered replication studies, we revealed that the higher payment was more effective in increasing people’s perceptions of willingness to help regardless of the market relationship (i.e., regardless of whether rewards were provided as cash, goods, or monetize goods). Notably, we found that the effect of the payment level was much larger when paid in cash than when paid with goods. Thus, while we found support for Heyman and Ariely’s (2004) main argument that the effect of the payment level would vary across different market relationships, we did not replicate their finding that the payment level did not matter in the social market relationship. Moreover, Heyman & Ariely (2004) held that monetized goods (goods with a price tag) would prime the money market relationship, yet our results suggested that it primes the social market relationship. The discrepancy between the original and the replication studies is of practical importance, suggesting that people perceive that more is better, and that this is especially true for cash. These findings help update knowledge regarding how payment form and level are related to expected willingness to help, denoting the value of replication studies.

Finally, we note that our design mirrored that of the original and that the large effects reported in our studies using behavioral intention proxies should not be taken to suggest that the increase in the payment level would make an observable and substantial impact outside of controlled laboratory settings. Moreover, these studies focused on small and medium incentives, and it would be a relevant avenue for future research whether these findings would hold when comparing, for instance, medium and large payment levels. Therefore, now that these findings have been revisited and adjusted, we see promise in further follow-up replications and studies that would extend these to examine higher stakes and practical implications.

Wan Fei Chan, Lee Hing Man , Yuk Ki Ng, and Mei Sze Wong conducted the replication, including original article analysis, power analyses, designed the experiments, constructed the surveys, wrote the pre-registrations, and conducted the data analysis and initial write-ups.

Boley Cheng guided and assisted the replication effort .

Hirotaka Imada verified and extended analyses, integrated the studies, and wrote the final manuscript for submission.

Gilad Feldman led the replication effort, supervised each step in the project, conducted the pre-registrations, and ran data collection. Gilad edited the final draft for submission.

The authors declared no potential conflicts of interests with respect to the authorship and/or publication of this article.

The authors received no financial support for the research and/or authorship of this article.

We would especially like to thank Max Korbmacher and Ashleigh Haydock-Symonds for serving as our red team for this project, pushing us to do better. We would also like to thank Farid Anvari and Jerome Olsen for their feedback and involvement in the initial stage of this project. We would like to thank Raluca Diana Szekely, Raj Aiyer, Qinyu Xiao, and Nicole Russell for their feedback on earlier drafts of this manuscript.

All materials, data, and code are provided on https://osf.io/y9p7u/

In the table below, employ CRediT (Contributor Roles Taxonomy) to identify the contribution and roles played by the contributors in the current replication effort. Please refer to the url (https://www.casrai.org/credit.html ) for details and definitions of each of the roles listed below.