“99% of Gamblers Lose in the Long Run”: An Experimental Comparison of Novel and Pre-Existing Harm Prevention (Safer Gambling) Messages Warning About the Likelihood of Losing Money Open Access

1. The null hypothesis will be tested that for each rated dimension that there are no significant differences between each message for participants overall

2. For each message and rated dimension, an interaction model will be run to see if the message’s rating depends on gamblers’ Problem Gambling Severity Index (PGSI) levels

3. Participants will rate each message as being more relevant to gamblers at risk of harm than they are to themselves, and we will explore this difference’s robustness to different PGSI levels

Materials, data, and analyses can be accessed at https://osf.io/bd2jk/, while the preregistration is available at https://osf.io/b3hfr. Ethical approval was granted by the University of Bristol’s School of Psychological Science Research Ethics Committee (#22746).

Participants were recruited through the online crowdsourcing platform Prolific. Eligible participants were required to be at least 18 years old, resident in the UK, and have reported engaging in at least one online gambling format beyond the lottery. Participants were compensated £1 (mean duration before exclusions = 6.1 min, equivalent to £9.9 per hour). We aimed to recruit an initial sample of 4,240 participants with a final target sample of 4,000 after data-quality exclusions. This sample size was chosen heuristically, based on our understanding that messaging interventions can have quite small effect sizes, and based on considerations regarding our access to research funds. We preregistered to first exclude the 5% fastest participants, and also exclude an anticipated 1% reporting that they were not paying attention at the end of the study. We used a fixed percentage for consistent exclusion rates rather than response-time thresholds based on the mean and standard deviation, as response times are always positively-skewed.

The study initially captured data from N=4,245 participants. Applying our preregistered criteria, we excluded eight responses for self-reported non-attention and 212 participants for completion times that were under 2.6 minutes (the 5% fastest). No participants were excluded due to missing data. Our final dataset comprised of N=4,025 participants. Demographic information and self-reported gambling engagement were automatically collected by Prolific and are displayed in Tables 1 and 2, respectively. Although participants were all required to have engaged in one non-lottery gambling format, the lottery was still the single most engaged-with gambling format (64%), which follows the pattern shown in the overall population. Bingo was the second-most engaged-with gambling format (58%), and this is generally seen to be, like the lottery, as a high-engagement gambling format with relatively weak links to harm (Allami et al., 2021). Constrastingly, slots were the third-most engaged-with gambling format (51%), and this is commonly seen as being a gambling format with relatively strong links to harm (either online or as part of land-based electronic gaming machines; Allami et al., 2021).

Participants completed the Problem Gambling Severity Index (PGSI), a widely-used gambling screening measure (Ferris & Wynne, 2001), with results showing the distribution of PGSI scores across the original four groupings shown in Table 2. The distribution of PGSI scores confirm a widespread trend of online samples having higher average risk levels than population base-rates, which allowed us to efficiently sample from the important group of high-risk gamblers (Russell et al., 2021).

After receiving an information sheet and providing informed consent, participants were presented with the ten harm prevention messages shown in Table 3 in a randomized sequence. Two existing messages were changed by replacing “bettors” or “betting” with the synonymous words “gamblers” or “gambling,” which made the language more in keeping with the other messages. We also chose to delete the word “over” from the rest of the message “99% of gamblers lose in the long run,” as larger percentages may be perceived as less believable, and this is also more in line with what was actually said by the gambling industry executive (Alexander, 2020).

Participants were asked to evaluate each message based on agreement with the four statements, which formed the study’s dependent measures and were also presented in a random order. Two statements were newly introduced:

• “This message challenges me to try to win at gambling” and

• “This message makes me want to gamble less”

The other two statements were derived from discrepancies observed between Chapman and Priestly (2022) and Newall, Torrance et al. (2024):

• “This message is relevant to me” and

• “This message is relevant to people experiencing gambling harms”

Responses were recorded on a seven-point Likert scale, ranging from “strongly disagree” to “strongly agree.” Subsequently, participants completed the PGSI. To help improve data quality, a self-reported carelessness check was implemented: “In your honest opinion, should we use your data in our analyses in this study? (Do not worry, this will not affect your payment, you will receive the payment code either way.)” (Brühlmann et al., 2020). Only data from participants who responded”yes” to this question were retained for analysis. After exclusions, participants completed the task in 6.3 minutes on average (SD=8.0).

All analyses were conducted using R 4.4.1 (R Core Team, 2023). Mixed effect models were estimated using the function lmer from the package lme4 (Bates et al., 2015). Model estimation was achieved using Restricted Maximum Likelihood and p-values were calculated using Kenward-Roger’s method for estimating degrees of freedom (McNeish, 2017) using the package lmerTest (Kuznetsova et al., 2017), as they have been shown to produce acceptable Type I errors (Luke, 2017). To further reduce Type I errors, all multiple comparisons were adjusted using the Sidák method (Gramm et al., 2007).

For RQ1, we ran four separate mixed effects models, one for each statement. There was a single fixed predictor (message), which was a factor identifying each of the 10 messages, and one random intercept for each participant. Due to the four statements being analyzed concurrently, we used a base confidence level (alpha) of p=.01 to reduce the potential for Type I error inflation. To identify differences between the responses given to each message, we present the results based on a compact-letter-display (CLD) analysis which identifies groupings of mean responses that are not significantly different (nsd) from each other using pairwise comparisons across the 10 messages, corrected for multiple comparisons using Sidák’s method, which equals to a pre-adjusted alpha threshold of p=.00022 for significant results.

For RQ2, we added centered PGSI scores as a covariate (with interaction) to each of the four models above, and we tested the effects of PGSI scores via a significant main effect and interaction with the fixed factor (message). To evaluate how PGSI affected each message individually, we tested the slope of PGSI score interaction for each message against zero, and present the results from a CLD analysis identifying different groupings of the slope of the PGSI interaction which are not different from each other using pairwise comparisons across the 10 messages, both corrected for multiple comparisons using the Sidák method.

For RQ3, we calculated a new dependent variable (“DV”) as the difference between the response given to the statement “relevant to people experiencing gambling harms” minus the answer to the statement “relevant to me.” Positive values of this difference indicate that participants rated the message as more relevant to people experiencing gambling harms than to themselves, and vice-versa. We ran a similar model as above, using this difference as the DV, with a fixed factor for message, PGSI score, their interaction, and a random effect for each participant. Results were evaluated by comparing the difference against zero for each message, as well as the slope of the PGSI score interaction against zero for each message, both corrected for multiple comparisons using the Sidák method. All analyses were preregistered, apart from the comparisons with the indifference mid-point, as well as the estimated marginal means shown in Figure 3C, which were exploratory.

For all PGSI-related analyses, PGSI was used as a continuous variable, which avoids the loss of information which occurs when dichotomizing a variable into sub-groups (e.g., the high-risk gambling group with PGSI scores of 8+). Furthermore, the analysis used is consistent with the public health conceptualization of harm as a continuous spectrum.

Outcomes for Research Question 1’s four dependent measures are shown in Figure 1. In Figure 1A, which shows the outcomes for the “challenges me” measure, it can be seen that participants did not on average find that the messages challenged them to try to win at gambling (all message averages were significantly below the indifference mid-point of 4, all ps<.001). However, there was significant variation between the messages (𝜒²(9)=1683, p<.001, pseudo-R²=0.43), with They Win presenting the greatest challenge (M=3.4), with the remaining messages being placed in four descending groups via the CLD. By contrast, the novel framing You Lose was in the third-lowest grouping (M=2.8). Cost You and Comes At Cost presented the joint-lowest challenge to gamble (both M=2.5). The difference from greatest to lowest, a measure of unstandardized effect size, was 0.92, almost one full point change in the response scale (i.e., between ‘disagree’ and ‘somewhat disagree’).

Figure 1B shows outcomes for the “makes me want to gamble less” measure. On average, participants thought that each message would make them want to gamble less (all message averages were significantly above the indifference mid-point of 4, all ps<.001). But the ranking of the messages were significantly different with this measure (𝜒²(9)=2381, p<.001, pseudo-R²=0.44), with Cost You (M=4.4) and Comes At Cost (M=4.8) ranking less well here. One similarity is that They Win was again the worst message on this measure (M=4.3), with the novel You Lose framing again performing better, being placed in the CLD’s second-highest grouping (M=5.0). Lose Long Run was the single best message on this measure (M=5.2). The difference from worst to best was 0.94, again close to one full point change in the response scale (i.e., from ‘neither agree nor disagree’ to ‘somewhat agree’).

Figure 1C shows outcomes for the “relevant to me” measure. Only Real Winners was rated as being on average relevant to the participants themselves (M=4.1, 99% CI = [4.2, 4.0], p=.001). Overall, there was significant variation between the messages (𝜒²(9)=978, p<.001, pseudo-R²=0.66). Five messages were joint-second highest, including Adverts Show (M=4.0) and You Lose (M=4.0). They Win (M=3.8) was in the second-lowest group, therefore again performing worse than You Lose, with Cost You being ranked as the single worst message (M=3.5). The difference from worst to best was 0.62.

Figure 1D shows outcomes for the “relevant to people experiencing gambling harms” measure. On average, participants thought that each message was relevant to this group (all message averages were significantly above the indifference mid-point of 4, all ps<.001, and the highest responses overall). Again, there was significant variation between the messages (𝜒²(9)=1814, p<.001, pseudo-R²=0.36), They Win was seen as being the least relevant to participants (M=5.1), with the novel You Lose framing again performing better and in the CLD’s second-highest group (M=5.8). Lose Long Run was again the single best message on this measure (M=6.0). The difference from worst to best was 0.84.

Figure 2 demonstrates outcomes for Research Question 2, exploring whether messages’ ratings across the four dependent measures depended on participants’ PGSI scores. In Figure 2A, which shows interaction effect outcomes for the “challenges me” measure, it can be seen that all individual interaction effects were significantly positive, meaning that all messages were seen as being a greater challenge for participants with higher PGSI scores. However, there were no significant differences between each message in terms of the strength of this interaction. Adding PGSI scores to the model significantly improved its fit in comparison to RQ1 (𝜒²(10)=272, p<.001).

Figure 2B shows interaction effect outcomes for the “makes me want to gamble less” measure. Adding PGSI scores to the model significantly improved model fit (𝜒²(10)=58, p<.001). The only messages without significant positive interactions were They Win (b=0.01, 99% CI = [0.03, -0.01]) and Adverts Show (b=0.02, 99% CI = [0.04, -0.00]), meaning that, for all other messages, participants with higher PGSI scores reported wanting to gamble less. However, again there was limited between-message difference in the slope of this interaction, with the top nine messages all being grouped together (including the highest Cost You, b=0.03, 99% CI = [0.02, 0.05]), as well as the lowest nine messages (including They Win and Adverts Show). Furthermore, these estimated interactions were all small in magnitude.

Figure 2C shows interaction effect outcomes for the “relevant to me” measure. The model with PGSI scores showed a significant improvement over a model without them (𝜒²(10)=934, p<.001). Here all messages had significant positive interaction effects, showing that they were judged to be more relevant to participants with higher PGSI scores (see OSF results Section 3.4.3 at https://osf.io/zhxg3?view_only=0b2f3a2f01ab419e99af7b628c37c785). However, the interaction effects were largely similar in estimated size, with the CLD grouping the largest seven (including the largest interaction, Lose Long Run, b=0.15, 99% CI = [0.14, 0.17]) and smallest eight messages together (including the smallest interaction, They Win, b=0.13, 99% CI = [0.11, 0.15]).

Figure 2D shows interaction effect outcomes for the “relevant to people experiencing gambling harms” measure. Here, none of the messages had a significant interaction, and none of these were identified as being significantly different (𝜒²(10)=18, p=.05). PGSI score did not appear to matter when it came to a message being considered as relevant to people experiencing gambling harms (see OSF results Section 3.4.4).

The model fit the data significantly better than the null model (𝜒²(19)=1560, p<.001, R²=0.59). All messages were seen as being on average more relevant to gamblers at risk of harm than to themselves (Figure 3A), which confirms Research Question 3. This is represented by significant positive means for the difference (Gamblers − Me) for each message. The smallest difference is for They Win (M=1.35, 99% CI=[1.25, 1.45]), while the largest differences are for Comes At Cost (M=1.91, 99% CI=[1.81, 2.01]) and Lose Long Run (M=1.98, 99% CI=[1.88, 2.07]).

The slopes of the effect of PGSI on the differences for all messages were significantly negative, with the CLD grouping both the top six and bottom six messages together as seen in Figure 3B. In effect, this meant that the overall tendency for participants to view messages as being less relevant to themselves was reversed for participants with very high PGSI scores (equal to or above 17, or about 3% of the participants), who contrastingly saw messages as being more relevant for themselves than for gamblers at risk of harm (Figure 3C), due to higher responses given to the “relevant to me” question as PGSI increased (see OSF results Section 3.4.3).

As each participant was shown 10 messages in a random order, we also evaluated if there was an order effect of message presentation. We added a new variable identifying the order in which each message was shown (from 1 to 10) as a main effect and interaction to the H1 models above, in an exploratory analysis that was not preregistered.

Overall, there was an order effect for two of the measures. It was significantly positive for “challenges me” (b=0.006, CI=[0.000, 0.012], p=.009), significantly negative for “makes me want to gamble less” (b=-0.013, CI=[-0.019, -0.007], p<.001), and not significant for “relevant to me” (b=-0.002, CI=[-0.006, 0.003], p=.35) or “relevant to people experiencing gambling harms” (b=0.002, CI=[-0.003, 0.007], p=.27). Therefore, responses to the “challenges me” question increased (slightly, given the small magnitude of the coefficient b) as the task progressed and more messages were shown, while responses to the “makes me want to gamble less” question decreased slightly. The combined effect, after 10 questions, was to increase the former by 0.05 and decrease the latter by 0.12, on average across all messages, which could be considered as relatively small effects.

There were also significant interactions between order effect and message for all four measures (all ps < .001), although not all individual slopes for each message were significant, after adjusting for multiple comparisons (see OSF supplement for detailed results). An analysis estimating the responses to each question assuming messages are seen first (as opposed to at the average midpoint after five messages as in the main results), resulted in their slight re-ordering, because of the different ways in which responses changed over time. To summarize, if the overall ranking (see Table 4 below) had been conducted with estimated first responses (1^st message shown) instead of their averages, the best and worst messages would not change, but we would see an improvement in Adverts Show moving it from 5^th into 3^rd place (pushing You Lose and Comes at Cost to 4^th and 5^th respectively), while Cost You was worse and would be ranked 9^th instead of joint 8^th. The differences between the worst and best scoring messages for each measure, an indication of unstandardized effect size, increase at the first message shown, from [0.62, 0.94] to [0.82, 1.23], or a larger differentiation between messages.

Messaging is often regarded as a way of raising awareness of gambling harms among large segments of the population (Ray et al., 2024), while more restrictive measures, such as self-exclusion, are generally deemed more appropriate for those already experiencing significant harms. Although harm prevention messages can be designed with multiple potential strategies in mind, the present work provided an in-depth exploration of existing and novel messages that warn gamblers about the likelihood of losing money.

Despite having a similar aim, systematic differences were found between the messages regarding the four dimensions of RQ1. How each message performed on average is summarized here in an exploratory fashion by ranking each message from best to worst on each dimension, and then calculating each message’s multidimensional average rank (reverse scoring the “challenges me” measure). Table 4 shows the outcomes of this ranking, where Lose Long Run and Real Winners were ranked highest, while the five lowest-ranked messages were all messages currently in use in Australia (Prepared To Lose; You Win Some; Chances Are), Belgium (Cost You), and the UK (They Win). Policymakers in these jurisdictions could consider implementing the novel better-ranked messages in future, especially since any positive effects of currently-used messages may weaken with repeated exposure over time (Cacioppo & Petty, 1979; Newall, Rockloff, Hing, Thorne, et al., 2023). One clear improvement appears to be replacing the existing UK message They Win (Mills et al., 2023), which was the lowest-ranked here, with the novel reframing of You Lose which was ranked joint third. This finding conceptually replicates previous work showing that the framing of statistical information matters in gambling (Newall et al., 2020).

Some of the findings do contradict messaging’s purpose as a population-based harm prevention measure (Ray et al., 2024). Only one message was perceived as being relevant to the average participant (“relevant to me”), with RQ2’s analysis showing that messages only began to be perceived as personally relevant as PGSI scores increased. Therefore, these message’s effects might be enhanced if they are shown in, for example, online gambling platforms and land-based gambling venues, where people who are already experiencing some level of gambling-related harm predominate. The significant interaction effect in RQ3’s analysis showed that only participants with very high PGSI scores (17 or above) saw the messages as being more relevant to themselves than they were to gamblers experiencing harm. This is actually quite a high bar for messages to exceed, and so this shows self-awareness among participants experiencing very high levels of harm. Furthermore, the significant interaction effect involving PGSI for the “challenges me” measure suggests that these messages may also counterproductively be seen as a greater challenge by participants with higher PGSI scores. Overall, these findings underscore to us also that messaging should only be used as one of a number of different interventions directed at attempting to prevent gambling-related harm.

These findings are subject to various limitations. Crowdsourced samples do not represent gamblers as a whole, for which representative surveys are more appropriate (Pickering & Blaszczynski, 2021). The present work focused on gamblers, so future work should see whether these and other messages can potentially prevent non-gamblers from experiencing harm by either taking up gambling or from another person’s gambling. Self-report studies are limited by the dependent measures that are selected by researchers, and different outcomes might have occurred with the measures used in other studies (Chapman & Priestly, 2022). Randomized controlled trials (RCTs) with behavioral outcomes are more ecologically-valid than self-report studies, but must be adequately powered given that effect sizes tend to become even smaller in naturalistic environments (Auer & Griffiths, 2015; Heirene & Gainsbury, 2021). Furthermore, RCTs require active collaboration with a gambling operator, which limits the number of researchers with access to this methodology (Cassidy et al., 2013).

These findings are also limited to the 10 messages tested here, and so input from multiple stakeholders is needed to continue generating and testing as many messages as possible. Message effectiveness may be influenced by past exposure (McCullock et al., 2024), and we note that the lowest-ranked message was the only one in current UK use (by one local government), which might have therefore been the message participants were most likely to have been previously exposed to. We did not measure prior message exposure. There were also significant order effects, with responses changing as participants were exposed to multiple messages during the experiment. While we believe this to be a realistic setting – messages do not exist in a vacuum and campaigns often employ multiple messages to reduce message fatigue (McCullock et al., 2024), some carryover effects were observed across messages. Further work is therefore needed to understand order effects in gambling messages, particularly in naturalistic environments. These ten messages were also not compared to other potential message types (Harris et al., 2018; Mutti-Packer et al., 2022), which future work should explore. Future work should continue to look to other areas, such as public health more generally (Abroms & Maibach, 2008), or fear-based tobacco warnings (Witte & Allen, 2000), as further inspiration for harm prevention message design.

Further considerations are also relevant to policymakers considering harm prevention messages as an intervention. The present results do not compare messaging against a control condition of no messaging, nor to other potential interventions, such as restrictions on advertising or product availability. Therefore, the present findings are perhaps most relevant to policymakers who have already decided to implement or update their harm prevention messages, and now face the decision of selecting specific messages. On the one hand, the relatively small observed differences across many of these messages suggest that this policymaker might be best placed to choose a large number of related messages to rotate between, so as to reduce repetition-based fatigue (McCullock et al., 2024; Newall, Rockloff, Hing, Thorne, et al., 2023; Ray et al., 2024). On the other hand, some clear differences do also emerge and should inform this choice, especially when comparing the current “odds are: they win” message to the novel “odds are: you lose” message. These initial conclusions should furthermore be updated as more evidence emerges.

In conclusion, the present work has contributed to the understanding of harm prevention messages that warn gamblers about the expectation of losing money. This can inform policymakers and other stakeholders with an interest in this population-based intervention.

Thanks to Dan Riley for providing helpful input during the conceptualization of this project.

This research was funded by a Policy Support Fund grant awarded to Dr Philip Newall from the University of Bristol.

Philip Newall is a member of the Advisory Board for Safer Gambling – an advisory group of the Gambling Commission in Great Britain. In the last three years, Philip Newall has contributed to research projects funded by the Academic Forum for the Study of Gambling, Clean Up Gambling, Gambling Research Australia, NSW Responsible Gambling Fund, and the Victorian Responsible Gambling Foundation. Philip Newall has received honoraria for reviewing from the Academic Forum for the Study of Gambling and the Belgium Ministry of Justice, travel and accommodation funding from the Alberta Gambling Research Institute and the Economic and Social Research Institute, and open access fee funding from the Academic Forum for the Study of Gambling and Greo Evidence Insights.

Leonardo Weiss-Cohen has received open access fee funding from Gambling Research Exchange Ontario.

Elena Petrovskaya has received funding from the Academic Forum for the Study of Gambling and is currently funded by GREO.

Jamie Torrance has received, in the last three years: (1) Open access publication funding from Gambling Research Exchange Ontario (GREO), (2) Conference travel and accommodation funding from the Academic Forum for the Study of Gambling (AFSG), (3) A minor exploratory research grant from the ASFG and GREO.

Madison Palmer has no disclosures to make.

In relation to the past three years, L.Y.X. has been providing paid consultancy for Public Group International Ltd (t/a PUBLIC) (Companies House number: 10608507), which has been commissioned by the UK Department for Culture, Media and Sport (DCMS) to conduct independent research on understanding player experiences of loot box protections (October 2024 – Present). L.Y.X. undertook a brief period of voluntary work experience at Wiggin LLP (Solicitors Regulation Authority number: 420659) in London, England in August 2022. L.Y.X. has contributed and continues to contribute to research projects that were enabled by data access provided by the video game industry, specifically Unity Technologies (NYSE:U) (October 2022 – Present). L.Y.X. has been invited to provide advice to the UK Department for Digital, Culture, Media and Sport and its successor (the Department for Culture, Media and Sport; DCMS) on the technical working group for loot boxes and the Video Games Research Framework. L.Y.X. was the (co-)recipient of three Academic Forum for the Study of Gambling (AFSG) Postgraduate Research Support Grants (March 2022, January 2023, & July 2024) and a Minor Exploratory Research Grant (May 2024) that were derived from ‘regulatory settlements applied for socially responsible purposes’ received by the UK Gambling Commission and administered by Gambling Research Exchange Ontario (GREO) and its successor (Greo Evidence Insights; Greo). L.Y.X. has accepted funding to publish academic papers open access from GREO and the AFSG that was received by the UK Gambling Commission as above (October, November, & December 2022, November 2023, & May 2024). L.Y.X. was the recipient of an Elite Research Travel Grant 2024 [EliteForsk-rejsestipendium 2024] awarded by the Agency for Higher Education and Science of the Danish Ministry of Higher Education and Science [Uddannelses- og Forskningsstyrelsen under Uddannelses- og Forskningsministeriet] (February 2024). L.Y.X. has accepted conference travel and attendance grants from the Socio-Legal Studies Association (February 2022 & February 2023); the Current Advances in Gambling Research Conference Organising Committee with support from GREO (February 2022); the International Relations Office of The Jagiellonian University (Uniwersytet Jagielloński), the Polish National Agency for Academic Exchange (NAWA; Narodowa Agencja Wymiany Akademickiej), and the Republic of Poland (Rzeczpospolita Polska) with co-financing from the European Social Fund of the European Commission of the European Union under the Knowledge Education Development Operational Programme (May 2022); the Society for the Study of Addiction (November 2022 & March 2023); the organisers of the 13th Nordic SNSUS (Stiftelsen Nordiska Sällskapet för Upplysning om Spelberoende; the Nordic Society Foundation for Information about Problem Gambling) Conference, which received gambling industry sponsorship (January 2023); the MiSK Foundation (Prince Mohammed bin Salman bin Abdulaziz Foundation) (November 2023); and the UK Gambling Commission (March 2024). L.Y.X. has received honoraria from the Center for Ludomani for contributing parent guides about mobile games for Tjekspillet.dk, which is funded by the Danish Ministry of Health’s gambling addiction pool (Sundhedsministeriets Ludomanipulje) (March & December 2023); from the Fundació Pública TecnoCampus Mataró-Maresme [TecnoCampus Mataró-Maresme Foundation] for a guest lecture (November 2023); and from the YMCA (Young Men’s Christian Association) of Greater Toronto Youth Gambling Awareness Program for a presentation, which is funded by the Government of Ontario, Canada (March 2024). L.Y.X. received royalties by virtue of the copyright subsisting in some of his publications from the Authors’ Licensing and Collecting Society (ALCS) (Companies House number: 01310636) (March 2023 & 2024). A full gifts and hospitality register-equivalent for L.Y.X. is available via: https://sites.google.com/view/leon-xiao/about/gifts-and-hospitality-register. The up-to-date version of L.Y.X.’s full conflict-of-interest statement beyond the last three years is available via: https://sites.google.com/view/leon-xiao/about/conflict-of-interest.