Possible malfunction in widely used methane sampler deserves attention but poses limited implications for supply chain emission estimates

The recent surge in U.S. natural gas and oil production, made possible by the use of horizontal drilling and hydraulic fracturing to tap hydrocarbons in unconventional geological formations like shale or tight sands, has been accompanied by concerns over potential climate, environmental, and health implications (Moore et al., 2014). These concerns were reinforced by reports (Brandt et al., 2014) that available inventories underestimated methane (CH₄) emissions from the natural gas supply chain, which in turn stimulated an extensive body of new research to improve emission estimates.

A research team led by the University of Texas at Austin published three papers reporting measurements of CH₄ emission sources at natural gas production sites, hereafter referred to as the UT Production Studies (Allen et al., 2013a, 2015a, 2015b). The UT Production Studies quantified emissions using a diversity of methods and instrumentation, among which was the Bacharach Hi Flow^® Sampler (BHFS) (Bacharach Inc., 2015). Similar instruments have been used to measure CH₄ emissions in the natural gas industry since the 1990s (Kirchgessner et al., 1997); the BHFS has been commercially available since 2001 (Howard et al., 2015). The performance of the BHFS under certain conditions has been questioned (Howard et al., 2015; Modrak et al., 2012), leading to concerns that the UT Production Studies may have substantially underestimated CH₄ emissions (Howard, 2015a, 2015b).

The BHFS is a portable, battery-powered instrument packaged inside of a backpack; it was designed to quantify CH₄ emission rates from leaking components common to natural gas operations, such as pipe fittings, valve packings and compressor seals and has a measurement range of 0.05–8 standard cubic feet per minute (scfm, equivalent to 1.4–230 L min^-1) (Bacharach, Inc., 2015). The operating principles of the BHFS are described elsewhere (Bacharach, Inc, 2015; Howard et al., 2015; Modrak et al., 2012; Howard, 2015b; Brantley et al., 2015); here we provide a brief summary. The BHFS generates suction flow rates of 8–10 scfm to fully capture an individual component’s emissions along with surrounding air. Emissions are captured by placing a sampling hose, connected to one of a variety of sampling tools, immediately next to or around a leaking component. The instrument performs measurements at two constant flow rates differing by 20–30% to test whether the emissions from a source are being fully captured (complete capture can be assumed if emissions determined at both flow rates agree within ∼10%) (Bacharach, Inc., 2015). The BHFS determines a source’s emission rate by multiplying the CH₄ concentration of the resultant gas stream passing through the instrument’s detector by the known flow rate through the sampler (the instrument applies a background correction for CH₄ in ambient air using a parallel sampling and detector train) (Bacharach, Inc., 2015). It should be noted that the measured CH₄ concentration inside the instrument is not the CH₄ concentration of the leak source (which typically ranges from 70–98 mole% for natural gas sources); the resulting concentration is lower due to dilution with suctioned air. The BHFS quantifies CH₄ concentrations with a dual-mode sensor designed to transition between a catalytic oxidation mode and a thermal conductivity mode depending on the CH₄ concentration of sample/air mixtures within the instrument (0–5% CH₄ and 5–100% CH₄, respectively) (Bacharach, Inc., 2015; Modrak et al., 2012). The sensor is calibrated using CH₄; in cases where the leaking natural gas source contains elevated amounts of non-CH₄ species, calibrations using actual gas compositions or response correction factors are recommended (Bacharach, Inc., 2015). The non-CH₄ fraction of natural gas streams exhibits significant spatial variability and includes other hydrocarbons like ethane, propane, butanes, and C5+ species, and may also include other species such as carbon dioxide, water, and hydrogen sulfide.

The BHFS has been reported to underestimate the actual CH₄ concentration of sampled gas streams – and therefore CH₄ emission rates of individual sources – because it can fail to transition from the catalytic oxidation mode to the thermal conductivity mode for reasons that have yet to be determined (Howard et al., 2015; Modrak et al., 2012; Howard, 2015b; Brantley et al, 2015). Howard et al. (2015) reported that the transition failure can occur when measuring natural gas emissions from sources with CH₄ content less than ∼91% (mole percent). Howard (2015b) subsequently suggested the BHFS may underestimate emissions from high emitting sources when CH₄ content is as high as 97%.

The possibility of BHFS sensor transition failure needs to be more closely examined. Until the underlying issues are better understood, users should proactively validate data collected with the instrument in order to reduce the risk of problems.

The objective of this paper is to constrain the magnitude of any possible underestimation of emissions resulting from the deployment of the BHFS in the UT Production Studies and discuss the potential impacts of such underestimation on the study’s primary conclusions and our understanding of the industry’s methane emissions more generally. We do not assess claims about the existence of sensor transition failure in the BHFS or whether it, in fact, occurred in the UT Production Studies. We note that primary co-authors of the UT Production Studies presented evidence (Allen et al., 2015c) leading them to conclude that sensor failure “did not significantly impact” measurements they made with the BHFS in both Allen et al. (2013a) and (2015a). Similarly, a recent study of natural gas compressor stations found no systematic bias between site-level emission estimates based on component-level measurements made predominantly with the BHFS and independent estimates based on a downwind tracer flux technique (Subramanian et al., 2015).

Potential malfunction of the BHFS would only affect a fraction of estimated emissions in the UT Production Studies. Specifically, measurements made with the BHFS were the primary basis for final emission estimates for only two of the five source categories examined: equipment leaks (“ELs”) and chemical injection pumps (“CIPs”) (Allen et al., 2015a, 2015b). Emissions from hydraulically fractured well completions and liquid unloadings were measured using other methods. While Allen et al. (2013a) reported emissions for pneumatic controllers (PCs) based on measurements taken with a BHFS, the UT Production Studies’ final estimate for PCs is based on the more systematic sample in Allen et al. (2015a) (600 Gg, Table 1); the latter relied primarily on a different measurement technique (only 12% of reported PC measurements in Allen et al. (2015a) were made with the BHFS). We do not assess the techniques used to quantify PC emissions in Allen et al. (2015a).

To constrain the possible underestimation in national CH₄ emissions from ELs and CIPs due to sensor transition failure in the BHFS, we analyzed Allen et al. (2013a) data from 278 individual measurements of ELs made with the BHFS (Figure 1) (Allen et al., 2013b). We simulated the potential underestimation by assuming that the BHFS systematically malfunctioned when measuring emissions from every EL and CIP with fractional CH₄ content (%CH₄) below fixed thresholds and then recalculated results after filtering out all potentially affected measurements. We considered two alternative thresholds of the %CH₄ in natural gas below which sensor transition failure could occur. The lower threshold is based on the Howard et al. (2015) observation that sensor failure can occur if the %CH₄ of the natural gas emission source is less than ∼91% (mol%). The upper threshold is based on additional analysis in Howard (2015b) that suggested the BHFS may underestimate high emitting sources when %CH₄ is as high as 97%. Because BHFS failure has not been reported for sources with %CH₄ >97%, we assume such measurements are unaffected.

More specifically, we constrain the possible error introduced into the UT Production Studies due to BHFS sensor transition failure by comparing the mean emissions of all 278 ELs measured in Allen et al. (2013a) to the average of the subsets of ELs with %CH₄ >91% and >97% (Figure 1). If sensor transition failure had systematically affected every measurement within the subset below a fixed %CH₄ threshold, then the mean of the subset of measurements above the threshold could be considered to be characteristic of the full dataset (since those measurements would presumably have been unaffected by the sensor problem). We discuss uncertainties associated with this approach in the Discussion section.

Finally, we examined the sensitivity of national emission estimates to increases of 40% and 80% in emissions from ELs and CIPs (Table 1, rightmost columns). This sensitivity range is intended to constrain the potential error caused by systematic sensor transition failure in the BHFS in the UT Production Studies and is based on our analysis of the Allen et al. (2013a) EL measurements, described above.

The UT Production Studies quantified CH₄ emissions from five sources or activities common to natural gas production sites (Table 1, Categories A-E). These five categories account for 50–60% of total estimated CH₄ emissions in 2012 from the entire natural gas production segment (Allen et al., 2015a; U.S. Environmental Protection Agency, 2014). The final emission estimates for only two categories (ELs and CIPs) were exclusively determined from BHFS-based measurements. ELs and CIPs account for only 30% of the total reported emissions from the five sources assessed in the UT Production studies and 17% of total estimated emissions from the natural gas production segment (Table 1, column 3). Howard (2015b) reports higher values of 98% and 41%, respectively, but he did not account for updated estimates to Allen et al. (2013a) presented in Allen et al. (2015a, 2015b).

The UT Production Studies report lower CH₄ emission rates for ELs having lower fractional CH₄ composition (Figure 1). It has been hypothesized that these results are flawed because of a sensor malfunction in the BHFS that may occur for emission sources with fractional CH₄ composition below 91–97% (Howard, 2015a, 2015b). To constrain the potential underestimation that such a malfunction could have caused in the UT Production Studies, we performed a sensitivity analysis by applying the mean emission rates of ELs with higher fractional CH₄ composition (>91–97% CH₄) to the entire EL dataset and determined the resulting difference in total CH₄ emissions (Table 1, columns 4 and 5).

If BHFS sensor transition failure had occurred at every site with <91% CH₄, then the mean of emissions from ELs may be up to ∼40% higher than the value reported in Allen et al. (2013a) (0.15 vs. 0.11 scfm CH₄; see Figure 1). The corresponding value if BHFS failure occurred at all sites with <97% CH₄ is ∼80% higher (0.19 vs. 0.11 scfm CH₄). For comparison, Allen et al. (2013a) reported a relative uncertainty of ±36% (95% confidence level) for the mean emission rate of U.S. ELs.

Increasing Allen et al.’s (2015a) national estimates of CH₄ emissions from ELs and CIPs by 40% and 80% to account for the possible underestimation due to BHFS sensor transition failure would increase the aggregate emissions from the five measured categories in the UT Production Studies by 12% and 24%, respectively (Table 1). After accounting for emissions from sources not measured in the UT Production Studies (using data from the EPA Greenhouse Gas Inventory) (U.S. Environmental Protection Agency, 2014), the potential increase in total emissions from the natural gas production sector is 7–14%. Moreover, the potential increase in natural gas supply chain emissions (production through local distribution) is 2–5%. We emphasize that these estimates are only intended to constrain the potential underestimation if BHFS malfunction had systematically affected every measurement of ELs and CIPs in Allen et al. (2013a) with %CH₄ <91–97% – they should not be interpreted to represent estimates of actual error in previously published results. In sum, the analysis is intended to provide context for interpreting concerns about BHFS problems on the major conclusions from the UT Production Studies.

In this section, we discuss why our assumptions are plausible and lead to a reasonable constraint on the potential error caused by BHFS sensor transition failure in the UT Production Studies. Our constrained error estimate assumes: (i) BHFS sensor transition failure is the sole cause of the lower mean of ELs with <91–97% CH₄ content (when compared to the mean of ELs >91–97% CH₄); and (ii) the emission distribution of ELs >97% CH₄ yields a reasonable upper bound for what the true mean of ELs would be in the absence of BHFS sensor transition failure. We are unable to assess these assumptions against empirical data because no comparable national datasets exist and almost all published EL emission rates are based on BHFS measurements; hence they may have been impacted by sensor failure. We acknowledge that these assumptions result in an indeterminate uncertainty and that further work is warranted to reduce it (for example, our assumptions could be tested with new datasets of EL measurements from sources with high and low %CH₄ content across multiple regions). At a minimum, our analysis helps inform the issue.

The first bounding assumption is based on Howard’s (2015b) conclusion that the relative absence of measurements >0.4 scfm in the subset of EL measurements <97% CH₄ (Figure 1) is caused by BHFS sensor transition failure. However, examination of the EL measurement data in Allen et al. (2013a) suggests that factors such as differing practices among natural gas producers (“producers”) may also play a role. Of the four regions surveyed for ELs in Allen et al. (2013a), the Appalachian region had the highest mean emission rate and the highest %CH₄ (94 of 100 ELs had >97% CH₄ and all were >95% CH₄). In turn, the Appalachian accounted for 11 of the 15 ELs measured across all regions with raw BHFS readings >0.4 scfm. Remarkably, ten of the 11 ELs in the Appalachian region >0.4 scfm were at six sites belonging to a single producer (five producers provided 47 sites in the Appalachian). Excluding the ten values >0.4 scfm attributable to that single producer would reduce the mean of Appalachian ELs from 0.18 to 0.071 scfm CH₄, a value intermediate to those observed in the regions with lower %CH₄ (0.024, 0.082 and 0.093 scfm CH₄ in the Rocky Mountain, Midcontinent, and Gulf Coast regions, respectively). Other factors like local regulations, equipment age, natural gas and oil production levels, and type of equipment could also lead to regional differences in emissions (Supplemental Material Text S1 contains analyses suggesting that lower absolute emission rates at sites with lower %CH₄ may be partially attributable to lower production rates). Different producer practices and other factors that could contribute to the data patterns in Allen et al. (2013) deserve further scrutiny. The fact that other factors can potentially contribute to the observed differences in emissions reduces the likelihood that the maximum underestimation in the UT Production Studies due to BHFS failure alone is as large as our constrained estimate.

The second bounding assumption relies on the expected absence of BHFS malfunction for the subset of sources with >97% CH₄. Specifically, we assume that the mean of emissions from the subset of EL measurements for sources >97% CH₄ represents an upper bound for EL emissions from sources with <97% CH₄. This assumption results in a reasonable constraint on the potential magnitude of BHFS-induced error unless EL emissions from low %CH₄ sources were higher than EL emissions from high %CH₄ sources. Here we discuss three reasons why our assumption is a more likely interpretation of the limited data available. First, the mass flow rate from a leaking source is a function of upstream pressure and orifice size (Green and Perry, 2007). We are unaware of a technical basis for why natural gas production sites with low %CH₄ should have higher upstream pressures or physically larger leak points than high %CH₄ sites. While the EL dataset in Allen et al. (2013b) includes site-level wellhead and separator pressure data for 80% and 60% of sites, respectively, analysis is inconclusive because it is not possible to determine what pressure corresponds to specific ELs. Notwithstanding, the median wellhead pressures at sites >97% CH₄ and <97% CH₄ are roughly equivalent (∼400 psi), while the median separator pressure is higher at sites with intermediate CH₄ content (91–97%) than at sites with < 91% or >97% CH₄ (610, 170, and 310 psi, respectively). There is no significant correlation between the magnitude of emissions and pressure (neither wellhead nor separator pressures), likely due to the uncertainty of attributing a specific pressure to each leak. Supplemental Material Spreadsheet S1 contains the pressure data and calculations discussed above.

Second, individual ELs with equal whole gas emission rates would have higher CH₄ emission rates at high %CH₄ sites compared to low %CH₄ sites, because a larger fraction of the gas emitted is CH₄. As a result, the effect of gas composition on the magnitude of CH₄ emissions supports the conservative nature of our second bounding assumption. We note that the magnitude of this effect (<20–30%) is insufficient to explain the much larger observed difference between EL emissions at low and high %CH₄ sites (a factor of 2 to 4, see Supplemental Material Spreadsheet S1).

Third, we considered the fact that low %CH₄ sites often co-produce hydrocarbon liquids (oil or condensate). Production of hydrocarbon liquids would require additional, and/or potentially more complex, equipment such as separators, tanks, compressors and liquid-level pneumatic controllers than are present at high %CH₄ sites that produce no hydrocarbon liquids. The larger population of potentially leaking components may result in a larger number of leaks per well: in Allen et al. (2013a), sites with <91% CH₄ had slightly more leaks per well compared to sites with >91% CH₄ (0.67 vs. 0.54 leaks/well). It is also possible that the use of an infrared camera to identify leaks can contribute to the higher rate at which leaks are identified at lower %CH₄ sites since smaller leaks can be detected more readily due to the infrared camera’s enhanced sensitivity to non-CH₄ hydrocarbons (Benson et al., 2006). However, even though low %CH₄ sites may have more leaks than high %CH₄ sites, there is no known relationship between the number of ELs at individual sites and the amount of gas emitted by individual ELs. In sum, we find these three potential factors are not likely to invalidate our assumption that the mean of ELs at high %CH₄ sites should be larger than the mean at low %CH₄ sites, and find no other evidence to the contrary. We thus conclude it is reasonable to assume the subset of ELs believed to be unaffected by BHFS failure (high %CH₄ sites) represents a plausible basis for constraining the potential error in BHFS measurements made at low %CH₄ sites.

A final assumption used to develop our constrained estimate of possible error due to BHFS sensor failure in the UT Production Studies is that the same scaling factors we determined for ELs also apply to the CIP dataset. Specifically, we applied an increase of 40–80% to CIP results, even though they do not exhibit patterns consistent with BHFS malfunction (CIP results are presented in the Supplemental Material Spreadsheet S1). The dataset for CIPs was smaller (N = 62) than for ELs, with a narrower range of %CH_4, with relatively fewer measurements that could have been affected. Only three sources were <91% CH₄ and 20 were <95% CH₄; no sources were >97% CH₄. The mean of measurements <95% CH₄ was more than twice as high as the mean of measurements >95% CH₄. Therefore, the direction of the bias for CIPs is the opposite of what would be expected if sensor transition failure had substantially affected results for sources with low %CH₄. It is also noteworthy that seven CIP measurements for sources between 92–96% CH₄ were higher than 0.4 scfm CH₄, the volumetric emission threshold that Howard (2015b) estimates to represent the threshold for sensor transition failure. Thus, the CIP dataset suggests that sensor transition failure does not always occur for measurements in the 91–97% CH₄ range, as we assume in the bounding scenario. In consideration of these factors, our constrained estimate is likely to overestimate the potential effect of BHFS malfunction on CIP measurements.

This work does not address Howard’s (2015b) analysis of results presented in Allen et al. (2013a) that compare site-level tracer flux measurements relative to component-by-component emission aggregations at sampled sites. While Howard (2015b) attributes higher estimates based on the tracer flux method to BHFS failure, we find that analysis to be inconclusive due to limitations in the data. Most notably, BHFS measurements were the basis for less than half of the on-site emissions estimate at two-thirds of sites; the balance of emissions was based on estimation methods (engineering estimates or emission factors). Therefore, any discrepancy may be due to an underestimate of modeled emissions from tanks or other unmeasured sources for which recent data suggest this might be the case (Lyon et al., 2016), rather than BHFS failure.

Potential BHFS sensor transition failures in the UT Production Studies would not affect our fundamental understanding that CH₄ emissions from the natural gas supply chain are substantial or that production site emissions, specifically, are an important contributor to a larger problem that needs to be addressed. For example, the estimated CH₄ leakage of Barnett Shale natural gas (where 1.6% of gas produced and delivered in that region is emitted) increases the 20-year cumulative radiative forcing of the natural gas fuel-cycle CH₄ and CO₂ emissions in the region, compared to CO₂ emissions from complete combustion of each unit of produced natural gas, by 50% (Zavala-Araiza et al., 2015b).

The possibility that the BHFS may under predict emissions from natural gas sources deserves careful examination. Given the widespread use of the BHFS in the oil and gas industry, this paper is not intended to diminish potential implications of inaccurate emission measurements from an individual source. Systematic BHFS malfunction in regions where natural gas contains low CH₄ could lead to underestimation of regional leakage estimates for source-specific populations. For example, if BHFS malfunction in the UT Production Studies had been the sole cause for differing whole gas emission rates between EL emissions in the low-%CH₄ Rocky Mountain sites and high-%CH₄ Appalachian sites, then reported Rocky Mountain leakage would have been underestimated by a factor of 4 (Supplemental Material Spreadsheet S1). BHFS users should be provided guidance on ways to prevent sensor transition failure when using the device. Howard et al. (2015) indicate that daily calibration and updated versions of firmware may minimize the occurrence of sensor transition failure.

At the same time, it is important that the effect of possible point source measurement errors on the results of previously published work be placed in context, particularly with regards to regional and national studies on CH₄ emissions from oil and gas systems. Measurement error is not the only challenge that researchers face when scaling measurement data from a sampled subset of a larger population. The challenge is compounded when the sampled population is heterogeneous along many dimensions, as is the case for natural gas production sites. Factors such as skewness of emission distributions, gas composition, well age, gas and liquids production, operating practices, and regulatory requirements, to name a few, complicate the collection of a representative sample.

For some emission categories, the effect of uncertainties in activity data may be an equally or more important source of uncertainty than measurement uncertainty in scaling up measurements into a national emission estimate. For example, Allen et al. (2015a) observed a nearly three-fold higher count of pneumatic controllers per well than estimated in the 2014 EPA Greenhouse Gas Inventory (GHGI) (2.7 vs. 1.0 controllers per well, respectively). The GHGI pneumatic count is also lower than what natural gas producers reported for 2014 to the Greenhouse Gas Reporting Program (1.6 controllers per well) (U.S. Environmental Protection Agency, 2015). Another example is ELs, for which Allen et al. (2013a) report an emission factor per well (total measured leak emissions divided by total wells at sites sampled). Yet, because of variability in the number of wells and equipment located on natural gas production sites, there is high uncertainty in applying a per-well emission factor nationally. If the national well count per site differed significantly from the count in Allen et al. (2013a), then estimating a national total for EL emissions based on the national well count alone would produce a biased result. We are unaware of any comprehensive, national-scale estimate of well counts per site. However, a study (Lyon et al., 2016) of seven US producing basins that used a spatial buffering technique to join proximate wells into sites reports an average of 1.4 wells per site (range of 1.1–2.3) – less than half the value observed in the Allen et al. (2013b) dataset. If the count of wells per site in Lyon et al. (2016) is representative for all national well sites, then the bias introduced by activity data when scaling EL emissions would be larger than this work’s constrained estimate of the potential underestimation in the national emission factor for ELs due to BHFS malfunction measurements reported in Allen et al. (2013a).

Howard et al. (2015) suggest sensor transition failure in the BHFS may play a role in the discrepancy between top-down estimates and bottom-up measurements, because historical inventories rely on measurements made with the BHFS. However, emerging data suggest that emission underestimation due to BHFS malfunction would only explain a small part of the difference, largely because the BHFS is unsuitable for measuring emissions from the sources that are likely to be the main cause of the discrepancy. Recent work documenting agreement between top-down and bottom-up estimates for oil and gas CH₄ emissions in the Barnett Shale highlights the importance of improved activity data (gathering facility counts in particular) and effective accounting for the influence of high-emitting facilities (Zavala-Araiza et al., 2015b). For example, the highest emitting 10% of production sites in the Barnett accounted for over 80% of all production site emissions; in turn, production sites were responsible for more than half of all oil and gas emissions. Understanding the component-level drivers of high-emitting facilities is a critical next step. Initial work indicates that abnormal or otherwise avoidable operating conditions, including malfunctions, contribute a substantial fraction of emissions at the highest emitting sites (sometimes called “super-emitters”) (Zavala-Araiza et al., 2015a; Mitchell et al., 2015, Zavala-Araiza et al., n.d.). A helicopter-based infrared camera survey of over 8,000 well pads in seven U.S. oil and gas production basins found roughly 500 high-emitting sources with hydrocarbon emission rates over 3.1 scfm; over 90% of observations were from tanks, with most of the remainder from flares, oil trucks, and dehydrator vents (Lyon et al., 2016). None of these source types were among the five assessed in the UT Production Studies. Importantly, the emission rates from the highest emitting sources exceed the measurement range of the BHFS (maximum emissions of 6–8 scfm) (Bacharach, Inc., 2015). Additionally, the BHFS is ill-suited for tank emission measurements due to their high concentrations of non-CH₄ hydrocarbons and airflow constraints (Brantley et al., 2015). Accurately quantifying all emissions from natural gas production sites will likely require improved measurement approaches such as site-level, downwind measurements to fully capture large emission sources that cannot be measured with the BHFS (Subramanian et al., 2015; Roscioli et al., 2015; Brantley et al., 2014; Rella et al., 2015; Yacovitch et al., 2015; Lan et al., 2015; Omara et al., 2016).

© 2016 Alvarez et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

The authors thank five anonymous reviewers for helpful suggestions that improved the manuscript.