Single-blind determination of methane detection limits and quantification accuracy using aircraft-based LiDAR

Bridger Photonics’ (2021) Gas Mapping LiDAR (GML) is a remote sensing system deployed by aircraft to detect and quantify methane emissions. The solution uses the absorption of light emitted from a tuned laser to produce a measurement of the path-integrated methane gas concentration between the aircraft and the ground. As the aircraft travels forward, the GML laser beam is scanned to produce a map of the path-integrated gas concentration. Aerial digital photography and topographical LiDAR are also acquired and used to provide attribution of measured emissions and processing tasks. Emission quantification estimates are produced using proprietary processing techniques, which take into account the measured gas concentrations, height information of the plume, and a local wind profile estimated by high-resolution rapid refresh (HRRR) and/or North American Mesoscale Forecast System (NAM12) (Benjamin et al., 2016; National Centers for Environmental Information, n.d.). In this study, we assess potential cross-species interference, methane detection limits, and quantification accuracy of the GML system.

Changes in regulatory requirements, reporting standards, and emerging certified gas programs are driving the oil and gas industry to consider new methods for detection and quantification of emissions, including continuous monitoring methods, mobile solutions, and remote sensing systems by aircraft and satellite. For example, the U.S. Environmental Protection Agency (2021) has proposed updates to the fugitive monitoring requirements allowing operators to use advanced leak detection technologies that can meet a rigorous minimum detection threshold; Cheniere Energy, the largest U.S. producer of liquefied natural gas, recently announced its quantification, monitoring, reporting, and verification program, which drives emission monitoring requirements up the company’s supply chain to midstream and upstream operations (Cheniere Energy, Inc., 2021); Certified gas programs, such as MiQ, Project Canary’s TrustWell Certification, and GTI’s Veritas program, seek to incentivize environmentally responsible gas production and each includes a set of methane monitoring and measurements standards (Pro, 2021; Ver, 2021; MiQ, n.d.); finally, methane emissions are also receiving international attention with the recent launch of the International Methane Emissions Observatory by the UN Environment Programme and the OGMP2.0 reporting framework (UN Environment Programme, 2021; OGMP, n.d.).

Remote emission detection methods provide an effective way to rapidly screen many facilities for emissions. Even though these systems may not be capable of detecting emission sources as small as a traditional leak detection and repair (LDAR) inspection, there is evidence that some remote sensing systems detect greater overall emissions than traditional LDAR inspections with equivalent monitoring frequency (Tyner and Johnson, 2021). Rapid screening of many facilities also potentially enables greater emissions mitigation by detecting and repairing large emission sources faster (Fox et al., 2019; Schwietzke et al., 2019; Sherwin et al., 2021). To understand the effectiveness of these systems relative to traditional LDAR methods, it is critical to understand their detection limits. Additionally, there is interest in using the quantification estimates of detected emission sources to prioritize repairs and inform emissions inventories from oil and gas systems. This requires the quantification accuracy of these solutions to be well characterized including uncertainty of individual measurements and potential bias across many measurements.

Detection limits and quantification accuracy of the GML have been investigated previously by Johnson et al. (2021) and Bridger Photonics (2021). Johnson et al. performed a total of 29 controlled releases (11 true positive [TP] detections, 10 false negative [FN] detections, and 8 other releases excluded from the analysis) during fully blind testing of the gas mapping LiDAR system. The study identified a minimum detection limit as a function of wind speed measured 3 m above ground level (AGL): $MDL[kg/h]=0.56×WindSpeed[m/s]+0.31$⁠. A linear correlation between the accuracy of quantification estimates with the accuracy of the underlying wind estimate is also reported. Bridger Photonics’ (2021) evaluation focuses on quantification accuracy of high emission rate sources (100–550 kg/h). The study completed a total of 70 measurements (all TP detections) and reports a measurement bias (regression slope) of 3%–13% depending on the wind data source with an aggregate bias across the set of measurements of ≤4%.

This study is motivated by the need for a robust, independent assessment of the detection limits and quantification accuracy of the GML system. Interference of nonmethane emissions on the instrument’s column-integrated methane signal is investigated in a laboratory experiment, and 650 individual measurement passes over single-blind controlled releases are used to develop a robust detection curve and quantification assessment.

Interference testing experiments were conducted in a controlled setting to assess the impact of other gas species on methane measurements made by the GML system. The GML instrument was installed in a stationary fixture trained on a target at a distance of 125 m. A 1-m-long sample tube was introduced within the beam path (see SI— Interference Testing). For each test, the sample tube was vacuum-evacuated and then filled to 1 atm by various blends of sample gas and nitrogen determined by partial pressures. Additionally, an independent measure of absorbance was made for each test by monitoring the GML beam power before and after the sample cell. Experiments were conducted using methane at 0.5–5% by volume in nitrogen, ethylene at 2–20% by volume in nitrogen, and ethane, propane, n-butane, i-butane, and carbon dioxide at 20–50% by volume in nitrogen and at 100% by volume.

The background-corrected, path-integrated methane concentration (ppm-m) measured by the GML instrument was compared to the volume fraction of the test gas within the sample cell using ordinary least-squares regressions and Pearson’s correlation coefficient. Very high, positive correlations were found for methane, ethylene, and n-butane. The fit for methane showed good agreement between measured and test values (slope = 0.97 and R² = 1.0). Although well-correlated, ethylene and n-butane showed only very weak responses “as methane” in the GML measurements with slope = 0.00179 and R² = 0.97 for ethylene and slope = 0.000049 and R² = 0.89 for n-butane. These responses are 500 times and 20,000 times weaker, on average, than the methane response of the GML instrument for ethylene and n-butane, respectively, which makes significant interference with methane measurements very unlikely.

During controlled release testing, a methane release rate of 453 SCFH was confirmed by the GML system at 424 SCFH, with a path-integrated methane concentration reading of 4,438 ppm-m at a flight altitude of 200 m. This is similar to the lab test case of methane at 0.5% by volume. If, somehow, the ethylene volume fraction in the column of air sampled was four times greater than methane, the GML reading would be biased high by <1% by the presence of ethylene. If the n-butane volume fraction in the column of air sampled was 200 times greater than methane, the methane reading would be biased high by <0.05%. Conversely, propane showed a very high negative correlation, and its presence in the column of air sampled at 200 times greater than methane would lead to a low bias in the methane measurement of <0.5%. In summary, the GML system seems largely insensitive to interference from the sample gas species tested. Ethylene shows the greatest level of interference, though the degree of interference is unlikely to have any meaningful impact on the results of real-world measurements made in an oil and gas production region.

Single-blind controlled release testing was performed following the survey test protocol developed by the Methane Emission Technology Evaluation Center at Colorado State University and adapted by the research team to remote sensing aircraft systems (Bell and Zimmerle, 2022). During this testing, the GML system repeated aerial passes over a controlled release system at a known location. Although the Bridger team was aware of the release location, the timing and emission rate of all controlled releases were held confidential by the study team. Bridger was instructed to conduct tests in a manner as similar to commercial operations as possible.

This approach to controlled testing allows data from many discreet passes, each under different environmental conditions and controlled release conditions to be collected in an efficient manner and is conducive to the data analysis with only a single emission source present in each pass. However, the testing approach does not perfectly match operational deployments, where emission sources may be more complex (e.g., multiple overlapping plumes, intermittent sources, varied composition, or buoyant exhaust gases) and the opportunity for optimizing the flight path over many repeated passes is limited. In this testing, the pilot maintained a figure-eight flight path for the duration of experiments and passes were not optimized to align with the wind direction or incrementally improve successive measurements.

Data from two series of controlled release experiments are included in this work.

1. Experiments in Midland, TX. In this series, the release system was overseen and operated by personnel from Colorado State University. A total of 536 passes over the release point were conducted across 5 days of testing from October 4 through October 8, 2021. These experiments were performed in Midland, TX, in an attempt to represent Bridger’s operations in the Permian basin and to provide more comparable conditions to Bridger’s internal quantification measurements (Bridger Photonics, 2021).

2. Experiments in Ehrenberg, AZ. In this series, the release system was overseen by Stanford University personnel and operated by Rawhide Leasing. A total of 114 passes over the release point were conducted across 2 days of testing from November 3 through November 4, 2021. These experiments were performed in Ehrenberg, AZ, to simplify access to high release rate equipment, and in conjunction with similar measurements performed by other airborne remote sensing instruments, the results of which will be presented elsewhere.

Descriptions of the controlled release system setup at both locations can be found in SI—Controlled Release Systems. Gas compositions from the Compressed Natural Gas trailer and Liquified Natural Gas system were analyzed by gas chromatograph (92.16% and 96.2% methane by volume, respectively, SI—Gas Composition Analyses) and were applied to all flow data to report methane flow rates in kg/h. Local wind speed and wind direction were measured and logged at 1 Hz during the release experiments using a sonic anemometer. In TX, the sonic anemometer was positioned at 4.3 m AGL. In AZ, the sonic anemometer was positioned at 10 m AGL. Wind data are extrapolated from the height of measurement to the reported plume height and to 10 m AGL assuming a logarithmic wind profile with zero-displacement plane at 0.066 m and ground roughness of 0.01 m as described by Johnson et al. (2021).

Two objectives guided the selection of release rates during the experiments.

1. Assess the detection limits of the GML system at nominal flight altitudes of 500 ft (≍152 m) AGL and 675 ft (≍205 m) AGL under different wind conditions.

2. Evaluate the accuracy of quantification estimates from the GML system.

Table 1 summarizes the aircraft passes during controlled release testing. Each pass was classified as TP, FN, false positive, or not established (NE). Further demographics of the release experiments including histograms of flight altitude, release rates, and wind speeds can be seen in SI—Controlled Release Demographics. Note, while zero false positive detections were observed in the experiments, only 4 measurement passes were performed with no controlled release present, which limits the significance of any interpretation regarding a false positive rate. Experiments conducted in TX were primarily focused on assessing the detection limits of the GML system, and therefore, the majority of data were collected at low emission rates (420 passes below 5 kg/h) and included both nominal flight altitudes. The TX experiments also included 104 passes with release rates from 5 to 172 kg/h to assess quantification accuracy at 675 ft AGL nominal flight altitude. The AZ experiments focused on characterizing quantification accuracy across a larger span of emission rates and included experiments from 3.9 to 1,428 kg/h at 675 ft AGL nominal flight altitude. By combining data from experiments conducted in both TX and AZ, the accuracy of quantification estimates from the GML system is assessed over TP detections with emission rates from 0.17 to 1,428 kg/h. Dates of the experiments were selected considering historical wind speed at the release locations in an attempt to characterize performance in wind speeds from 0 to 10 m/s. However, actual wind speed during experiments (10 m AGL) ranged from 0.48 to 8.0 m/s.

After the controlled release experiments, Bridger analyzed each pass of the aircraft independently and reported data to the study team. The reported data included the timing of the pass, the altitude of the pass, and whether or not a plume was detected. For each pass where a plume was detected, Bridger also provided the height of the detected plume, maximum concentration observed, an emission rate estimate, and the windspeed used to produce the emission rate estimate. In addition, a KMZ file was provided by Bridger showing the flight path, lidar scan coverage area (scan swath), and integrated path concentration (ppm-m) observed during each pass.

Bridger processed flight data using each of three different wind data sources; therefore, three emission rate estimates are associated with each detection. Initially, Bridger reported an emission rate estimate using wind data from the HRRR and NAM12 weather observations. After these initial two estimates were provided to the study team, the sonic anemometer data collected near the release system were provided to Bridger. A third emission estimate was produced using these local wind measurements.

Data from each pass reported by Bridger were paired with controlled release and meteorological data measured by the study team using the timestamps. Controlled release data and meteorological data for each pass are reported as the average for 65 s preceding each detection time reported by Bridger, where the detection time was unavailable (e.g., FN detections), and data were averaged instead for a 65-s window centered on the flight feature time (a timestamp reported by Bridger indicating the start of a 30-s “flight feature” within which the aircraft passed over the controlled release location). An averaging period of 65 s was selected assuming a 1-m/s wind would carry the plume from the center to the edge of a 130-m detection swath in 65 s. In some cases, the plume observed by the GML system may correspond to a longer time frame (e.g., if the plume is oriented parallel to the flight path), and in other cases, the plume observed may correspond to a shorter time frame (e.g., if the wind speed is higher than 1 m/s and the plume is oriented perpendicular to the flight path).

Passes were excluded from the analysis if the emission plume was NE for a sufficient time prior to each detection. These criteria were assessed by integrating the wind speed as measured by the sonic anemometer from the moment the controlled release started until the detection time reported by Bridger to estimate a plume length (m), where the detection time was unavailable (e.g., FN detections), the flight feature time was utilized when computing the plume length instead. The plume was considered NE and excluded from further analysis if the estimated distance the gas had traveled since the start of the release did not exceed 150 m.

Detection limits were assessed for nominal flight altitudes of 500 ft AGL and 675 ft AGL using binomial logit regression (Table 2 and Figures 1 and 2). Data are presented in terms of the release flowrate normalized by the windspeed at 10 m AGL to collapse variability in wind speed and flowrate into a single independent parameter, consistent with the analysis performed by Sherwin et al. (2021); 90.0% detection rates of 0.25 and 0.41 (kg/h)/(m/s) are observed at 500 ft AGL and 675 ft AGL, respectively. Bridger’s stated detection sensitivity is 3 kg/hr (90% probability of detection under typical conditions).

To check the assumption that detection sensitivity is a function of emission rate normalized by wind speed, data are filtered by windspeed and the binomial logit regression is repeated using the controlled release rate as the independent variable. This exercise results in detection limits similar to those predicted at the wind speed used to filter the data (SI—Detection Curves).

Quantification accuracies of TP detections were assessed for emission rate estimates calculated by Bridger using each wind source (HRRR, NAM12, and Sonic; see Table 3). The HRRR data, which is the primary wind source used by Bridger in their commercial operations, produced the most accurate emission estimates in this study with an average bias of +8.1% (R² = 0.89) as identified by linear regression with a fixed zero intercept (Figure 3, top panel). Quantification error of individual measurements ranges from −58.8% to +161.4% of the metered release rate (95% CI). Figure 3 (middle and lower panels) shows the dependence of quantification error on emission rate. In general, higher fractional error is observed at lower release rates; however, these data have little impact on the linear regression since errors at higher release rates contribute a disproportionate fraction of total residuals. This increase in quantification error may be attributed to decreasing gas concentration measurement fidelity and fewer data points per gas plume for the lower emission rate releases. It is important to note that while increased fractional error is observed, these low release rates are below Bridger’s stated detection sensitivity.

Figure 4 illustrates how the emission rate quantification results are impacted by wind speed. The top panel shows the reported emission rate (using HRRR data) versus wind speed. Increased fractional error is observed at low wind speeds (Figure 4, middle and lower panels) with data below 2 m/s biased high. The bias may be attributed to increased column densities (ppm-m) measured by the GML system from pooling, swirling, and otherwise inconsistent flow of the emission plume under low, unstable wind conditions. Reproducing Figure 3 and excluding these regions of higher fractional error (wind speed less than 2 m/s and emission rate less than 3 kg/h) does not change the regression results significantly (bias = 7.7%, R² = 0.85; SI—Quantification Accuracy). By inspecting the center panel of Figure 4, it is clear that for wind speeds greater than 3 m/s, the upper whiskers in the box and whisker plot are driven by estimates of controlled releases with low emission rates, consistent with the findings in Figure 3.

The NAM12 data produced estimates with a low bias of −20.2% (R² = 0.87; SI—Quantification Accuracy). When wind as measured by the sonic anemometer at the release site was utilized the quantification estimates had less scatter (R² = 0.96) but an increased bias of 34.2% (SI—Quantification Accuracy). Both estimates produced with NAM12 and sonic wind show a higher average fractional error accompanied by a wider confidence interval at emission rates less than 1 kg/h. The NAM12 results also exhibit increased fractional error and wider confidence interval at low wind speeds, similar to that seen in the HRRR results. Using the sonic wind measurements improved the fractional error at low wind speed, reducing the bias and confidence interval of emissions estimates below 2 m/s wind (SI—Quantification Accuracy). However, using onsite wind measurements to improve emission rate estimates is limited in practicality since oil and gas infrastructure are widely distributed and local, site-specific wind measurements are generally unavailable.

Many field studies have identified a skewed distribution of emission rates, where the largest sources contribute a disproportionately large portion of the total emissions measured during the study (Brandt et al., 2014; Zavala-Araiza et al., 2017; Zimmerle et al., 2020; Chen et al., 2022). These skewed distributions suggest an effective emissions management strategy may be to target the “fat-tail,” whereby eliminating the largest emission sources will mitigate the majority of emissions at a lower cost than spending the time and effort to detect and mitigate all of the sources. Additionally, several studies have shown “bottom-up” emissions inventories underestimate total emissions from onshore oil and gas producing regions in the United States relative to “top-down" measurements which assess total regional emissions (Alvarez et al., 2018; Rutherford et al., 2021). This has been largely attributed to underestimating the frequency, duration, and emission rate of large emission sources or “superemitters.” Improving our understanding of these emission sources has proven difficult using traditional LDAR techniques and field measurements due to (1) the relatively slow pace of traditional LDAR techniques, resulting in only a few facilities inspected and measured each day; (2) the rarity of these events, resulting in a relatively low chance of being at the “right place at the right time" to capture a large emitter; and (3) limitations in performing direct measurements of large emission sources due to accessibility, safety, and measurement equipment. Remote sensing techniques have been identified as a potential method to rapidly scan many facilities and inform bottom-up inventory models to help reconcile these differences.

Understanding the detection limits of the proposed monitoring solutions is a critical step to evaluating their suitability to support this type of screening methodology. Results from this study indicate GML have a ≥90% probability of detecting emission sources ≥1 kg/h when flying at 500 ft AGL and ≥2 kg/h when flying at 675 ft AGL, under the environmental conditions present during this series of experiment (average wind speed of 3.9 m/s). When flown under low wind conditions, the detection limits improve, likely due to emissions pooling instead of being advected away from the source, thereby resulting in larger column integrated ppm-m measurements. These detection limits are well below the “fat-tail" of emission rates typically noted in recent field measurement campaigns and demonstrate the potential of data collected by GML to contribute to improving activity estimates of smaller emission sources as well.

For example, the table in SI—Example Detections from Another Study compares the detection limits to categories of emissions measured in a recent large study of gathering compressor stations, not considering large emitters, which were handled separately in that study (Zimmerle et al., 2020). For this analysis, we assume the sources measured in that campaign were released as clearly defined point sources, suitable for recovery by GML. All of the emitter categories analyzed are “cold sources” (i.e., not combustion sources with dilute methane, high temperatures, or co-emitted combustion products), where concentrations near the source location are high, and dispersion is governed primarily by local winds. Other source types such as diffuse (underground, still vents, etc.) or low-concentration (e.g., combustion slip, heater vents, flare slip, etc.) may be less likely to detect. In 16% or 4% of emission categories analyzed, GML would detect more than 25% of emitters at 1 or 2 kg/h, respectively. However, even detecting a small number of emitters captures the majority of emissions. At 1 kg/h, GML would capture more than half the emissions in 52% of categories; at 2 kg/h in 40% of categories.

A recent study showed discrepancies in emissions detected and measured by GML and those detected and measured during OGI surveys at the same facilities, where GML found fewer sources but greater emissions overall than recorded by OGI (Tyner and Johnson, 2021). The study identified many sources that are not the subject of traditional LDAR screening, or are difficult or impossible to measure using other methods, were identified by GML and estimated to be significantly higher than current inventory estimates. This highlights the potential for remote sensing systems like GML to improve inventories and emissions models, by characterizing the emission rate of these sources, but requires a clear understanding of the quantification accuracy of these systems. It is important to note the difference between lower limits for detection and that for accurate quantification when considering use cases. In the context of prioritized and directed repairs, the quantification accuracy of individual measurements as illustrated in Figures 3 and 4 is critical to understand. However, in the context of emissions accounting, it is important to understand the bias in the cumulative emission estimate. Figure 5 shows cumulative error when emission sources are summed from smallest to largest. The high bias at low emission rates is reflected in the cumulative error from the smallest releases (far left of lower panel). However, these small sources contribute very little to the total cumulative emission rate and hence very little to total cumulative error. As the release rate increases, the error in cumulative emission rates quickly falls for all wind sources, where the quantification error of larger emission sources dominates the cumulative error. The bias in cumulative emission estimates across the full dataset is 8.2%, −17%, and 28% for the HRRR, NAM12, and sonic wind sources, respectively.

It is important to note that the detection limits identified in this study are the representative of clearly defined “point sources,” which result in a concentrated emission plume (e.g., an outdoor component leak, tank thief hatch, vent stack, compressor exhaust, or flare exhaust). It is possible that the detection limit would be higher for emission sources, which are located inside a structure (e.g., fugitive components in a compressor house, enclosed separator, or enclosed dehydrator unit) since GML will need to detect the gas leaving the structure, where the methane gas concentration is lower due to dispersion within the structure itself. While the specific location chosen for this study may be qualitatively similar to other topography in the Permian basin, no effort was made in this study to treat the effect of surface reflectivity on the results. Moreover, the wind speeds during the testing were lower than normal for the time of year of the study in Midland, TX. Lower wind speeds are more favorable conditions for detection sensitivity performance but are more difficult conditions for emission rate quantification performance.

All data required to repeat the analysis are included as supplementary material.

The supplemental files for this article can be found as follows:

1. SI.pdf - Supplementary information

We acknowledge the Bridger Photonics team including Pete Roos, Mike Thorpe, Chris Wilson, and Cameron Murdock for their support preparing and executing the testing programs.

This research was funded in part by:

1. The Environmental Partnership (TEP) of the American Petroleum Institute (API) via the Methane Emission Technology Evaluation Center (METEC) at Colorado State University.

2. ExxonMobil and the Stanford Natural Gas Initiative, an industry consortium that supports independent research at Stanford University.

The authors have no competing interests to declare.

Contributed to conception and design: CB, DZ, TV, JR, AB, ES.

Contributed to acquisition of data: CB, DZ, TV, JR, AB, ES.

Contributed to analysis and interpretation of data: CB, DZ, TV, JR, AB, ES.

Drafted and/or revised this article: CB, DZ, TV, JR, AB, ES.

Approved the submitted version for publication: CB, DZ, TV, JR, AB, ES.

How to cite this article: Bell, C, Rutherford, J, Brandt, A, Sherwin, E, Vaughn, T, Zimmerle, D. 2022. Single-blind determination of methane detection limits and quantification accuracy using aircraft-based LiDAR. Elementa: Science of the Anthropocene 10(1). DOI: https://doi.org/10.1525/elementa.2022.00080

Domain Editor-in-Chief: Detlev Helmig, Boulder AIR LLC, Boulder, CO, USA

Guest Editor: Stefan Schwietzke, Environmental Defense Fund, Boulder, CO, USA

Knowledge Domain: Atmospheric Science

Part of an Elementa Special Forum: Oil and Natural Gas Development: Air Quality, Climate Science, and Policy