Methane leakage from point sources in the oil and gas industry is a major contributor to global greenhouse gas emissions. The majority of such emissions come from a small fraction of “super-emitting” sources. We evaluate the emission detection and quantification capabilities of Kairos Aerospace’s airplane-based hyperspectral imaging methane emission detection system for methane fluxes of 18–1,025 kg per hour of methane (kgh(CH4)). In blinded controlled releases of methane conducted over 4 days in San Joaquin County, CA, Kairos detected 182 of 200 valid nonzero releases, including all 173 over 15 kgh(CH4) per meter per second (mps) of wind and none of the 12 nonzero releases below 8.3 kgh(CH4)/mps. Nine of the 26 releases in the partial detection range of 5–15 kgh(CH4)/mps were detected. There were no false positives: Kairos did not detect methane during any of the 21 negative controls. Plume quantification accuracy depends on the wind measurement technique, with a parity slope of 1.15 (σ = 0.037, R2 = 0.84, N = 185) using a cup-based wind meter and 1.45 (σ = 0.059, R2 = 0.80, N = 157) using an ultrasonic anemometer. Performance is comparable even with only modeled wind data. For emissions above 15 kgh/mps, quantification error scales as roughly 30%–40% of emission size, even when using wind reanalysis data instead of ground-based measurements. This reflects both uncertainty in wind measurements and in Kairos’ estimates. These findings suggest that at 2 mps winds under favorable environmental conditions in the United States, Kairos could detect and quantify over 50% of total emissions by identifying super-emitting sources.

1. Introduction

U.S. natural gas (NG) production reached 40.1 trillion cubic feet in 2019, a 56% increase since 2009 (Energy Information Administration, 2020). The shift from coal toward less carbon-intensive NG and renewables has reduced the carbon intensity of the U.S. power sector (Schivley et al., 2018). However, the climate benefits of NG cannot be fully realized if methane leaks into the atmosphere at significant rates, as methane has a global warming potential that is 28–36 times that of carbon dioxide over a 100-year period (Environmental Protection Agency [EPA], 2017).

The U.S. EPA greenhouse gas inventory states that NG and petroleum systems accounted for 32% of total U.S. methane emissions and about 4% of total U.S. greenhouse gas emissions in 2017 (EPA, 2019). Field surveys in gas-producing regions suggest that the EPA inventory underestimates NG methane emissions, likely because EPA’s process-based approach does not sufficiently account for emissions from extremely large sources (Brandt et al., 2014; Zavala-Araiza et al., 2015; Lyon et al., 2016). Emission sizes in the North American NG supply chain are found to follow a heavy-tailed distribution, where the top 5% of point sources, so-called super-emitters, contribute over 50% of total emissions (Brandt et al., 2014). A recent study indicates that 10% of the methane point sources in California, including oil and gas facilities, landfills, wastewater treatment plants, and dairy manure management sites, are responsible for 60% of the detected point-source emissions (Duren et al., 2019). Therefore, leak detection and repair (LDAR) programs could reduce the cost of detection and mitigation by allowing mitigation efforts to focus on the largest sources. Given the limited resources and manpower available for detection and repair, technologies for rapidly and accurately identifying super-emitters are essential for guiding mitigation efforts.

Close-range approaches, such as optical gas imaging, are widely employed in ground-based LDAR programs in the oil and gas industry. These methods are effective for source identification (Ravikumar et al., 2018), but can be slow and labor-intensive. Mobile systems with sensors placed on trucks, drones, or aircraft have the potential advantage of speeding up detection by avoiding the need for manual detection via in-person site visits (Ravikumar et al., 2019). In particular, mobile remote sensing via airplanes or satellites can be used to target super-emitters, providing benefits of “low per-site cost, high spatial coverage, and frequent sampling” (Fox et al., 2019).

We examine a system developed by Kairos Aerospace (henceforth “Kairos”). Kairos’ LeakSurveyor is a hyperspectral methane imaging system that is mounted on a light aircraft flown at general aviation altitudes of approximately 900 m (3,000 ft.) above ground level. The system uses an infrared imaging spectrometer to detect methane with 3 m resolution, an optical camera to create an optical surface map of the surveyed region, and GPS and inertial measurement units to record the position and orientation of the sensor (Berman et al., 2021). This system is capable of surveying roughly 400 square kilometers (150 square miles) of oil and gas infrastructure in a single day (Berman et al., 2021). See Supplementary Information (SI) Section S2 for further details.

Several other airborne methane detection technologies exist, including other forms of airborne remote sensing infrared imaging spectrometry, both thermal and shortwave (Tratt et al. 2014; Thorpe et al., 2016). The Airborne Visible-Infrared Imaging Spectrometer - Next Generation (AVIRIS-NG) airplane-based instrument developed at Jet Propulsion Laboratory is one such infrared spectrometer (Thorpe et al., 2016; Duren et al., 2019), as are technologies produced by Advisian, Baker Hughes (GE), and Seek Ops Inc., which employ laser absorption spectrometry on a helicopter, a drone, and a drone, respectively (Ravikumar et al., 2019). Another airborne methane sensing approach, employed by Scientific Aviation, requires the aircraft to circle a suspected source multiple times at different altitudes while taking in situ methane concentration measurements to estimate the total flux within the encircled area (Conley et al., 2017; Schwietzke et al., 2019). Picarro employs a similar cavity ringdown spectrometer in a joint drone-vehicle system that relies on transects of the plume (Ravikumar et al., 2019). Ball Aerospace detects methane using airplane-based differential LIDAR (Ravikumar et al., 2019). Satellites such as GHGSat have begun to produce estimates of local methane emissions, often using infrared spectrometry approaches (Varon et al., 2020).

Many of these technologies, although none of the satellites, have been evaluated using controlled release experiments. Only some of these trials demonstrate a clear blinded experimental design (Ravikumar et al., 2019). Some have sample sizes below 10, too small to draw meaningful statistical conclusions (Thorpe et al., 2016; Conley et al., 2017; Schwietzke et al., 2019). Some, including Tratt et al. (2014) and the Kairos evaluation in Schwietzke et al. (2019), focus only on detection, without evaluating methane emission quantification performance. None test genuine controlled, metered emissions above 100 kgh, far below emissions Kairos reports quantifying in the field. See the SI, Section S1 for further detail.

This study performs large-volume single-blind controlled releases, motivated in part by the Mobile Monitoring Challenge (MMC), organized by the Stanford Natural Gas Initiative and the Environmental Defense Fund. The 2018 MMC tested 10 methane detection technologies through single-blind controlled releases, with 6 out of the 10 participating technologies “correctly detecting over 90% of test scenarios (true positive plus true negative rates)” (Ravikumar et al., 2019). A similar set of single-blind tests through the Methane Observation Networks with Innovative Technology (MONITOR) program compares 12 handheld, mobile, and continuous monitoring approaches to methane detection at modest emission rates (Bell et al., 2020). The MONITOR findings demonstrate higher accuracy for handheld and mobile methods over continuous monitoring techniques, which similarly highlight the importance of high-precision follow-up detection for methane remote sensing systems.

We focus on characterizing quantification accuracy of the super-emitting methane point sources that Kairos’ technology was designed to quickly identify through aerial surveys. As a result, our emission rates are two to three orders of magnitude larger than those in the MMC, reaching over 1,000 kg of methane per hour (kgh(CH4)), as opposed to 0.29 kgh(CH4) for most near-ground technologies in the MMC and 29 kgh(CH4) for two airplane- and truck-based technologies (Ravikumar et al., 2019).

2. Materials and methods

2.1. Airplane-based methane sensing technology

Kairos’ methane detection technology uses hyperspectral imaging from the wing of a small aircraft to construct a two-dimensional image of excess methane concentrations integrated along the path between the airplane and the ground. Each image is generated through a single pass over an area. Kairos’ automated processing identifies methane plumes and calculates a wind-adjusted methane emission rate in kilograms of methane per hour per meter per second of wind (kgh(CH4)/mps, henceforth denoted kgh/mps).

As described in Schwietzke et al. (2019) and Berman et al. (2021), the Kairos system combines signal processing of optical and hyperspectral infrared images to produce estimates of the probability that a given 3 m pixel displays excess methane above the background. Kairos then identifies connected clusters of pixels identified as having high probability of excess methane and applies a simple physics-based algorithm to estimate the associated emission rate (assuming a point source) (Branson et al., 2021). As described in the patent for this system (Jones and Dieker, 2019), the spectral resolution for systems of this sort is “typically around 0.5 nm or better/finer,” suggesting a spectral resolution in this range for the Kairos system. Since data collection for Schwietzke et al. (2019), Kairos’ technology has improved in several ways, including a more sensitive infrared camera and a completely new atmospheric retrieval algorithm. Most importantly, Kairos now produces methane quantification estimates, which it did not during the trials in Schwietzke et al. (2019), which focused solely on detection.

The quantification algorithm described in Branson et al. (2021) uses a simple cross-sectional integration of excess methane concentrations within a detected plume. First, Kairos’ proprietary algorithm uses data from the spectrometer, optical camera, and GPS receiver, described in Jones and Dieker (2019), to compute pixel-level estimates of excess methane column density between the airplane and the ground. Next, they designate a spatially contiguous region as a plume if each pixel in the region has a methane level that is statistically distinguishable from the background concentration (based on a proprietary metric). Kairos estimates the wind direction based on the orientation of the vector between the point of highest excess methane concentration in the plume and the furthest point within the plume from this maximum concentration. Kairos then selects a core segment of the plume along this direction, passing through the maximum and extending about 50% of the way to each end of the plume. Kairos then estimates total excess methane levels in the plume by summing the excess methane levels in each pixel within the plume along that core segment, as in Equation 1.

Methane(kg)=ΣiExcessColumnDensityi(kg/m2)*Areai(m2).
1

Converting this methane mass to an emission rate requires assumptions of a constant emission rate, constant wind speed, and slow methane diffusion compared to wind speed. Under these conditions,

Rate(kg/s)=Methane(kg)Length(m)*WindSpeed(m/s),
2

where Length is the length of the core segment used to estimate the excess methane mass within the plume. Thus, Kairos’ algorithm produces an estimate of the wind speed-normalized methane emission rate, which must then be multiplied by an estimate or measurement of wind speed at the height of the plume (Branson et al., 2021).

Note that the spectrometer detects only methane, not other constituent components of natural gas, such as ethane. See the SI, Section S2 and Kairos’ patent for its system and Jones and Dieker (2019) for further technical detail.

2.2. Test location and set-up

The Stanford team performed 4 days of single-blind controlled releases in San Joaquin County, CA, on October 8, 10, 11, and 15, 2019. Kairos personnel were in the aircraft but were not present at the ground release site. Stanford personnel designed the methane release schedule and controlled the release rates with assistance from a natural gas release operator, Rawhide Leasing.

We measured methane flow rates through Sierra Instruments QuadraTherm 740i thermal mass flow meters (Sierra, 2019). We measured wind speed and direction using both a Vantage Vue Sensor Suite with a cup-based wind meter and a Gill Instruments WindSonic 60 two-dimensional ultrasonic anemometer (not present on the first day of data collection) (Davis, 2018; Gill, 2019). See the SI, Section S3 for further detail.

2.3. Single-blind experimental design

The aerial test used a two-person airplane occupied by one pilot and one Kairos engineer, with Kairos’ LeakSurveyor instrument fastened to one wing strut. The Kairos engineer oversaw operations and radio communication with ground crews from Stanford and Rawhide. As the aircraft passed over the test site, the Kairos instrument attempted to detect any methane below. The aircraft flew repeated North–South round-trip passes on a fixed route, passing overhead roughly every 4 min, varying from 3 to 5 min depending on wind and other environmental conditions.

Kairos did not have access to data collected on the ground until they reported final results to Stanford on October 24. Kairos then received actual release rates and ground-based wind measurements on October 29. See the SI, Section S4 for further detail.

2.4. Performance metrics

We test Kairos’ technology for detection accuracy, minimum detection threshold, and quantification accuracy. Here, detection accuracy is defined as the sum of true positive and true negative rates. The minimum detection threshold analysis characterizes both the minimum release rate that the technology can detect with some nonzero probability and the rate above which all releases are detected. Quantification accuracy compares the estimated methane release rates to the true release rate. We compute quantification accuracy using a linear fit of released v. detected methane to assess the accuracy of the detection method. For simplicity and intercomparability with other controlled release tests of methane detection technologies, we use an ordinary least squares linear regression in the main analysis, although we discuss the potential implications of weighted least squares approaches that account for variation in uncertainty across points in the SI, Section S5.

3. Results

3.1. Data summary

A total of 230 data points were collected during the 4-day single-blind tests, among which 21 (approximately 9%) were negative controls during which no methane was released. Forty releases (approximately 17%) were dedicated to characterizing the detection threshold by releasing at a rate between 0 and 50 kgh. The remaining large releases (approximately 74% of releases) were focused on testing the ability of the system to quantify high release volumes. Among these, 110 releases were within the range of 50 and 500 kgh; and 59 were over 500 kgh. Note that the reported methane flow rates are 93.5% of metered natural gas flow rates (Pacific Gas and Electric Co., 2019). The volume rates are converted to mass rates based on the molar mass of methane and the molar volume at the standard condition of 1 atmosphere and 15 °C (GPSA, 2011).

Of the 230 data points collected, we exclude 9 from the baseline analysis due to technical issues such as an incomplete plume image or controlled release practices that deviated from protocol. Four additional overflights did not result in valid data collection due to an incorrect flight altitude (see the SI, Section S3 for detail). When using wind speed from the cup meter, we exclude an additional 8 data points from the 230 data points with measured 1-min gust wind speed lower than 0.9 mps (2 miles per hour), the rated uncertainty. We also exclude data points from the quantification analysis if there is not sufficient time after a change in release level for full plume development. See the SI, Section S6 for further detail.

Figure 1 shows false color images of methane plumes detected by the Kairos instrument during the trial, with blue and white representing low and high concentrations, respectively. All connected pixels, with high enough confidence in detected excess methane, are considered to be within a single plume. If there are multiple disconnected plumes, we consider the closest plume to the release point, consistent with Kairos’ internal practices. Figure 1a shows a Kairos image while no methane is being released. Figure 1b shows a small plume at a release rate of 36 kgh, approaching the minimum detection threshold of the instrument. The plume in Figure 1c is clearly visible, with a wind-adjusted release rate of 87 kgh. Figures 1d–f show larger plumes with a wider field of view. See the SI, Section S7 for plume images in terms of raw pixel-level excess methane column concentration.

Figure 1.

Examples of detected plumes associated with different methane release rates. Colorized plume images are based on post-processed spectrometer images, with blue and white representing low and high confidence of detected excess methane, respectively. Optical images were taken from the airplane as it passed overhead. Each image includes the measured methane release rate, in kgh, and wind speed from the ultrasonic anemometer. Note that the scale changes in the bottom row, d–f. (a) No release. (b) Small release, close to detection threshold. (c) Medium-sized release, low wind. (d) Medium-sized release, moderate wind. (e) Large release, moderate wind. (f) Approaching maximum release rate, moderate wind. Note that the plume images are based not on direct methane concentration measurements but on assessed confidence in the presence of excess methane, based on both the background concentration and the local variance in the strength of methane-indicating spectra. DOI: https://doi.org/10.1525/elementa.2021.00063.f1

Figure 1.

Examples of detected plumes associated with different methane release rates. Colorized plume images are based on post-processed spectrometer images, with blue and white representing low and high confidence of detected excess methane, respectively. Optical images were taken from the airplane as it passed overhead. Each image includes the measured methane release rate, in kgh, and wind speed from the ultrasonic anemometer. Note that the scale changes in the bottom row, d–f. (a) No release. (b) Small release, close to detection threshold. (c) Medium-sized release, low wind. (d) Medium-sized release, moderate wind. (e) Large release, moderate wind. (f) Approaching maximum release rate, moderate wind. Note that the plume images are based not on direct methane concentration measurements but on assessed confidence in the presence of excess methane, based on both the background concentration and the local variance in the strength of methane-indicating spectra. DOI: https://doi.org/10.1525/elementa.2021.00063.f1

3.2. Detection probability and false positive rate

Kairos previously published work reporting a 50% probability of detection at 8.2 kgh/mps (Berman et al., 2021). Considering the limited resources available for this study and the interest in testing quantification accuracy at large emission rates, only 17% of data points have nonzero release rates below 50 kgh, generally corresponding to at most rates of 25 kgh/mps, a level at which previous internal Kairos tests suggest the instrument reliably detects emissions with close to 100% probability. After accounting for data exclusion criteria and the wind speed conditions at the time of the release, 36 valid data points fall in the range of 0–25 kgh/mps. We present this subset of the full data set in Figure 2.

Figure 2.

Binary detection results and the proportion of releases detected when the true release rates fall in the range of 0–25 kgh/mps. Each bin has a width of 5 kgh/mps. Kairos detected 100% of emissions above 15 kgh/mps. The smallest release detected was 8.3 kgh/mps. Error bars show twice the standard error assuming a binomial distribution. The fraction at the bottom of each bin denotes the number of true positives divided by the total number of releases in this range. Small circles on the top and bottom of the histogram represent each emission and whether it was detected. DOI: https://doi.org/10.1525/elementa.2021.00063.f2

Figure 2.

Binary detection results and the proportion of releases detected when the true release rates fall in the range of 0–25 kgh/mps. Each bin has a width of 5 kgh/mps. Kairos detected 100% of emissions above 15 kgh/mps. The smallest release detected was 8.3 kgh/mps. Error bars show twice the standard error assuming a binomial distribution. The fraction at the bottom of each bin denotes the number of true positives divided by the total number of releases in this range. Small circles on the top and bottom of the histogram represent each emission and whether it was detected. DOI: https://doi.org/10.1525/elementa.2021.00063.f2

Note that we present results in these wind-normalized units for two reasons. First, Kairos’ instrument outputs readings in wind-normalized terms, so this presentation of results disentangles instrument capabilities from the wind profile of the region in question. Second, these releases were on the low end of what our release apparatus could accurately meter. As a result, for many of these smaller releases we left the release level constant and allowed the wind to provide the variability. Thus, converting wind-normalized releases to absolute methane fluxes would remove this variability. See the SI, Section S6.4 for minimum detection results presented without wind normalization.

Figure 2 shows the fraction of emissions detected by Kairos as a function of the wind-speed-normalized methane release rate for the 35 points below 25 kgh/mps, using 1-min gust measurements from the cup wind meter. Small circles on the top and bottom of the histogram represent each emission and whether it was detected. Only 1 of the 14 data points in the 5–10 kgh/mps range was detected, with a true emission rate of 8.3 kgh/mps. The detection rate rises to 67% for release rates of 10–15 kgh/mps. Above 15 kgh/mps, 100% of emissions were detected, both in this subsample and in the data set as a whole. Thus, the 50% probability of detection threshold likely occurs between 8.3 and 15 kgh/mps, roughly consistent with Kairos’ internal trials. Error bars represent twice the standard error assuming a binomial distribution, with no error bars shown for cases with 100% or 0% detection rates. This suggests that the instrument can detect all emissions above about 15 kgh/mps with high probability.

Note that due to sensitive manual flow controls and high relative meter error and flow variability at these low flow rates, for this section of the analysis, we opted to hold the overall methane release rate relatively constant for extended periods of time, allowing changes in wind speed to provide variability in the wind-normalized release rates that Kairos’ method produces. As a result, we characterize the minimum detection threshold in terms of wind-normalized methane release rates but do not have sufficient variability in the overall flow rate to quantify the minimum detection threshold in terms of methane flow rate.

To test for false positives, we devote approximately 9% of releases (21 releases) to negative controls with a release rate of 0 kgh. Kairos reported no detections during these periods, leading to a false positive rate of 0%. This is in part because such remote sensing techniques are less sensitive than many other methane detectors, missing small emissions but rarely triggering false positives. Thus, Kairos detected all 173 releases over 15 kgh/mps and none of the 12 nonzero releases below 8.3 kgh/mps.

In all, Kairos detected 182 of 200 valid nonzero releases and had no false positives in the 21 negative controls, resulting in an overall accuracy of 91.9%, with 100% accuracy for releases above 15 kgh/mps.

Hundred percent detection above 15 kgh/mps represents an apparent improvement in the technology’s minimum detection capabilities compared to a controlled release field trial by Schwietzke et al. (2019) of an earlier version of the technology. Schwietzke et al. (2019) found that Kairos detected emissions of 68.6 and 91.5 kgh with probabilities of 50% and 67%, respectively, including only passes with favorable environmental conditions, or 17% and 29%, respectively using all six or seven passes, respectively, regardless of environmental conditions. That said, these results are not directly comparable because wind speed is not reported in Schwietzke et al. (2019).

3.3. Quantification accuracy

Figure 3a shows 185 valid data points associated with nonzero release rates, comparing the metered release rates (x-axis) to the estimated rates generated by Kairos. For consistency with Kairos’ internal testing procedures, we use 1-min gust wind speed from the cup wind meter to convert the Kairos-reported emission estimate in kgh/mps to kgh, using wind speed measured at the time of each pass. Kairos reports point estimates in kgh/mps with no estimate of the uncertainty in these results. There is, of course, uncertainty surrounding these point estimates, and the data collected in this field trial allow us to estimate it empirically. Uncertainties in wind measurements are approximately ± 0.9 mps (±2 miles per hour) for the cup wind meter. This introduces errors in Kairos’ estimates of release rates due to uncertainty in wind speed, shown in the Y error bars in Figure 3a. The length of the error bars is thus dependent on the magnitude of the Kairos-reported number in kgh/mps. The y-axis of Figure 3b shows the Kairos-reported number multiplied by 1-min gust wind speed measured with the ultrasonic anemometer, which has a rated accuracy of roughly ±2%, with some variation depending on wind speed (Gill, 2019). In this case, the length of the error bars depends on both the magnitude of the measured wind speed and the Kairos-reported quantification in kgh/mps. Because these error bars do not include uncertainty in Kairos’ quantification estimates, they necessarily underestimate uncertainty. Although the ultrasonic anemometer has a much smaller measurement uncertainty, it was not present for the first day of data collection. Therefore, we use results from the cup wind meter, shown in Figure 3a as a baseline. See the SI, Section S8 for further detail. See the SI, Section S9 for further detail on uncertainty and variability in the measured natural gas flow rate.

Figure 3.

Parity chart of actual nonzero methane release rates and the corresponding Kairos-reported estimate in kgh/mps multiplied by 1-min gust wind speed measured by (a) the cup wind meter, (b) the ultrasonic anemometer or reported by (c) height-adjusted values from the Dark Sky commercial wind reanalysis database, and (d) surface gusts from the High-Resolution Rapid Refresh (HRRR) database. The type of wind used in (a–c) is 1-min gust wind speed. The X = Y parity line indicates perfect quantification. All four cases show a relatively close linear fit. (a–c) show mild to moderate bias toward overestimation based on 1-min gust wind and (d) shows a mild underestimation based on hourly gust. The Dark Sky wind used in (c) is converted to 2.5-m wind from 10-m wind by applying a height adjustment factor. The HRRR wind used in (d) uses the method from Duren et al. (2019), averaging hourly surface gusts over 3 h in the nearest 3 × 3 measurement locations (a box of 9 km by 9 km). See the SI, Section S6.3.2 for further detail on HRRR winds. 95% confidence intervals of the regression fits are shown. n = number of data points shown in each graph, which depends on data exclusion criteria described in the SI, Section S6.1. Y error bars are based on wind uncertainties, described in the SI, Section S8. Note that wind measurement uncertainty in the ultrasonic anemometer is smaller than point size, while Dark Sky does not report uncertainty. X error bars, not visible, are based on the observed flow variability and flow meter error, described in the SI, Section S9. DOI: https://doi.org/10.1525/elementa.2021.00063.f3

Figure 3.

Parity chart of actual nonzero methane release rates and the corresponding Kairos-reported estimate in kgh/mps multiplied by 1-min gust wind speed measured by (a) the cup wind meter, (b) the ultrasonic anemometer or reported by (c) height-adjusted values from the Dark Sky commercial wind reanalysis database, and (d) surface gusts from the High-Resolution Rapid Refresh (HRRR) database. The type of wind used in (a–c) is 1-min gust wind speed. The X = Y parity line indicates perfect quantification. All four cases show a relatively close linear fit. (a–c) show mild to moderate bias toward overestimation based on 1-min gust wind and (d) shows a mild underestimation based on hourly gust. The Dark Sky wind used in (c) is converted to 2.5-m wind from 10-m wind by applying a height adjustment factor. The HRRR wind used in (d) uses the method from Duren et al. (2019), averaging hourly surface gusts over 3 h in the nearest 3 × 3 measurement locations (a box of 9 km by 9 km). See the SI, Section S6.3.2 for further detail on HRRR winds. 95% confidence intervals of the regression fits are shown. n = number of data points shown in each graph, which depends on data exclusion criteria described in the SI, Section S6.1. Y error bars are based on wind uncertainties, described in the SI, Section S8. Note that wind measurement uncertainty in the ultrasonic anemometer is smaller than point size, while Dark Sky does not report uncertainty. X error bars, not visible, are based on the observed flow variability and flow meter error, described in the SI, Section S9. DOI: https://doi.org/10.1525/elementa.2021.00063.f3

Using winds from the cup wind meter, the linear fit is relatively close to parity, with an R2 of 0.84 and a slope of 1.15 (σ = 0.037), shown in Figure 3a. The slope is statistically distinguishable from zero at the P = 0.05 level. This finding is robust to several techniques that correct for heteroskedasticity in the data, shown in the SI, Section S5. Note that the confidence intervals in Figure 3 assume homoskedasticity, which residual plots in Figure S7 suggest does not hold. Heteroskedastic confidence intervals would widen further at higher release rates. Using ultrasonic anemometer wind data, R2 drops to 0.80 and the best fit line exhibits a larger slope of 1.45 (σ = 0.059), indicating somewhat more bias.

In the field, Kairos may not have access to on-the-ground wind measurements. In these circumstances, one would likely use third-party data products to approximate local wind speed and direction. Figure 3c uses 1-min gust wind reanalysis data from Dark Sky, a private company that estimates minute-resolution wind speed at high spatial resolution across the United States based largely on publicly available data sources and atmospheric modeling (Apple, 2016). Dark Sky reports wind speed values at 10 m height, which we convert to 2.5 m values using a factor of (2.5/10)0.15 for grassland terrain, based on Banuelos-Ruedas et al. (2011). See the SI, Section S6.3.2 for further detail.

Figure 3d shows results using hourly surface gust data from the High-Resolution Rapid Refresh (HRRR) wind reanalysis database, produced by the U.S. National Oceanic and Atmospheric Administration (NOAA; 2020) averaging wind speed estimates over the nearest 9 × 9 km area for the 3 h before, during, and after the Kairos measurement, based on Duren et al. (2019). For further discussion of HRRR data, see the SI, Section S6.3.2. Note that (c) and (d) only exclude 6 and 10 data points, respectively, due to insufficient time for plume formation, while (a) excludes 15 of 200 nonzero valid data points with either incomplete time for plume formation or wind speed measurements whose uncertainty range contains zero. See the SI, Section S6.1 for further detail.

For both forms of wind reanalysis data, overall quantification performance is similar to the results with ground-based wind data, with a slightly less precise linear fit. With Dark Sky in Figure 3c, the R2 falls slightly to 0.77 with a parity slope of 1.19, between the cup wind meter and ultrasonic anemometer slopes. The R2 for HRRR falls to 0.67 with a slope of 0.88, indicating average underestimates rather than overestimates of total methane emissions.

Thus, Dark Sky data appear to provide a more precise estimate of overall emissions when ground-based wind data are not available. However, because this is a proprietary product, the underlying algorithms may change without notice. In addition, the data will likely not be publicly available after the end of 2021 (Grossman, 2020). Although HRRR data have a lower spatial and temporal resolution, the underlying process behind their production is more transparent. In addition, 15-min HRRR data are available for download within 48 h of a given date, so future Kairos flights could likely acquire publicly available HRRR data with a higher temporal resolution, potentially improving performance.

Although absolute residual plots in Figure S7a-b exhibit heteroskedasticity, percent residuals in Figure S7c–d appear relatively stationary in release size. Analysis of the smallest and largest 50% of the data (above the 100% detection threshold) demonstrates that the mean and variance are not statistically distinguishable, indicating that it is reasonable to assume that percent measurement error is roughly stationary as methane emission size increases, with a standard deviation of roughly 30%–40% of emission size. This error range represents an estimate of the uncertainty associated with methane emissions quantification from the Kairos system together with one of several potential sources of wind measurement (including model-based estimates from HRRR and Dark Sky). Even with perfect ground-based wind speed measurements, wind speeds may also vary at different heights in the atmosphere as the plume rises. As a result, it is not possible to fully disentangle uncertainty in Kairos’ measurements from wind uncertainty. Note that this computation does not include points below the 15 kgh/mps 100% detection threshold because false negatives introduce an additional form of error that is not representative of the error profile of larger emissions. See the SI, Section S5 for further detail.

These results demonstrate a high level of quantification performance, even without on-the-ground wind measurements, in terms of high R2 and low bias compared to past controlled releases for mobile methane detectors in Ravikumar et al. (2019), Duren et al. (2019), Schwietzke et al. (2019), Conley et al. (2017), and Foster-Wittig et al. (2015). That said, most controlled release studies operate at one to two orders of magnitude lower release volumes with smaller sample sizes predominantly clustered near the minimum detection threshold and most do not appear to employ a blinded experimental design. See the SI, Section S1 for further detail.

3.4. Estimate of field efficacy

Using a bottom-up inventory of 1,009 methane emission sites from the U.S. oil and gas system from Omara et al. (2018), a compilation of data from nine separate studies and eight oil and gas-producing basins, we estimate that given 2 mps winds and emission detection fractions based on the probabilities from Figure 2, adoption of this technology would detect 53% of total emissions, with 49% coming from 24 sites above the 100% detection threshold of 15 kgh/mps. At 1 mps winds, this rises to 63% of total emissions. At 4 mps winds, this falls to 41% of total emissions. At 7 mps, the maximum wind speed at which it is safe for these airplane-based surveys, Kairos would still detect 32% of total emissions. Note that this inventory combines emissions from multiple basins. In practice, detector efficacy would likely vary across basins due to different emission profiles. In addition, we do not perform a full stochastic techno-economic analysis, such as that in the Fugitive Emissions Abatement Simulation Toolkit, which would be necessary to determine the cost-effective mitigation potential of airplane-based methane sensing technology (Kemp et al., 2016).

4. Discussion

These results suggest that in suitable contexts, aerial surveys at modest wind speeds could detect 50% or more of total methane emissions even without ground-based wind measurements. This process can screen assets much more rapidly than traditional LDAR methods, with few if any resource-diverting false positives. Thus, this technology could provide rapid detection of super-emitting methane leaks in upstream and midstream oil and gas, likely as a supplement to more precise but more labor-intensive LDAR programs. More sensitive instruments would likely be required for most distribution system applications.

The overall cost of this field trial was roughly $50,000 including materials, natural gas release equipment rental, gas, personnel, flight time, space rental, and miscellaneous expenses (not including Stanford researchers’ time, which is difficult to quantify but could approximately double the cost). Companies will often participate in trials of their technology free of charge. Note that controlled release testing for more sensitive sensors aimed at lower emission volumes is substantially less expensive. Basin-wide or state-wide aerial methane emissions survey campaigns can cost $1 million or more. Testing new instruments with blinded controlled releases at a range of methane emission levels approaching those expected in the field, with a statistically meaningful sample size of at least a few dozen, would increase confidence in the capabilities of these methods, thus adding substantial value to the data from such field campaigns.

Data accessibility statement

All data and code required to reproduce the results of this article are available on GitHub at https://github.com/yuliachen/Single-blind-test-of-airplane-based-hyperspectral-methane-detection-via-controlled-releases.

Supplemental files

The supplemental materials for this article can be found as follows:

  • Text S1. Supplementary Information for single-blind test of airplane-based hyperspectral methane detection via controlled releases (https://osf.io/vqnpb/).

    • - S1. Comparison with other controlled release studies

    • - S2. Kairos Aerospace technology

    • - S3. Controlled release set-up

    • - S4. Single-blind experimental design

    • - S5. Addressing heteroskedasticity

    • - S6. Data exclusion, sensitivity analyses

    • - S7. Excess methane concentration plume images

    • - S8. Wind variability and uncertainty

    • - S9. Flow variability and uncertainty

Acknowledgments

The authors would like to thank Jingfan Wang, Jeffrey Rutherford, and Alison L. Marsden at Stanford, and Jeff Gamble and Walter Godsil from Rawhide Leasing for help with the controlled release, as well as Chris Field, Jennifer Johnson, Eric Kort, Keith Andre, and Jon Carlson for their support with wind measurements. The authors gratefully acknowledge the assistance of Tony Ramirez and the staff of Zuckerman Family Farms.

Funding

This study was funded by the Stanford Natural Gas Initiative, an industry consortium that supports independent research at Stanford University. No funding was provided by participating or tested companies.

Competing interests

The authors have no competing interests to declare.

Author contributions

EDS and YC contributed equally.

Substantial contributions to conception and design: EDS, YC, APR, ARB.

Acquisition of data: EDS, YC.

Analysis and interpretation of data: EDS, YC, ARB.

Drafting the article or revising it critically for important intellectual content: EDS, YC, APR, ARB.

Final approval of the version to be published: EDS, YC, APR, ARB.

ORCID iD

References

Apple
.
2016
.
Dark Sky data sources
.
Apple Inc
.
Available at
https://darksky.net/dev/docs/sources.
Accessed 30 October 2019
.
Banuelos-Ruedas
,
F
,
Angeles-Camacho
,
C
,
Rios-Marcuello
,
S
.
2011
.
Methodologies used in the extrapolation of wind speed data at different heights and its impact in the wind energy resource assessment in a region
.
Wind farm - Technical regulations, potential estimation and siting assessment
.
97
114
. DOI: http://dx.doi.org/10.5772/20669.
Bell
,
CS
,
Vaughn
,
T
,
Zimmerle
,
D
.
2020
.
Evaluation of next generation emission measurement technologies under repeatable test protocols
.
Elementa: Science of the Anthropocene
8
(
1
):
32
. DOI: http://dx.doi.org/10.1525/elementa.426.
Berman
,
ES
,
Wetherley
,
EB
,
Jones
,
BB
.
2021
.
Technical white paper: Methane detection
.
Kairos Aerospace
. DOI: http://dx.doi.org/10.17605/OSF.IO/HZG52.
Brandt
,
AR
,
Heath
,
GA
,
Kort
,
EA
,
O’Sullivan
,
F
,
Petron
G
,
Jordaan
SM
, Tans
P
,
Wilcox
J
,
Gopstein
AM
,
Arent
D
,
Wofsy
S
,
Brown
NJ
,
Bradley
R
,
Stucky
GD
,
Eardley
D
,
Harriss
R
.
2014
.
Methane leaks from North American natural gas systems
.
Science
343
(
6172
):
733–735
,
1095–9203
. DOI: http://dx.doi.org/10.1126/science.1247045.
Branson
,
K
,
Jones
,
BB
,
Berman
,
ES
.
2021
.
Methane emissions quantification
.
Kairos Aerospace
. DOI: http://dx.doi.org/10.17605/OSF.IO/2UNWQ.
Conley
,
S
,
Faloona
,
I
,
Mehrotra
,
S
,
Suard
,
M
,
Lenschow
,
DH
,
Sweeney
,
C
,
Herndon
,
S
,
Schwietzke
,
S
,
Pétron
,
G
,
Pifer
,
J
,
Kort
,
EA
.
2017
.
Application of Gauss’s theorem to quantify localized surface emissions from airborne measurements of wind and trace gases
.
Atmospheric Measurement Techniques
10
(
9
):
3345
3358
.
ISSN 1867–8548
. DOI: http://dx.doi.org/10.5194/amt-10-3345-2017.
Davis
.
2018
.
Vantage vue weather station. Davis Instruments
.
Available at
https://www.davisinstruments.com/product_documents/weather/spec_sheets/6250_6351_57_SS.pdf.
Accessed 19 December 2020
.
Duren
,
RM
,
Thorpe
,
AK
,
Foster
,
KT
,
Rafiq
,
T
,
Hopkins
,
FM
,
Yadav
,
V
,
Bue
,
BD
,
Thompson
,
DR
,
Conley
,
S
,
Colombi
,
NK
,
Frankenberg
,
C
.
2019
.
California’s methane super-emitters
.
Nature
575
(
7781
):
180
184
. DOI: http://dx.doi.org/10.1038/s41586-019-1720-3.
Energy Information Administration
.
2020
.
U.S. natural gas gross withdrawals
.
Washington, DC
:
U.S. Energy Information Administration
.
Available at
https://www.eia.gov/dnav/ng/hist/n9010us2a.htm.
Accessed 3 September 2020
.
Environmental Protection Agency
.
2017
.
Greenhouse gas emissions: Understanding global warming potentials
.
U.S. Environmental Protection Agency
.
Available at
https://www.epa.gov/ghgemissions/understanding-global-warming-potentials.
Environmental Protection Agency
.
2019
.
Inventory of U.S. greenhouse gas emissions and sinks: 1990–2017
.
U.S. Environmental Protection Agency
(
EPA430-R-19-001
).
Available at
https://www.epa.gov/sites/production/files/2019-04/documents/us-ghg-inventory-2019-main-text.pdf.
Foster-Wittig
,
TA
,
Thoma
,
ED
,
Albertson
,
JD
.
2015
.
Estimation of point source fugitive emission rates from a single sensor time series: A conditionally-sampled Gaussian plume reconstruction
.
Atmospheric Environment
115
:
101
109
.
ISSN 13522310
. DOI: http://dx.doi.org/10.1016/j.atmosenv.2015.05.042.
Fox
,
TA
,
Barchyn
,
TE
,
Risk
,
D
,
Ravikumar
,
AP
,
Hugenholtz
,
CH
.
2019
.
A review of close-range and screening technologies for mitigating fugitive methane emissions in upstream oil and gas
.
Environmental Research Letters
14
(
5
):
053002
. DOI: http://dx.doi.org/10.1088/1748-9326/ab0cc3.
Gill
.
2019
.
Wind speed & direction sensor. Gill Instruments
.
Available at
http://gillinstruments.com/data/datasheets/windsonic-1405-027-iss7.pdf.
Accessed 19 December 2020
.
GPSA
.
2011
. Section 1 general information, in
Price
,
B
ed.,
Engineering data book, 13th Edition (Electronic) Volume I & II
.
Tulsa, OK
:
Gas Processors Supply Association
.
Grossman
A
.
2020
.
Dark sky has a new home
.
Dark sky
.
Available at
https://blog.darksky.net/.
Accessed 26 May 2020
.
Kemp
,
CE
,
Ravikumar
,
AP
,
Brandt
,
AR
.
2016
.
Comparing natural gas leakage detection technologies using an open-source “Virtual Gas Field” simulator
.
Environmental Science & Technology
50
(
8
):
4546
4553
. DOI: http://dx.doi.org/10.1021/acs.est.5b06068.
Lyon
,
DR
,
Alvarez
,
RA
,
Zavala-Araiza
,
D
,
Brandt
,
AR
,
Jackson
,
RB
,
Hamburg
,
SP
.
2016
.
Aerial surveys of elevated hydrocarbon emissions from oil and gas production sites
.
Environmental Science & Technology
50
(
9
):
4877
4886
. DOI: http://dx.doi.org/10.1021/acs.est.6b00705.
National Oceanic and Atmospheric Administration
.
2020
.
High Resolution Rapid Refresh (HRRR) CONUS 2-D Fields GRIB2 Table Documentation
.
National Oceanic and Atmospheric Administration
.
Available at
https://rapidrefresh.noaa.gov/hrrr/HRRRv4_GRIB2_WRFTWO.txt.
Accessed 21 May 2020
.
Omara
,
M
,
Zimmerman
,
N
,
Sullivan
,
MR
,
Li
,
X
,
Ellis
,
A
,
Cesa
,
R
,
Subramanian
,
R
,
Presto
,
AA
,
Robinson
,
AL
.
2018
.
Methane emissions from natural gas production sites in the United States: Data synthesis and national estimate
.
Environmental Science & Technology
52
(
21
):
12915
12925
. DOI: http://dx.doi.org/10.1021/acs.est.8b03535.
Pacific Gas and Electric Co
.
2019
.
California gas transmission | pipe ranger | operating data | gas quality
.
Pacific Gas and Electric Co
.
Available at
https://www.pge.com/pipeline/operations/gas_quality/index.page.
Accessed 19 December 2020
.
Ravikumar
,
AP
,
Sreedhara
,
S
,
Wang
,
J
,
Englander
,
J
,
Roda-Stuart
,
D
,
Bell
,
C
,
Zimmerle
,
D
,
Lyon
,
D
,
Mogstad
,
I
,
Ratner
,
B
,
Brandt
,
AR
.
2019
.
Single-blind inter-comparison of methane detection technologies - Results from the Stanford/EDF mobile monitoring challenge
.
Elementa: Science of the Anthropocene
7
(
37
):
29
. DOI: http://dx.doi.org/10.1525/elementa.373.
Ravikumar
,
AP
,
Wang
,
J
,
McGuire
,
M
,
Bell
,
CS
,
Zimmerle
,
D
,
Brandt
,
AR
.
2018
.
“Good versus good enough?” empirical tests of methane leak detection sensitivity of a commercial infrared camera
.
Environmental Science & Technology
52
(
4
):
2368
2374
. DOI: http://dx.doi.org/10.1021/acs.est.7b04945.
Schivley
,
G
,
Azevedo
,
I
,
Samaras
,
C
.
2018
.
Assessing the evolution of power sector carbon intensity in the United States
.
Environmental Research Letters
13
(
6
):
064018
. DOI: http://dx.doi.org/10.1088/1748-9326/aabe9d.
Schwietzke
,
S
,
Harrison
,
M
,
Lauderdale
,
T
,
Branson
,
K
,
Conley
,
S
,
George
,
FC
,
Jordan
,
D
,
Jersey
,
GR
,
Zhang
,
C
,
Mairs
,
HL
,
Pétron
,
G
.
2019
.
Aerially guided leak detection and repair: A pilot field study for evaluating the potential of methane emission detection and cost-effectiveness
.
Journal of the Air & Waste Management Association
69
(
1
):
71
88
. DOI: http://dx.doi.org/10.1080/10962247.2018.1515123.
Sierra
.
2019
.
Sierra: QuadraTherm 780i. Sierra Instruments
.
Available at
https://www.sierrainstruments.com/products/quadratherm/780i.html.
Accessed 19 December 2020
.
Thorpe
,
AK
,
Frankenberg
,
C
,
Aubrey
,
AD
,
Roberts
,
DA
,
Nottrott
,
AA
,
Rahn
,
TA
,
Sauer
,
JA
,
Dubey
,
MK
,
Costigan
,
KR
,
Arata
,
C
,
Steffke
,
AM
.
2016
.
Mapping methane concentrations from a controlled release experiment using the next generation airborne visible/infrared imaging spectrometer (AVIRIS-NG)
.
Remote Sensing of Environment
179
:
104
115
. DOI: http://dx.doi.org/10.1016/j.rse.2016.03.032.
Tratt
,
DM
,
Buckland
,
KN
,
Hall
,
JL
,
Johnson
,
PD
,
Keim
,
ER
,
Leifer
,
I
,
Westberg
,
K
,
Young
,
SJ
.
2014
.
Airborne visualization and quantification of discrete methane sources in the environment
.
Remote Sensing of Environment
154
:
74
88
. DOI: http://dx.doi.org/10.1016/j.rse.2014.08.011.
Varon
,
DJ
,
Jacob
,
DJ
,
Jervis
,
D
,
McKeever
,
J
.
2020
.
Quantifying time-averaged methane emissions from individual coal mine vents with GHGSat-D satellite observations
.
Environmental Science & Technology
54
(
16
):
10246
10253
. DOI: http://dx.doi.org/10.1021/acs.est.0c01213.
Zavala-Araiza
,
D
,
Lyon
,
DR
,
Alvarez
,
RA
,
Davis
,
KJ
,
Harriss
,
R
,
Herndon
,
SC
,
Karion
,
A
,
Kort
,
EA
,
Lamb
,
BK
,
Lan
,
X
,
Marchese
,
AJ
.
2015
.
Reconciling divergent estimates of oil and gas methane emissions
.
Proceedings of the National Academy of Sciences
1091
6490
. DOI: http://dx.doi.org/10.1073/pnas.1522126112.

How to cite this article: Sherwin, ED, Chen, Y, Ravikumar, AP, Brandt, AR. 2021. Single-blind test of airplane-based hyperspectral methane detection via controlled releases. Elementa: Science of the Anthropocene 9(1). DOI: https://doi.org/10.1525/elementa.2021.00063

Domain Editor-in-Chief: Detlev Helmig, Boulder AIR LLC, Boulder, CO, USA

Guest Editor: Stefan Schwietzke, Environmental Defense Fund, Boulder, CO, USA

Knowledge Domain(s): Atmospheric Sciences; Sustainability Transitions

This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC-BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. See http://creativecommons.org/licenses/by/4.0/.

Supplementary data