Single-blind test of airplane-based hyperspectral methane detection via 1 controlled releases

8 Methane leakage from point sources in the oil and gas industry is a major contributor to global greenhouse 9 gas emissions. The majority of such emissions come from a small fraction of “super-emitting” sources. 10 We evaluate the emission detection and quantiﬁcation capabilities of Kairos Aerospace’s airplane-based 11 hyperspectral imaging methane emission detection system for methane ﬂuxes of 18 to 1,025 kilograms 12 per hour of methane (kgh( CH 4 )). In blinded controlled releases of methane conducted over four days in 13 San Joaquin County, California, USA, Kairos detected 182 of 200 valid nonzero releases, including all 14 173 over 15 kgh( CH 4 ) per meter per second (mps) of wind and none of the 12 nonzero releases below 15 8.3 kgh( CH 4 )/mps. 9 of the 26 releases in the partial detection range of 5 to 15 kgh( CH 4 )/mps were 16 detected. There were no false positives: Kairos did not detect methane during any of the 21 negative 17 controls. Plume quantiﬁcation accuracy depends on the wind measurement technique, with a parity slope 18 of 1.15 ( σ =0.037, R 2 =0.84, N=185) using a cup-based wind meter and 1.45 ( σ =0.059, R 2 =0.80, N=157) 19 using an ultrasonic anemometer. Performance is comparable even with only modeled wind data. For 20 emissions above 15 kgh/mps, quantiﬁcation error scales as roughly 30-40% of emission size, even when 21 using wind reanalysis data instead of ground-based measurements. This reﬂects both uncertainty in wind 22 measurements and in Kairos’ estimates. These ﬁndings suggest that at 2 mps winds under favorable 23 environmental conditions in the US, Kairos could detect and quantify over 50% of total emissions by 24 identifying super-emitting sources. 25


26
US natural gas (NG) production reached 40.1 trillion cubic feet in 2019, a 56% increase since 2009 EIA 27 (2020). The shift from coal toward less carbon-intensive NG and renewables has reduced the carbon 28 intensity of the US power sector Schivley et al. (2018). However, the climate benefits of NG cannot be 29 fully realized if methane leaks into the atmosphere at significant rates, as methane has a global warming 30 potential that is 28-36 times that of carbon dioxide over a 100-year period EPA (2017).

31
The US Environmental Protection Agency (EPA) greenhouse gas inventory states that NG and 32 petroleum systems accounted for 32% of total US methane emissions and about 4% of total US green-    This version of the article has been accepted for publication at Elementa: Science of the Anthropocene, after peer review but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. trolled the release rates with assistance from a natural gas release operator, Rawhide Leasing. 140 We measured methane flow rates through Sierra Instruments QuadraTherm 740i thermal mass flow October 29 th . See the SI, Section S4 for further detail.

155
Performance metrics 156 We test Kairos' technology for detection accuracy, minimum detection threshold, and quantification accu-157 racy. Here, detection accuracy is defined as the sum of true positive and true negative rates. The minimum 158 detection threshold analysis characterizes both the minimum release rate that the technology can detect 159 with some nonzero probability and the rate above which all releases are detected. Quantification accuracy 160 compares the estimated methane release rates to the true release rate. We compute quantification accu-161 racy using a linear fit of released v. detected methane to assess the accuracy of the detection method. For 162 simplicity and intercomparability with other controlled release tests of methane detection technologies, 163 we use an ordinary least squares linear regression in the main analysis, although we discuss the potential 164 implications of weighted least squares approaches that account for variation in uncertainty across points 165 in the SI, Section S5.
This version of the article has been accepted for publication at Elementa: Science of the Anthropocene, after peer review but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections.

167
Data summary 168 A total of 230 data points were collected during the four-day single-blind tests, among which 21 (∼9%) 169 were negative controls during which no methane was released. 40 releases (∼17%) were dedicated to 170 characterizing the detection threshold by releasing at a rate between 0-50 kgh. The remaining large re-171 leases (∼74% of releases) were focused on testing the ability of the system to quantify high release 172 volumes. Among these, 110 releases were within the range of 50 and 500 kgh; and 59 were over 500 kgh.

176
Of the 230 data points collected, we exclude 9 from the baseline analysis due to technical issues such 177 as an incomplete plume image or controlled release practices that deviated from protocol. 4 additional 178 overflights did not result in valid data collection due to an incorrect flight altitude (see the SI, section 179 S3 for detail). When using wind speed from the cup meter, we exclude an additional 8 data points from 180 the 230 data points with measured 1-minute gust wind speed lower than 0.9 mps (2 miles per hour), the 181 rated uncertainty. We also exclude data points from the quantification analysis if there is not sufficient 182 time after a change in release level for full plume development. See the SI, Section S6 for further detail. This version of the article has been accepted for publication at Elementa: Science of the Anthropocene, after peer review but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections.  a true emission rate of 8.3 kgh/mps. The detection rate rises to 67% for release rates of 10-15 kgh/mps.

212
Above 15 kgh/mps, 100% of emissions were detected, both in this subsample and in the data set as a 213 whole. Thus, the 50% probability of detection threshold likely occurs between 8.3 and 15 kgh/mps, con-214 sistent with Kairos' internal trials. Error bars represent twice the standard error assuming a binomial 215 distribution, with no error bars shown for cases with 100% or 0% detection rates. This suggests that the 216 instrument can detect all emissions above about 15 kgh/mps with high probability.

217
Note that due to sensitive manual flow controls and high relative meter error and flow variability at 218 these low flow rates, for this section of the analysis we opted to hold the overall methane release rate 219 relatively constant for extended periods of time, allowing changes in wind speed to provide variability 220 in the wind-normalized release rates that Kairos' method produces. As a result, we characterize the min-221 imum detection threshold in terms of wind-normalized methane release rates but do not have sufficient 222 variability in the overall flow rate to quantify the minimum detection threshold in terms of methane flow This version of the article has been accepted for publication at Elementa: Science of the Anthropocene, after peer review but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections.   has a much smaller measurement uncertainty, it was not present for the first day of data collection.

255
Therefore, we use results from the cup wind meter, shown in Figure 3(a) as a baseline. See the SI,

256
Section S8 for further detail. See the SI, Section S9 for further detail on uncertainty and variability in the 257 measured natural gas flow rate.

258
Using winds from the cup wind meter, the linear fit is relatively close to parity, with an R 2 of 0.84 and 259 a slope of 1.15 (σ = 0.037), shown in in Figure 3(a). The slope is statistically distinguishable from zero 260 at the p=0.05 level. This finding is robust to several techniques that correct for heteroskedasticity in the 261 data, shown in the SI, Section S5. Note that the confidence intervals in Figure 3 assume homoskedasticity, 262 which residual plots in Figure S7 suggest does not hold. Heteroskedastic confidence intervals would 263 widen further at higher release rates. Using ultrasonic anemometer wind data, R 2 drops to 0.80 and the 264 best fit line exhibits a larger slope of 1.45 (σ = 0.059), indicating somewhat more bias.

265
In the field, Kairos may not have access to on-the-ground wind measurements. In these circumstances, 266 one would likely use third-party data products to approximate local wind speed and direction. Figure   267 3(c) uses 1-minute gust wind reanalysis data from Dark Sky, a private company that estimates minute-  For both forms of wind reanalysis data, overall quantification performance is similar to the results 281 with ground-based wind data, with a slightly less precise linear fit. With Dark Sky in Figure 3(c), the R 2 282 falls slightly to 0.77 with a parity slope of 1.19, between the cup wind meter and ultrasonic anemometer 283 slopes. The R 2 for HRRR falls to 0.67 with a slope of 0.88, indicating average underestimates rather than 284 overestimates of total methane emissions.

285
Thus, Dark Sky data appears to provide a more precise estimate of overall emissions when ground-286 based wind data are not available. However, because this is a proprietary product, the underlying algo-287 rithms may change without notice. In addition, the data will likely not be publicly available after the 288 end of 2021 Grossman (2020). Although HRRR data have a lower spatial and temporal resolution, the 289 underlying process behind their production is more transparent. In addition, 15-minute HRRR data are 290 available for download within 48 hours of a given date, so future Kairos flights could likely acquire 291 publicly available HRRR data with a higher temporal resolution, potentially improving performance.

292
Although absolute residual plots in Figure S7(a-b) exhibit heteroskedasticity, percent residuals in Fig-293 ure S7(c-d) appear relatively stationary in release size. Analysis of the smallest and largest 50% of the 294 data (above the 100% detection threshold) demonstrates that the mean and variance are not statistically 295 This version of the article has been accepted for publication at Elementa: Science of the Anthropocene, after peer review but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: http://dx.doi.org/10.1525/elementa.2021.00063. , averaging hourly surface gusts over three hours in the nearest 3x3 measurement locations (a box of 9 km by 9 km). See the SI, Section S6.3.2 for further detail on HRRR winds. 95% confidence intervals of the regression fits are shown. n = number of data points shown in each graph, which depends on data exclusion criteria described in the SI, Section S6.1. Y error bars are based on wind uncertainties, described in the SI, Section S8. Note that wind measurement uncertainty in the ultrasonic anemometer is smaller than point size, while Dark Sky does not report uncertainty. X error bars, not visible, are based on observed flow variability and flow meter error, described in the SI, Section S9.
distinguishable, indicating that it is reasonable to assume that percent measurement error is roughly sta-