mirror of
https://github.com/wassname/satellite_leak_detection.git
synced 2026-08-20 12:50:37 +08:00
spelling
This commit is contained in:
@@ -1,78 +1,96 @@
|
||||
# Leak detection
|
||||
|
||||
*Michael Clark, Craig Baldachinno*
|
||||
**DRAFT 2017-05-01**
|
||||
|
||||
This project explores the idea that we can find leaks using satellite images and machine learning. The primary hypothesis is this: given the time and place of leak repairs and satellite images of 10-15m2 resolution we predict leaks better than a baseline defined by random guessing. Our results give and f1-score of around 0.6 compared to a dummy result of 0.5. This small result supports our hypothesis but the predictive capability is too small to be used for pipe maintenance. Our conclusion is that we can make small and uncertain predictions but with higher resolution data predictive capability may improve to useful amounts.
|
||||
*Michael Clark, Craig Baldacchino*
|
||||
|
||||
This project explores the idea that we can find leaks using satellite images and machine learning. The primary hypothesis is that given the time and place of leak repairs and given satellite images of 10-15m2 resolution we can predict leaks better than a baseline defined by random guessing. We a random forest model then tweak the data filters and model parameters using hyper-optimization. Our results give a f1-score of around 0.6 compared to a dummy f1-score of 0.5. This small improvement supports our hypothesis but the it represents a predictive capability is too poor to be used for pipe maintenance. We anticipate that repeating the experiment with higher resolution imagery may give useful levels of predictive capability.
|
||||
|
||||
# Introduction
|
||||
|
||||
Water leakage in Western Australia account for more than 10 billion litres of water and costs over a billion dollars each year. The Water Corporation can only manually survey 10-12 percent of the pipe network each year for non-visible leaks. [[0]](https://www.watercorporation.com.au/-/media/files/about-us/our-performance/annual-report-2015/water-corporation-annual-report-2015.pdf). A solution that can detect leaks without additional on-site inspections would be valuable for water utilities around the world.
|
||||
Water leakage in Western Australia accounts for more than 10 billion liters of water and costs over a billion dollars each year. The Water Corporation of Western Australia can only manually survey 10-12 percent of the pipe network each year for non-visible leaks [[1]](#wc2015). A solution that can detect leaks without additional on-site inspections would be valuable for water utilities around the world.
|
||||
|
||||
One solution is to use satellite images. Satellite derived vegetation indices can be used for the detection of water pipeline leakages in semiarid areas. [[1]](http://spie.org/Publications/Proceedings/Paper/10.1117/12.2028241). Spectral signiture profiles can also be used to detect leaks but the signitures can be complicated [[2]](http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1757291).
|
||||
One such solution is to use satellite images to detect leaks. Satellite derived vegetation indices can be used for the detection of water pipeline leakages in semiarid areas [[2]](#2). Spectral signature profiles can also be used to detect leaks but the signatures can be complicated [[3]](#3).
|
||||
|
||||
Deep learning has been successful at detecting objects from satellite images even when the images can give complicated response profiles [[3]](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.1679&rep=rep1&type=pdf). This project attempts to find leaks using satellite images - utilizing machine learning to overcome difficulties with compelex data.
|
||||
Deep learning has been successful at detecting objects from satellite images even when the images give complicated response profiles [[4]](#4). This project attempts to find leaks using satellite images by utilizing machine learning to overcome difficulties with complex data.
|
||||
|
||||
# Data
|
||||
|
||||
We have a leak datasets, obtained by a freedom of information request to the very helpful people at Austin Water. Quality is hard to verify since little context is available. For example some entries could be meter repairs, false call-outs, typos, duplicates, or multi-stage repairs.
|
||||
We have a dataset of leak repairs which was obtained by a freedom of information requests to the very helpful people at Austin Water. The dataset includes the time of repair and the location which is accurate to within one street address. The data includes some obvious false call outs, duplicates, and multi-stage repairs but these are a minority and should only decrease our accuracy by a few percent.
|
||||
|
||||
Leaks locations have been combined with satellite images taken before or after a leak repair. Images *t* days before a leak were classified as positive examples of leaks – where t is a parameter determined by experimentation to be 1.5 days (See the appendix). Images after a leak are classified as negative examples. Images were downloaded from Google Earth Engine but many satellites only pass over once a week so not every leak has corresponding images. For more see [the scraping notebooks](notebooks/scraped_satellite_images) and the readme files for each dataset. An example of this data is below:
|
||||
|
||||
These have been combined with satellite image from Google Earth Engine to make machine learning datasets. For more see [the scraping notebooks](notebooks/scraped_satellite_images) and the readme files for each dataset. An example of this data is below:
|
||||
|
||||
|
||||
|
||||
Satellites only pass over once a week at best so not every leak has a corresponding image.
|
||||
*An example of a sentinel-2 image centered on a leak*
|
||||
|
||||
|
||||
| dataset | Satellite | Leak dataset | Resolution ** |
|
||||
| dataset | Satellite | Leak dataset | Resolution * |
|
||||
|-----------------------------|--------------|----------------|-----------------|
|
||||
| s2-AUTX_v6_COPERNICUS-S2 | [Sentinel 2](https://explorer.earthengine.google.com/#detail/COPERNICUS%2FS2) | ATX | >10m2 |
|
||||
| l8-AUTX_v2_LANDSAT-LC8_L1T | [Landsat 8](https://explorer.earthengine.google.com/#detail/LANDSAT%2FLC8_L1T) | ATX | >15m2 |
|
||||
| l7-AUTX_v2_LANDSAT-LE7_L1T | [Landsat 7](https://explorer.earthengine.google.com/#detail/LANDSAT%2FLE7_L1T) | ATX | >15m2 |
|
||||
| s1-AURX_COPERNICUS-S1_GRD | [Sentinel 1](https://explorer.earthengine.google.com/#detail/COPERNICUS%2FS1_GRD) | ATX | >10m2 |
|
||||
| s2-AUTX_v6_COPERNICUS-S2 | [Sentinel 2](https://explorer.earthengine.google.com/#detail/COPERNICUS%2FS2) | ATX | >10m2 |
|
||||
| l7-AUTX_v2_LANDSAT-LE7_L1T | [Landsat 7](https://explorer.earthengine.google.com/#detail/LANDSAT%2FLE7_L1T) | ATX | >15m2 |
|
||||
| l8-AUTX_v2_LANDSAT-LC8_L1T | [Landsat 8](https://explorer.earthengine.google.com/#detail/LANDSAT%2FLC8_L1T) | ATX | >15m2 |
|
||||
|
||||
Notes:
|
||||
\* This gives source resolution. All data was interpolated
|
||||
|
||||
\* This data is combined from other rows in the table
|
||||
*A list of datasets and resolutions*
|
||||
|
||||
** This gives source resolution. All data was interpolated
|
||||
After combining the datasets we must consider filtering the data. We might filter out cloudy images or images that to many days before a leak repair. For example a leak might be repaired two days after it's first observed, making an images three days before a poor example of a leak.
|
||||
|
||||
What data filters yield the best results? To answer this question we used hyperparameter optimisation to find the data filters that have the most predictive power for each dataset. For more see the [hyperopt notebook](notebooks/3a_hyperopt.ipynb) and the data tables in the appendix.
|
||||
Which data filters yield the best results? To answer this question we used hyper-parameter optimization to find the data filters that have the most predictive power for each dataset. For more see the [hyperopt notebook](notebooks/3a_hyperopt.ipynb) and the data tables in the appendix.
|
||||
|
||||
# Models
|
||||
|
||||
Modelling and prediction is done in the [3_model notebook](notebooks/3_model.ipynb) where outputs include a map of false positives, predictions, and metrics. We tried many models and archetectures including U-Net (citation), pretrained VGG-16 (citation), decision-tree regressor, multi-layer perceptrons. No model gave any significant improvement over random forest methods so this was our primary benchmark.
|
||||
Modeling and prediction is done in the [3_model notebook](notebooks/3_model.ipynb) where outputs include a map of false positives, predictions, and metrics. We tried many models and architectures including U-Net [[5]](#unet), pre-trained VGG-16 [[6]](#vgg16), decision-tree regressers, and multi-layer perceptrons. No model gave any significant improvement over random forest methods so this was our primary model.
|
||||
|
||||
# Results
|
||||
|
||||
Using a random forest model we tested 500 models on each of our satellite datasets. Dummy scores were generated by scikit's dummy classifiers with a uniform strategy. Since we are using balanced data we can use f1-scores as our primary benchmark. If max score doesn't exceed the dummy score then it's a dataset with lots of misleading features. The table below shows the results, for more see the [2_qc_scraped_data notebook](notebooks/2_qc_scraped_data.ipynb).
|
||||
We tested 500 random forest models on each of our satellite datasets then took the mean of the best 50 models. Dummy scores were generated by scikit's dummy classifiers with a uniform strategy. Since we used balanced data we can use f1-scores as our primary benchmark. If the models max score doesn't exceed the dummy score then it's a dataset with lots of misleading features. If the max score does exceed the dummy score then it may be an example of a small sample size or it may be an improvement over random guessing. The table below shows the results, for more see the [2_qc_scraped_data notebook](notebooks/2_qc_scraped_data.ipynb).
|
||||
|
||||
| | support | dummy_f1_leak | f1_leak_max | f1_leak_mean |
|
||||
|:----------------------------|----------:|----------------:|--------------:|---------------:|
|
||||
| s1-AUTX_COPERNICUS-S1_GRD | 281 | 0.58 | 0.68 | 0.511 |
|
||||
| s2-AUTX_COPERNICUS-S2 | 224 | 0.57 | 0.66 | 0.521 |
|
||||
| l7-AUTX_LANDSAT-LE7_L1T | 1082 | 0.54 | 0.66 | 0.426 |
|
||||
| l8-AUTX_LANDSAT-LC8_L1T | 434 | 0.55 | 0.69 | 0.55 |
|
||||
| s1-AUTX_COPERNICUS-S1_GRD | 281 | 0.58 | 0.68 | 0.511 |
|
||||
|
||||
Our results show that random forest can improve on dummy benchmarks. However the difference is small, and the f1-scores are low, implying to many false positives to be useful to water utility repair teams.
|
||||
*Results of the top 50 random forest models compared to dummy scores for each dataset.*
|
||||
|
||||
Our results show that random forest can improve on dummy benchmarks. However the difference is small, and the f1-scores are low, implying many false positives.
|
||||
|
||||
# Conclusion
|
||||
|
||||
The results support our hypothesis, that machine learning can use satellite images (of 10m2 resolution) to predict water leaks. However the f1-scores indicate predictions that are to unreliable to be used in the field.
|
||||
The results support our hypothesis, that machine learning can use satellite images (of 10m2 resolution) to predict water leaks. However the f1-scores indicate that the predictions that are too unreliable to be used in the field.
|
||||
|
||||
The largest limitation in this experiment was the resolution of satellite images. We expect better results with higher resolution images, unfortunately only a limited amout of high resolution imagery is available in 2017. Using the entire digital globe catalogue would yield around 20 matches with our leaks datasets, and the same goes for the high resolution US National Agriculture Imagery Program (NAIP).
|
||||
The largest limitation in this experiment was the resolution of satellite images. We expect better results with higher resolution images, unfortunately only a limited amount of high resolution imagery is available in 2017. Using the entire Digital Globe catalog would yield around 20 overlaps with our leaks datasets and the same goes for the high resolution US National Agriculture Imagery Program (NAIP). We expect that repeating this experiment when higher resolution imagery is available will yield improved predictions.
|
||||
|
||||
# References
|
||||
|
||||
<a id="wc2015">1</a>: Water Coporation. 2015. [2015 annual report"](https://www.watercorporation.com.au/-/media/files/about-us/our-performance/annual-report-2015/water-corporation-annual-report-2015.pdf) *Online*.
|
||||
|
||||
<a id="2">2</a>: Agapiou, Athos, et al. "Use of satellite derived vegetation indices for the detection of water pipeline leakages in semiarid areas." *First International Conference on Remote Sensing and Geoinformation of Environment. International Society for Optics and Photonics*, 2013.
|
||||
|
||||
<a id="3">3</a>: Agapiou, Athos, et al. "Variations of spectral signature profiles of wet and dry targets for supporting the detection of water-leakages using satellite data." *SPIE Remote Sensing. International Society for Optics and Photonics*, 2013.
|
||||
|
||||
|
||||
<a id="4">4</a>: Mnih, Volodymyr, and Geoffrey Hinton. "Learning to detect roads in high-resolution aerial images." *Computer Vision–ECCV* 2010 (2010): 210-223.
|
||||
|
||||
<a id="vgg16">5</a>: Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. ["U-net: Convolutional networks for biomedical image segmentation."](http://arxiv.org/abs/1505.04597) *International Conference on Medical Image Computing and Computer-Assisted Intervention*. Springer International Publishing, 2015.
|
||||
|
||||
<a id="unet">6</a>: K. Simonyan, A. Zisserman. ["Very Deep Convolutional Networks for Large-Scale Image Recognition."](https://arxiv.org/pdf/1409.1556) *arXiv preprint arXiv:1409.1556*
|
||||
|
||||
# Appendix
|
||||
|
||||
## Files
|
||||
|
||||
The main files are
|
||||
- `./notebooks/0_process_data` where the leak datasets are converted
|
||||
- `./notebooks/scraped_satellite_images` where images are downloaded for each leak
|
||||
- `notebooks/0_process_data` where the leak datasets are converted
|
||||
- `notebooks/scraped_satellite_images` where images are downloaded for each leak
|
||||
- `notebooks/2_qc_scraped_data.ipynb` where test each dataset for consistency and predictive power
|
||||
- `notebooks/2a_hyperopt.ipynb` where I try varied filters to see which has the most predictive power
|
||||
- `./notebooks/2_model.ipynb` which is a jupyter-notebook script for python 3.
|
||||
- `notebooks/2_model.ipynb` which is a jupyter-notebook script for python 3.
|
||||
|
||||
## Data filters
|
||||
|
||||
@@ -85,7 +103,6 @@ To solve this I ran hyperparamater optimisation where I tried all variations usi
|
||||
top 50 scores,
|
||||
|
||||
- by metric="result_metrics_report_f1-score_leak"
|
||||
- for key="../output/hyperopt/data_filters_derived_l7_ATX.json"
|
||||
- number of trials=564
|
||||
|
||||
| | corr | mean | std | units |
|
||||
@@ -104,22 +121,15 @@ top 50 scores,
|
||||
| result_dummy_metrics_report_dummy_support_leak | -0.259 | 217.64 | 207.412 | int |
|
||||
|
||||
|
||||
Here's how I interpret this table.
|
||||
Here's an example of how we interpret the table. For the l7 datasets should we used balanced classes? We have a decent positive correlation so it's important. We had a mean of 0.98, but it's bool, so we should set it to True.
|
||||
|
||||
- balanced_classes: For the l7 datasets should we used balanced classes?
|
||||
- We have a decent positive correlation so it's important.
|
||||
- We had a mean of 0.98, but it's bool, so we should set it to True.
|
||||
|
||||
- channel_shift_range: How about channel shift range?
|
||||
- This has a decent negative correlation so it's important and having it too high gives a low f1 score.
|
||||
- The mean is 0.08+-0.06 so lets set it to 0.8.
|
||||
If we instead look at `channel_shift_range` we see it has a decent sized negative correlation which means it's important and having it too high gives a low f1 score. The mean is 0.08+-0.06 so lets set it to 0.8.
|
||||
|
||||
### Best Landsat-8 filters
|
||||
|
||||
top 50 scores,
|
||||
|
||||
- by metric="result_metrics_report_f1-score_leak"
|
||||
- for key="../output/hyperopt/data_filters_derived_l8_joined_ATX.json"
|
||||
- number of trials=564
|
||||
|
||||
| | corr | mean | std | units |
|
||||
@@ -142,7 +152,6 @@ top 50 scores,
|
||||
top 50 scores,
|
||||
|
||||
- by metric="result_metrics_report_f1-score_leak"
|
||||
- for key="../output/hyperopt/derived_s2_joined_ATX.json"
|
||||
- number of trials=1000
|
||||
|
||||
| | mean | std | corr | units |
|
||||
@@ -165,7 +174,6 @@ top 50 scores,
|
||||
top 50 scores,
|
||||
|
||||
- by metric="result_metrics_report_f1-score_leak"
|
||||
- for key="../output/hyperopt/data_filters_s1-AUTX_COPERNICUS-S1_GRD.json"
|
||||
- number of trials=1000
|
||||
|
||||
| | mean | std | corr | units |
|
||||
|
||||
Reference in New Issue
Block a user