Draft

data/leak_datasets/austin_leaks/readme.md

@@ -0,0 +1,50 @@
+
+Data received from Austin Water as part of public information request: PIR 32348 on February 16, 2017. Locations are the nearest street address and no other contextual information was given.
+
+Parsing the data, working out the columns and other details are in `notebooks/0_process_data/process_austin_leaks.ipynb`.
+
+----------
+
+Dates are in US/Central timezone.
+
+Locations seem to be EPSG:6578.
+
+Columns:
+
+The columns are unlabelled but the top left 47486 match leaks pending inspections https://services1.arcgis.com/PuB3FWUAxkScvfQy/ArcGIS/rest/services/LPI/FeatureServer/0
+
+The bottom right data matches leak pending repair https://services1.arcgis.com/PuB3FWUAxkScvfQy/ArcGIS/rest/services/LWOPR/FeatureServer/0
+
+```py
+columns = [
+ 'OBJECTID',
+ 'INSPFLAG',
+ 'PRI',
+ 'PROB',
+ 'PROBDTTM', # Problem datetime US/Central timezone.
+ 'SCHEDDTTM', # Scheduled datetime US/Central timezone.
+ 'SERVNO',
+ 'MAPNO',
+ 'STARTDTTM', # Start datetime US/Central timezone.
+ 'PROBCODE',
+ 'PROBDESC',
+ 'CITY',
+ 'PREDIR',
+ 'STNAME',
+ 'STNO',
+ 'STSUB',
+ 'SUFFIX',
+ 'ZIP',
+ 'ADDRKEY',
+ 'FullStreetName',
+ 'X', # X location in EPSG:6578
+ 'Y', # Y location in EPSG:6578
+ '22', # CMPLKEY (Complementary key?, INT)?, SERVER (nah thats 47486-627096, Objectid (no too high)?? 1913-1496485.
+ 'QTYCALLS', # Quantity of calls?
+ 'INITDTTM', # Initial date time US/Central timezone.
+ 'WONO', # Work order number?
+ 'LOC',
+ 'DESCRIPT', # Description
+ 'COMPDTTM' # Completion data time US/Central timezone.
+]
+```

data/leak_datasets/sawater_leaks/readme.md

@@ -0,0 +1,5 @@
+The information here is from FOI request to South Australia Water. FOI id "SN840 - FOI Determination - Leak repair work orders"
+
+Date: Thursday, March 23, 2017 1:47 PM
+
+The time column is report data, so hopefully it's the time of repair as I asked not the time that the leak was reported. I will clarify this...

data/scraped_satellite_images/NAIP_AUTX_USDA-NAIP-DOQQ/readme.md

@@ -0,0 +1,16 @@
+Files:
+- script_metadata.json - information on scraping script
+- data.h5 contains X and y
+    - X: tiff files for each band loaded into an array of shape (Leak, Bands, width, length)
+    - y: True for before the leak, False for after
+- data_metadata: array of metadata for each leak in X. Each contain info on leak, image, and image search
+
+Loading:
+```py
+# load
+metadatas = json.load(open('data_metadata.json'))
+with h5py.File('data.h5','r') as h5f:
+    X2 = h5f['X'][:]
+    y2 = h5f['y'][:]
+y
+```

data/scraped_satellite_images/NAIP_AUTX_USDA-NAIP-DOQQ/script_metadata.json

@@ -0,0 +1 @@
+{"temp_dir": "/tmp/scraping_earth_engine_NAIP_all_v2", "bands": ["R", "G", "B", "N"], "resolution_min": 1.0, "ts": "20170321-02-54-41", "crs_grid": 3857, "notebook_name": "scraping_earth_engine_NAIP_all_v2", "cache_dir": "../data/downloaded_images-scraping_earth_engine_NAIP_all_v2-USDA-NAIP-DOQQ/cache", "pixel_length": 129.0, "output_dir": "../data/downloaded_images-scraping_earth_engine_NAIP_all_v2-USDA-NAIP-DOQQ", "satellite": "USDA/NAIP/DOQQ", "time_bin_delta": 2419200}

data/scraped_satellite_images/l7-AUTX_LANDSAT-LE7_L1T/readme.md

@@ -0,0 +1,16 @@
+Files:
+- script_metadata.json - information on scraping script
+- data.h5 contains X and y
+    - X: tiff files for each band loaded into an array of shape (Leak, Bands, width, length)
+    - y: True for before the leak, False for after
+- data_metadata: array of metadata for each leak in X. Each contain info on leak, image, and image search
+
+Loading:
+```py
+# load
+metadatas = json.load(open('data_metadata.json'))
+with h5py.File('data.h5','r') as h5f:
+    X2 = h5f['X'][:]
+    y2 = h5f['y'][:]
+y
+```

data/scraped_satellite_images/l7-AUTX_LANDSAT-LE7_L1T/script_metadata.json

@@ -0,0 +1 @@
+{"resolution_min": 15.0, "crs_grid": 3857, "output_dir": "../data/20170314-05-26-52_testing_earth_engine-l7-AUTX_v2", "ts": "20170318-10-08-12", "satellite": "LANDSAT/LE7_L1T", "notebook_name": "testing_earth_engine-l7-AUTX", "bands": ["B1", "B2", "B3", "B4", "B5", "B6_VCID_1", "B6_VCID_2", "B7", "B8"], "cache_dir": "../data/20170314-05-26-52_testing_earth_engine-l7-AUTX_v2/ee_l7_AUTX-leaks_cache_v2", "pixel_length": 25.0, "temp_dir": "/tmp/testing_earth_engine-l7-AUTX-48wtrw24-20170318-10-08-12"}

data/scraped_satellite_images/l8-AUTX_LANDSAT-LC8_L1T/readme.md

@@ -0,0 +1,16 @@
+Files:
+- script_metadata.json - information on scraping script
+- data.h5 contains X and y
+    - X: tiff files for each band loaded into an array of shape (Leak, Bands, width, length)
+    - y: True for before the leak, False for after
+- data_metadata: array of metadata for each leak in X. Each contain info on leak, image, and image search
+
+Loading:
+```py
+# load
+metadatas = json.load(open('data_metadata.json'))
+with h5py.File('data.h5','r') as h5f:
+    X2 = h5f['X'][:]
+    y2 = h5f['y'][:]
+y
+```

data/scraped_satellite_images/l8-AUTX_LANDSAT-LC8_L1T/script_metadata.json

@@ -0,0 +1,13 @@
+{
+    "resolution_min": 15.0,
+    "notebook_name": "testing_earth_engine-l8-AUTX_v2",
+    "satellite": "LANDSAT/LC8_L1T",
+    "pixel_length": 25.0,
+    "crs_grid": 3857,
+    "time_bin_delta": 2419200,
+    "cache_dir": "../data/20170314-05-26-52_testing_earth_engine-l8-AUTX_v2/ee_l8_AUTX-leaks_cache_v2/ATX_",
+    "ts": "20170321-02-12-16",
+    "temp_dir": "/tmp/testing_earth_engine-l8-AUTX_v2-82dzkbkv-20170321-02-12-16",
+    "output_dir": "../data/20170314-05-26-52_testing_earth_engine-l8-AUTX_v2",
+    "bands": ["B1", "B2", "B3", "B4", "B5", "B6", "B7", "B8", "B9", "B10", "B11", "BQA"]
+}

data/scraped_satellite_images/s1-AUTX_COPERNICUS-S1_GRD/readme.md

@@ -0,0 +1,16 @@
+Files:
+- script_metadata.json - information on scraping script
+- data.h5 contains X and y
+    - X: tiff files for each band loaded into an array of shape (Leak, Bands, width, length)
+    - y: True for before the leak, False for after
+- data_metadata: array of metadata for each leak in X. Each contain info on leak, image, and image search
+
+Loading:
+```py
+# load
+metadatas = json.load(open('data_metadata.json'))
+with h5py.File('data.h5','r') as h5f:
+    X2 = h5f['X'][:]
+    y2 = h5f['y'][:]
+y
+```

data/scraped_satellite_images/s1-AUTX_COPERNICUS-S1_GRD/script_metadata.json

@@ -0,0 +1 @@
+{"ts": "20170318-10-39-35", "satellite": "COPERNICUS/S1_GRD", "crs_grid": 3857, "bands": ["VV", "HH", "VH", "HV", "angle"], "resolution_min": 10.0, "output_dir": "../data/downloaded_images_scraping_earth_engine_s1-all_COPERNICUS-S1_GRD", "temp_dir": "/tmp/scraping_earth_engine_s1-all", "notebook_name": "scraping_earth_engine_s1-all", "pixel_length": 25.0, "cache_dir": "../data/downloaded_images_scraping_earth_engine_s1-all_COPERNICUS-S1_GRD/cache", "time_bin_delta": 2419200}

data/scraped_satellite_images/s2-AUTX_COPERNICUS-S2/readme.md

@@ -0,0 +1,16 @@
+Files:
+- script_metadata.json - information on scraping script
+- data.h5 contains X and y
+    - X: tiff files for each band loaded into an array of shape (Leak, Bands, width, length)
+    - y: True for before the leak, False for after
+- data_metadata: array of metadata for each leak in X. Each contain info on leak, image, and image search
+
+Loading:
+```py
+# load
+metadatas = json.load(open('data_metadata.json'))
+with h5py.File('data.h5','r') as h5f:
+    X2 = h5f['X'][:]
+    y2 = h5f['y'][:]
+y
+```

data/scraped_satellite_images/s2-AUTX_COPERNICUS-S2/script_metadata.json

@@ -0,0 +1,12 @@
+{
+    "bands": ["B1", "B2", "B3", "B4", "B5", "B6", "B7", "B8", "B8A", "B9", "B10", "B11", "B12", "QA60"],
+    "crs_grid": 3857,
+    "ts": "20170316-06-09-45",
+    "output_dir": "../data/downloaded_images_image_testing_earth_engine_s2-AUTX_v6_COPERNICUS-S2",
+    "temp_dir": "/tmp/image_testing_earth_engine_s2-AUTX_v6",
+    "satellite": "COPERNICUS/S2",
+    "pixel_length": 25.0,
+    "resolution_min": 10.0,
+    "notebook_name": "image_testing_earth_engine_s2-AUTX_v6",
+    "cache_dir": "../data/downloaded_images_image_testing_earth_engine_s2-AUTX_v6_COPERNICUS-S2/cache"
+}