You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by GitBox <gi...@apache.org> on 2021/07/23 20:34:41 UTC
[GitHub] [incubator-sdap-ingester] wphyojpl opened a new pull request #37: wip: feat: Multiband reader
wphyojpl opened a new pull request #37:
URL: https://github.com/apache/incubator-sdap-ingester/pull/37
- converting Grid Reading Processor to accept multiple bands
- converting Swath Reading Processor to accept multiple bands
- update Collection Manager to accept both `variable` and `variables` for single and multiple bands
- convert single band input to multiple bands with 1 entry
- using temporary `nexusproto` repo
- results are stored in the following structure
```
each dataset dimension: 1x50x100
slicing (x: 30, y: 30)
ingested dimension: (11, 30, 30)
Result dataset:
[0][0][0..29] = B01[0][0][0]..B01[0][0][29]
[0][10][0..29] = B01[0][10][0]..B01[0][10][29]
[0][29][0..29] = B01[0][29][0]..B01[0][29][29]
.
.
.
[10][0][0..29] = B11[0][0][0]..B11[0][0][29]
[10][10][0..29] = B11[0][10][0]..B11[0][10][29]
[10][29][0..29] = B11[0][29][0]..B11[0][29][29]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] ngachung merged pull request #37: SDAP-322 Add support for ingestion of data with multiple variables
Posted by GitBox <gi...@apache.org>.
ngachung merged pull request #37:
URL: https://github.com/apache/incubator-sdap-ingester/pull/37
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] wphyojpl commented on a change in pull request #37: feat: Multiband reader
Posted by GitBox <gi...@apache.org>.
wphyojpl commented on a change in pull request #37:
URL: https://github.com/apache/incubator-sdap-ingester/pull/37#discussion_r687229545
##########
File path: granule_ingester/granule_ingester/processors/reading_processors/__init__.py
##########
@@ -1,5 +1,6 @@
from granule_ingester.processors.reading_processors.EccoReadingProcessor import EccoReadingProcessor
from granule_ingester.processors.reading_processors.GridReadingProcessor import GridReadingProcessor
+from granule_ingester.processors.reading_processors.GridMultiVariableReadingProcessor import GridMultiVariableReadingProcessor
Review comment:
add swath multi variable reading processor here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] skorper commented on a change in pull request #37: wip: feat: Multiband reader
Posted by GitBox <gi...@apache.org>.
skorper commented on a change in pull request #37:
URL: https://github.com/apache/incubator-sdap-ingester/pull/37#discussion_r678701644
##########
File path: granule_ingester/tests/reading_processors/test_SwathMultiBandReadingProcessor.py
##########
@@ -0,0 +1,79 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+from os import path
+
+import xarray as xr
+from nexusproto import DataTile_pb2 as nexusproto, from_shaped_array
+
+from granule_ingester.processors.reading_processors import SwathReadingProcessor
+
+
+class TestReadAscatbData(unittest.TestCase):
+ def test_read_not_empty_ascatb(self):
+ reading_processor = SwathReadingProcessor(variable=['wind_speed'],
+ latitude='lat',
+ longitude='lon',
+ time='time')
+ granule_path = path.join(path.dirname(__file__), '../granules/not_empty_ascatb.nc4')
+
+ input_tile = nexusproto.NexusTile()
+ input_tile.summary.granule = granule_path
+
+ dimensions_to_slices = {
+ 'NUMROWS': slice(0, 1),
+ 'NUMCELLS': slice(0, 82)
+ }
+ with xr.open_dataset(granule_path, decode_cf=True) as ds:
+ output_tile = reading_processor._generate_tile(ds, dimensions_to_slices, input_tile)
+
+ self.assertEqual(granule_path, output_tile.summary.granule, granule_path)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.time.shape)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.variable_data.shape)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.latitude.shape)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.longitude.shape)
+
+ def test_read_not_empty_ascatb_mb(self):
+ reading_processor = SwathReadingProcessor(variable=['wind_speed', 'wind_dir'],
Review comment:
@wphyojpl I think you meant to use `SwathMultiBandReadingProcessor` here instead of `SwathReadingProcessor`? The test is failing as-is but when I replace the processor it's working as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] wphyojpl commented on a change in pull request #37: wip: feat: Multiband reader
Posted by GitBox <gi...@apache.org>.
wphyojpl commented on a change in pull request #37:
URL: https://github.com/apache/incubator-sdap-ingester/pull/37#discussion_r678781079
##########
File path: granule_ingester/tests/reading_processors/test_SwathMultiBandReadingProcessor.py
##########
@@ -0,0 +1,79 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+from os import path
+
+import xarray as xr
+from nexusproto import DataTile_pb2 as nexusproto, from_shaped_array
+
+from granule_ingester.processors.reading_processors import SwathReadingProcessor
+
+
+class TestReadAscatbData(unittest.TestCase):
+ def test_read_not_empty_ascatb(self):
+ reading_processor = SwathReadingProcessor(variable=['wind_speed'],
+ latitude='lat',
+ longitude='lon',
+ time='time')
+ granule_path = path.join(path.dirname(__file__), '../granules/not_empty_ascatb.nc4')
+
+ input_tile = nexusproto.NexusTile()
+ input_tile.summary.granule = granule_path
+
+ dimensions_to_slices = {
+ 'NUMROWS': slice(0, 1),
+ 'NUMCELLS': slice(0, 82)
+ }
+ with xr.open_dataset(granule_path, decode_cf=True) as ds:
+ output_tile = reading_processor._generate_tile(ds, dimensions_to_slices, input_tile)
+
+ self.assertEqual(granule_path, output_tile.summary.granule, granule_path)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.time.shape)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.variable_data.shape)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.latitude.shape)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.longitude.shape)
+
+ def test_read_not_empty_ascatb_mb(self):
+ reading_processor = SwathReadingProcessor(variable=['wind_speed', 'wind_dir'],
Review comment:
Yes.. I think I haven't updated the test for swatch multi band processor.
We used to combine both single and multi band in a single swath reading processor.. But it is split now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] wphyojpl commented on a change in pull request #37: feat: Multiband reader
Posted by GitBox <gi...@apache.org>.
wphyojpl commented on a change in pull request #37:
URL: https://github.com/apache/incubator-sdap-ingester/pull/37#discussion_r687225915
##########
File path: granule_ingester/granule_ingester/processors/TileSummarizingProcessor.py
##########
@@ -69,29 +79,40 @@ def process(self, tile, dataset, *args, **kwargs):
elif tile_type == 'grid_tile':
# Grid tiles need to repeat the weight for every longitude
# TODO This assumes data axis' are ordered as latitude x longitude
- tile_summary.stats.mean = type(self).calculate_mean_for_grid_tile(data, latitudes, longitudes)
+ logger.debug(f'set grid mean. tile_summary.data_var_name: {tile_summary.data_var_name}')
+
+ try:
+ tile_summary.stats.mean = type(self).calculate_mean_for_grid_tile(data, latitudes, longitudes, len(data_var_name))
Review comment:
Need to store mean, min, max for multiple variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] wphyojpl commented on a change in pull request #37: wip: feat: Multiband reader
Posted by GitBox <gi...@apache.org>.
wphyojpl commented on a change in pull request #37:
URL: https://github.com/apache/incubator-sdap-ingester/pull/37#discussion_r681078461
##########
File path: granule_ingester/tests/reading_processors/test_SwathMultiBandReadingProcessor.py
##########
@@ -0,0 +1,79 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+from os import path
+
+import xarray as xr
+from nexusproto import DataTile_pb2 as nexusproto, from_shaped_array
+
+from granule_ingester.processors.reading_processors import SwathReadingProcessor
+
+
+class TestReadAscatbData(unittest.TestCase):
+ def test_read_not_empty_ascatb(self):
+ reading_processor = SwathReadingProcessor(variable=['wind_speed'],
+ latitude='lat',
+ longitude='lon',
+ time='time')
+ granule_path = path.join(path.dirname(__file__), '../granules/not_empty_ascatb.nc4')
+
+ input_tile = nexusproto.NexusTile()
+ input_tile.summary.granule = granule_path
+
+ dimensions_to_slices = {
+ 'NUMROWS': slice(0, 1),
+ 'NUMCELLS': slice(0, 82)
+ }
+ with xr.open_dataset(granule_path, decode_cf=True) as ds:
+ output_tile = reading_processor._generate_tile(ds, dimensions_to_slices, input_tile)
+
+ self.assertEqual(granule_path, output_tile.summary.granule, granule_path)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.time.shape)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.variable_data.shape)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.latitude.shape)
+ self.assertEqual([1, 82], output_tile.tile.swath_tile.longitude.shape)
+
+ def test_read_not_empty_ascatb_mb(self):
+ reading_processor = SwathReadingProcessor(variable=['wind_speed', 'wind_dir'],
Review comment:
@skorper I updated the swath multi band processor and its test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org