You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by "Thomas G Loubrieu (Jira)" <ji...@apache.org> on 2020/08/11 21:29:00 UTC
[jira] [Updated] (SDAP-281) bigdata ingestion feedback
[ https://issues.apache.org/jira/browse/SDAP-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas G Loubrieu updated SDAP-281:
-----------------------------------
Description:
While doing the ingestion on bigdata wth configuration file https://github.com/tloubrieu-jpl/sdap-ingester-config/blob/bigdata/collections.yml
None of the dataset have been ingested so farexcept mur25 (for 3-4 days).
Here are some information from the logs and netcdf files regarding the possible issues:
Granule ingester log:
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/consumer/Consumer.py", line 79, in _received_message
await pipeline.run()
File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/pipeline/Pipeline.py", line 187, in run
self._slicer.generate_tiles(dataset, granule_name)]
File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/TileSlicer.py", line 50, in generate_tiles
self._tile_spec_list = self._generate_slices(dimensions)
File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/SliceFileByStepSize.py", line 36, in _generate_slices
raise KeyError('Provided dimension "{}" not found in dataset'.format(dim_name))
KeyError: 'Provided dimension "time" not found in dataset'
Dataset measures-sla-jpl-1603-regrid1x1-ssh has dimension Time with Uppercase T:
float Time(Time) ;
Time:units = "Days since 1985-01-01 00:00:00" ;
Time:long_name = "Time" ;
Time:standard_name = "time" ;
Time:calendar = "gregorian" ;
Time:axis = "T" ;
dataset smap-l2b-sss-daily-smap-sss does not have a time dimension at all.
dataset modis-l3-aqua-11um-v2014-0-4km-daily-sst no time dimension
dataset avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
has a time dimension.
dataset avhrr-l4-glob-v2-clim-seasonal-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
does not have a time dimension.
was:
While doing the ingestion on bigdata wth configuration file https://github.com/tloubrieu-jpl/sdap-ingester-config/blob/bigdata/collections.yml
None of the dataset have been ingested so farexcept mur25 (for 3-4 days).
Here are some information from the logs and netcdf files regarding the possible issues:
Granule ingester log:
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/consumer/Consumer.py", line 79, in _received_message
await pipeline.run()
File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/pipeline/Pipeline.py", line 187, in run
self._slicer.generate_tiles(dataset, granule_name)]
File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/TileSlicer.py", line 50, in generate_tiles
self._tile_spec_list = self._generate_slices(dimensions)
File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/SliceFileByStepSize.py", line 36, in _generate_slices
raise KeyError('Provided dimension "{}" not found in dataset'.format(dim_name))
KeyError: 'Provided dimension "time" not found in dataset'
Dataset measures-sla-jpl-1603-regrid1x1-ssh has dimension Time with Uppercase T:
float Time(Time) ;
Time:units = "Days since 1985-01-01 00:00:00" ;
Time:long_name = "Time" ;
Time:standard_name = "time" ;
Time:calendar = "gregorian" ;
Time:axis = "T" ;
dataset smap-l2b-sss-daily-smap-sss does not have a time dimension at all.
dataset modis-l3-aqua-11um-v2014-0-4km-daily-sst no time dimension
> bigdata ingestion feedback
> --------------------------
>
> Key: SDAP-281
> URL: https://issues.apache.org/jira/browse/SDAP-281
> Project: Apache Science Data Analytics Platform
> Issue Type: Bug
> Reporter: Thomas G Loubrieu
> Priority: Major
>
> While doing the ingestion on bigdata wth configuration file https://github.com/tloubrieu-jpl/sdap-ingester-config/blob/bigdata/collections.yml
> None of the dataset have been ingested so farexcept mur25 (for 3-4 days).
> Here are some information from the logs and netcdf files regarding the possible issues:
> Granule ingester log:
> Traceback (most recent call last):
> File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/consumer/Consumer.py", line 79, in _received_message
> await pipeline.run()
> File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/pipeline/Pipeline.py", line 187, in run
> self._slicer.generate_tiles(dataset, granule_name)]
> File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/TileSlicer.py", line 50, in generate_tiles
> self._tile_spec_list = self._generate_slices(dimensions)
> File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/SliceFileByStepSize.py", line 36, in _generate_slices
> raise KeyError('Provided dimension "{}" not found in dataset'.format(dim_name))
> KeyError: 'Provided dimension "time" not found in dataset'
> Dataset measures-sla-jpl-1603-regrid1x1-ssh has dimension Time with Uppercase T:
> float Time(Time) ;
> Time:units = "Days since 1985-01-01 00:00:00" ;
> Time:long_name = "Time" ;
> Time:standard_name = "time" ;
> Time:calendar = "gregorian" ;
> Time:axis = "T" ;
> dataset smap-l2b-sss-daily-smap-sss does not have a time dimension at all.
> dataset modis-l3-aqua-11um-v2014-0-4km-daily-sst no time dimension
> dataset avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
> has a time dimension.
> dataset avhrr-l4-glob-v2-clim-seasonal-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
> does not have a time dimension.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)