You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by "Thomas G Loubrieu (Jira)" <ji...@apache.org> on 2020/09/23 21:41:00 UTC
[jira] [Updated] (SDAP-281) bigdata ingestion feedback (reading
time)
[ https://issues.apache.org/jira/browse/SDAP-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas G Loubrieu updated SDAP-281:
-----------------------------------
Resolution: Fixed
Status: Done (was: In Progress)
> bigdata ingestion feedback (reading time)
> -----------------------------------------
>
> Key: SDAP-281
> URL: https://issues.apache.org/jira/browse/SDAP-281
> Project: Apache Science Data Analytics Platform
> Issue Type: Bug
> Reporter: Thomas G Loubrieu
> Assignee: Eamon Ford
> Priority: Major
>
> While doing the ingestion on bigdata wth configuration file https://github.com/tloubrieu-jpl/sdap-ingester-config/blob/bigdata/collections.yml
> None of the dataset have been ingested so far except mur25 (for 3-4 days). No tiles are availble in solr for the other datasets.
> Some granules have been ingested thought as shown in the solr nexusgranules collection:
> "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",13752,
> "smap-l2b-sss-daily-smap-sss",7032,
> "mur25-jpl-l4-glob-v41-analysed-sst",6062,
> "modis-l3-aqua-11um-v2014-0-4km-daily-sst",6036,
> "measures-sla-jpl-1603-regrid1x1-ssh",1805,
> "avhrr-l4-glob-v2-clim-seasonal-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",365]
> Here are some information from the logs and netcdf files regarding the possible issues:
> Granule ingester log:
> Traceback (most recent call last):
> File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/consumer/Consumer.py", line 79, in _received_message
> await pipeline.run()
> File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/pipeline/Pipeline.py", line 187, in run
> self._slicer.generate_tiles(dataset, granule_name)]
> File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/TileSlicer.py", line 50, in generate_tiles
> self._tile_spec_list = self._generate_slices(dimensions)
> File "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/SliceFileByStepSize.py", line 36, in _generate_slices
> raise KeyError('Provided dimension "{}" not found in dataset'.format(dim_name))
> KeyError: 'Provided dimension "time" not found in dataset'
> Dataset measures-sla-jpl-1603-regrid1x1-ssh has dimension Time with Uppercase T:
> float Time(Time) ;
> Time:units = "Days since 1985-01-01 00:00:00" ;
> Time:long_name = "Time" ;
> Time:standard_name = "time" ;
> Time:calendar = "gregorian" ;
> Time:axis = "T" ;
> dataset smap-l2b-sss-daily-smap-sss does not have a time dimension at all.
> dataset modis-l3-aqua-11um-v2014-0-4km-daily-sst no time dimension
> dataset avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
> has a time dimension.
> dataset avhrr-l4-glob-v2-clim-seasonal-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
> does not have a time dimension.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)