[jira] [Comment Edited] (SDAP-235) Implement Collection Ingester scheduler

["Thomas G Loubrieu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SDAP-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118909#comment-17118909 ] 

Thomas G Loubrieu edited comment on SDAP-235 at 5/28/20, 5:18 PM:
------------------------------------------------------------------

Forward processing will be implemented as a specific priority assigned to granules which "arrived" (update time) after the initial ingestion.

The "historical" granules will be all the other granules which have not been initially ingested. The will be processed with their own priority.

The ingestion configuration will have attributes to manage the 2 priority levels for the same dataset.


was (Author: tloubrieu):
Forward processing will be a specific priority assigned to granules which "arrived" (update time) after the initial ingestion.

> Implement Collection Ingester scheduler
> ---------------------------------------
>
>                 Key: SDAP-235
>                 URL: https://issues.apache.org/jira/browse/SDAP-235
>             Project: Apache Science Data Analytics Platform
>          Issue Type: New Feature
>          Components: collection-ingester
>            Reporter: Eamon Ford
>            Assignee: Eamon Ford
>            Priority: Major
>              Labels: automated-ingestion
>
> Implement a scheduling mechanism so that changes to the collections config yml file, as well as new granule files, will get picked up by the Collection Ingester. If the forward processing criteria are met (i.e. the granules are dated within the acceptable date range specified in the collections config file and the granules have not yet been ingested), then ingestion config messages should be pushed to RabbitMQ to be picked up by the granule ingester.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)