CAS pge-configuration - stage file on batch stub without shared filesystem

[Andrea Fonti <an...@altecspace.it>]

Hi,

 

I would like to implement the following scenario (all components are
virtually separated host, implemented as a linked docker containers
topology):

 

1.       [PUSHPULL] - The push pull component download files to the staging
area

2.       [CRAWLER] - The crawler component extracts metadata and triggers
filemgr ingestion

3.       [FILEMGR] - Filemgr ingest metadata in solr and archives the
ingested file

4.       [CRAWLER] - Sends a ProductType-Ingest event to WMGR

5.       [WMGR] - Receives the event that trigger a data processing
workflow: the ingested file should be processed and a new data product
should be generated and ingested

6.       [RMGR] - Handles the request to schedule the processing task and
dispatches it to the batch stub

7.       [BATCHSTUB] - a configured batch stub performs the processing task
over the ingested file and produces the new data product

 

I already implemented 1,2,3,4,5,6 but  I'm stuck with a problem:

 

How can I instruct the batchstub to query the filemgr, initiate a data
transfer to stage the source data product in a directory on the batch stub
host, perform the processing and then stage back the new product? I would
like to not rely on a shared filesystem.

 

Any help would be really appreciated.

 

Best regards

Andrea Fonti

---- ALTEC S.p.A. ----

COMUNICAZIONE IMPORTANTE
La presente email � destinata solo al/i destinatario/i sopra indicato/i.
Pu� riguardare un messaggio confidenziale e contenere informazioni e dati di
natura RISERVATA O CLASSIFICATA. Non � consentita la sua lettura indebita,
la sua duplicazione o la sua divulgazione. Le opinioni dell'autore di questa
email non sono  necessariamente le opinioni di ALTEC S.p.A.
Nulla di quanto contenuto in questa email pu� sostituire un contratto
ovvero una obbligazione vincolante per ALTEC S.p.A.
Nel caso doveste ricevere la presente email per errore, siete
cortesemente pregati di distruggere la Vostra copia e di contattarci
telefonicamente o a mezzo email o facsimile. Grazie.

IMPORTANT NOTICE
This email is intended only for the use of the above mentioned addressee(s).
It is confidential and may contain legally PRIVILEGED OR CLASSIFIED
information and data. Unauthorized reading, copying, disclosure or use is
strictly prohibited. The views of the author hereof may not necessarily
constitute the views of ALTEC S.p.A.
Nothing in this email shall bind ALTEC S.p.A. in any contract or obligation.
If you received this email/facsimile through an error, please destroy your
copy and telephone or email/fax us immediately. Thank you.

Re: CAS pge-configuration - stage file on batch stub without shared filesystem

[Lewis John Mcgibbney <le...@gmail.com>]

Hi Andrea,
Just picked this up from my inbox.
Did you find a solution?
If not then we can try to help.
Thanks

On Mon, Oct 5, 2015 at 9:28 AM, Andrea Fonti <an...@altecspace.it>
wrote:

> Hi,
>
>
>
> I would like to implement the following scenario (all components are
> virtually separated host, implemented as a linked docker containers
> topology):
>
>
>
> 1.       [PUSHPULL] - The push pull component download files to the staging
> area
>
> 2.       [CRAWLER] - The crawler component extracts metadata and triggers
> filemgr ingestion
>
> 3.       [FILEMGR] - Filemgr ingest metadata in solr and archives the
> ingested file
>
> 4.       [CRAWLER] - Sends a ProductType-Ingest event to WMGR
>
> 5.       [WMGR] - Receives the event that trigger a data processing
> workflow: the ingested file should be processed and a new data product
> should be generated and ingested
>
> 6.       [RMGR] - Handles the request to schedule the processing task and
> dispatches it to the batch stub
>
> 7.       [BATCHSTUB] - a configured batch stub performs the processing task
> over the ingested file and produces the new data product
>
>
>
> I already implemented 1,2,3,4,5,6 but  I'm stuck with a problem:
>
>
>
> How can I instruct the batchstub to query the filemgr, initiate a data
> transfer to stage the source data product in a directory on the batch stub
> host, perform the processing and then stage back the new product? I would
> like to not rely on a shared filesystem.
>
>
>
> Any help would be really appreciated.
>
>
>
> Best regards
>
> Andrea Fonti
>
> ---- ALTEC S.p.A. ----
>
> COMUNICAZIONE IMPORTANTE
> La presente email è destinata solo al/i destinatario/i sopra indicato/i.
> Può riguardare un messaggio confidenziale e contenere informazioni e dati
> di
> natura RISERVATA O CLASSIFICATA. Non è consentita la sua lettura indebita,
> la sua duplicazione o la sua divulgazione. Le opinioni dell'autore di
> questa
> email non sono  necessariamente le opinioni di ALTEC S.p.A.
> Nulla di quanto contenuto in questa email può sostituire un contratto
> ovvero una obbligazione vincolante per ALTEC S.p.A.
> Nel caso doveste ricevere la presente email per errore, siete
> cortesemente pregati di distruggere la Vostra copia e di contattarci
> telefonicamente o a mezzo email o facsimile. Grazie.
>
> IMPORTANT NOTICE
> This email is intended only for the use of the above mentioned
> addressee(s).
> It is confidential and may contain legally PRIVILEGED OR CLASSIFIED
> information and data. Unauthorized reading, copying, disclosure or use is
> strictly prohibited. The views of the author hereof may not necessarily
> constitute the views of ALTEC S.p.A.
> Nothing in this email shall bind ALTEC S.p.A. in any contract or
> obligation.
> If you received this email/facsimile through an error, please destroy your
> copy and telephone or email/fax us immediately. Thank you.
>



-- 
*Lewis*