You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/10/07 04:54:13 UTC

[GitHub] [incubator-pinot] fx19880617 opened a new issue #6115: Support environment variables injection for Pinot ingestion job

fx19880617 opened a new issue #6115:
URL: https://github.com/apache/incubator-pinot/issues/6115


   Pinot supports launch ingestion jobs with a template file and a command line option to set the values.
   
   This logic is handled in class: `org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher`.
   
   Here we want to also support environment variables to set those values.
   
   This will be convenient for users in k8s environment with dynamic configs settings.
   
   E.g. user can do dynamic job loading with k8s cronjob by setting:
   
   - Set `inputDirURI` to `'s3://event-logs/offlinepinot/dt=${JOB_DATE}/'` in `batch_job_spec.yaml` file.
   - Set container  `env` with `JOB_DATE` to `$(date -u +"%Y-%m-%d")`, so `JOB_DATE` will be the runtime date.
   - Set container `args` to
   ```
   [
       "LaunchDataIngestionJob",
       "-jobSpecFile",
       "/home/pinot/pinot-config/local_batch_job_spec.yaml"
   ]
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] vgtom commented on issue #6115: Support environment variables injection for Pinot ingestion job

Posted by GitBox <gi...@apache.org>.
vgtom commented on issue #6115:
URL: https://github.com/apache/incubator-pinot/issues/6115#issuecomment-707837637


   hi...i would like to work on this issue(although my response maybe slow because i have a FullTime job)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] JoaoComini commented on issue #6115: Support environment variables injection for Pinot ingestion job

Posted by GitBox <gi...@apache.org>.
JoaoComini commented on issue #6115:
URL: https://github.com/apache/incubator-pinot/issues/6115#issuecomment-831586016


   Hi folks! 😄
   Given that we already have Groovy templating working on `IngestionJobLauncher`, what would be the priority when there's a
   `-values` parameter and an enviroment variable with the same name?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org