You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/05/06 11:01:07 UTC
[GitHub] [incubator-pinot] fx19880617 opened a new pull request #5341: Adding template support for Pinot Ingestion Job Spec
fx19880617 opened a new pull request #5341:
URL: https://github.com/apache/incubator-pinot/pull/5341
Per https://github.com/apache/incubator-pinot/issues/5330, we need to support Pinot Ingestion job take an ingestion job spec template file and passing thru values.
Templating is based on [jinjava](https://github.com/HubSpot/jinjava) syntax.
- Adding jinjava template render support for Pinot.
- Update pinot ingestion launcher command to take a list of parameters which will be the context to set in the template.
- The template ingestion spec file will be rendered to final ingestion spec with the given context values.
- Adding support for default values `today`, `yesterday` into context for dev simplicity.
Usage example:
user can define `inputDirURI` as `'file:///path/to/input/{{year}}/{{month}}/{{day}}'`
during job launcher, the new command line is :
```
bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile ingestionJobSpec.yaml -values year=2020 month=05 day=06
```
After that the real ingestion spec passed to ingestion job will have `inputDirURI` as `'file:///path/to/input/2020/05/06'`
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io commented on pull request #5341: Adding template support for Pinot Ingestion Job Spec
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #5341:
URL: https://github.com/apache/incubator-pinot/pull/5341#issuecomment-626324823
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5341?src=pr&el=h1) Report
> Merging [#5341](https://codecov.io/gh/apache/incubator-pinot/pull/5341?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/2bda544f967e405a6aac20d2bb2abce430da9603&el=desc) will **increase** coverage by `0.39%`.
> The diff coverage is `82.02%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/5341?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #5341 +/- ##
==========================================
+ Coverage 66.08% 66.47% +0.39%
==========================================
Files 1072 1076 +4
Lines 54668 54799 +131
Branches 8152 8170 +18
==========================================
+ Hits 36125 36427 +302
+ Misses 15895 15702 -193
- Partials 2648 2670 +22
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5341?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/pinot/common/utils/SchemaUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/5341/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvU2NoZW1hVXRpbHMuamF2YQ==) | `9.85% <0.00%> (ø)` | |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/incubator-pinot/pull/5341/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `71.42% <ø> (ø)` | |
| [...edicate/BaseDictionaryBasedPredicateEvaluator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5341/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvcHJlZGljYXRlL0Jhc2VEaWN0aW9uYXJ5QmFzZWRQcmVkaWNhdGVFdmFsdWF0b3IuamF2YQ==) | `54.16% <ø> (ø)` | |
| [...predicate/BaseRawValueBasedPredicateEvaluator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5341/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvcHJlZGljYXRlL0Jhc2VSYXdWYWx1ZUJhc2VkUHJlZGljYXRlRXZhbHVhdG9yLmphdmE=) | `87.87% <ø> (ø)` | |
| [...e/operator/dociditerators/MVScanDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5341/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9NVlNjYW5Eb2NJZEl0ZXJhdG9yLmphdmE=) | `62.50% <14.28%> (-3.61%)` | :arrow_down: |
| [...inot/spi/ingestion/batch/IngestionJobLauncher.java](https://codecov.io/gh/apache/incubator-pinot/pull/5341/diff?src=pr&el=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvaW5nZXN0aW9uL2JhdGNoL0luZ2VzdGlvbkpvYkxhdW5jaGVyLmphdmE=) | `8.51% <27.27%> (+8.51%)` | :arrow_up: |
| [...rg/apache/pinot/broker/routing/RoutingManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5341/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9Sb3V0aW5nTWFuYWdlci5qYXZh) | `78.62% <50.00%> (-2.53%)` | :arrow_down: |
| [...indexsegment/generator/SegmentGeneratorConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/5341/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9pbmRleHNlZ21lbnQvZ2VuZXJhdG9yL1NlZ21lbnRHZW5lcmF0b3JDb25maWcuamF2YQ==) | `72.44% <50.00%> (+10.18%)` | :arrow_up: |
| [...main/java/org/apache/pinot/spi/data/FieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/5341/diff?src=pr&el=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9GaWVsZFNwZWMuamF2YQ==) | `86.25% <50.00%> (-1.25%)` | :arrow_down: |
| [...e/operator/dociditerators/SVScanDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5341/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9TVlNjYW5Eb2NJZEl0ZXJhdG9yLmphdmE=) | `68.96% <60.00%> (-0.55%)` | :arrow_down: |
| ... and [98 more](https://codecov.io/gh/apache/incubator-pinot/pull/5341/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5341?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5341?src=pr&el=footer). Last update [44b0a91...48033d1](https://codecov.io/gh/apache/incubator-pinot/pull/5341?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org