You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "jvenant (via GitHub)" <gi...@apache.org> on 2023/05/06 12:45:14 UTC
[GitHub] [pinot] jvenant opened a new pull request, #10731: set reader spec config and add test
jvenant opened a new pull request, #10731:
URL: https://github.com/apache/pinot/pull/10731
Setup the config of the recordReaderSpec
tracked by : #10727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a diff in pull request #10731: set reader spec config and add test
Posted by "snleee (via GitHub)" <gi...@apache.org>.
snleee commented on code in PR #10731:
URL: https://github.com/apache/pinot/pull/10731#discussion_r1187974822
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskExecutor.java:
##########
@@ -287,6 +287,9 @@ protected SegmentGenerationTaskSpec generateTaskSpec(Map<String, String> taskCon
recordReaderSpec.setDataFormat(taskConfigs.get(BatchConfigProperties.INPUT_FORMAT));
recordReaderSpec.setClassName(taskConfigs.get(BatchConfigProperties.RECORD_READER_CLASS));
recordReaderSpec.setConfigClassName(taskConfigs.get(BatchConfigProperties.RECORD_READER_CONFIG_CLASS));
+ recordReaderSpec.setConfigs(
Review Comment:
Have we set up the code style? If not, let's set up the pinot style in IDE and apply the code style change.
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskGeneratorTest.java:
##########
@@ -78,4 +86,52 @@ public void testRealCluster()
ControllerTest.sendPostRequest(_controllerRequestURLBuilder.forClusterConfigs(), request);
Assert.assertEquals(_generator.getNumConcurrentTasksPerInstance(), 1);
}
+
+ @Test
+ public void testGenerateTaskSpec() throws Exception {
+ URL resourcesLoc = SegmentGenerationAndPushTaskGeneratorTest.class.getClassLoader().getResource(".");
+ SegmentGenerationAndPushTaskExecutor executor = new SegmentGenerationAndPushTaskExecutor();
+ Schema schema = new Schema.SchemaBuilder().build();
+ // FieldUtils.writeField(executor, "_pinotTaskConfig", new PinotTaskConfig("",
Review Comment:
(nit) Can we remove this line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jvenant commented on a diff in pull request #10731: set reader spec config and add test
Posted by "jvenant (via GitHub)" <gi...@apache.org>.
jvenant commented on code in PR #10731:
URL: https://github.com/apache/pinot/pull/10731#discussion_r1189460774
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskExecutor.java:
##########
@@ -287,6 +287,9 @@ protected SegmentGenerationTaskSpec generateTaskSpec(Map<String, String> taskCon
recordReaderSpec.setDataFormat(taskConfigs.get(BatchConfigProperties.INPUT_FORMAT));
recordReaderSpec.setClassName(taskConfigs.get(BatchConfigProperties.RECORD_READER_CLASS));
recordReaderSpec.setConfigClassName(taskConfigs.get(BatchConfigProperties.RECORD_READER_CONFIG_CLASS));
+ recordReaderSpec.setConfigs(
Review Comment:
Fixed in bb90840fffa9e1f3d2f03af2f8270127bfe91787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a diff in pull request #10731: set reader spec config and add test
Posted by "snleee (via GitHub)" <gi...@apache.org>.
snleee commented on code in PR #10731:
URL: https://github.com/apache/pinot/pull/10731#discussion_r1187974822
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskExecutor.java:
##########
@@ -287,6 +287,9 @@ protected SegmentGenerationTaskSpec generateTaskSpec(Map<String, String> taskCon
recordReaderSpec.setDataFormat(taskConfigs.get(BatchConfigProperties.INPUT_FORMAT));
recordReaderSpec.setClassName(taskConfigs.get(BatchConfigProperties.RECORD_READER_CLASS));
recordReaderSpec.setConfigClassName(taskConfigs.get(BatchConfigProperties.RECORD_READER_CONFIG_CLASS));
+ recordReaderSpec.setConfigs(
Review Comment:
Have we set up the code style? If not, let's set up the pinot style in IDE and apply the code style change.
https://docs.pinot.apache.org/developers/developers-and-contributors/code-setup#setup-ide
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #10731: set reader spec config and add test
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10731:
URL: https://github.com/apache/pinot/pull/10731#issuecomment-1539198258
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10731?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10731](https://app.codecov.io/gh/apache/pinot/pull/10731?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (80647eb) into [master](https://app.codecov.io/gh/apache/pinot/commit/c47e7e55ce1713a0e63e1eb75c26ca9e23192f3f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c47e7e5) will **decrease** coverage by `54.66%`.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## master #10731 +/- ##
=============================================
- Coverage 68.53% 13.87% -54.66%
+ Complexity 6452 439 -6013
=============================================
Files 2122 2068 -54
Lines 114285 111800 -2485
Branches 17253 16956 -297
=============================================
- Hits 78322 15512 -62810
- Misses 30401 95021 +64620
+ Partials 5562 1267 -4295
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| unittests1 | `?` | |
| unittests2 | `13.87% <100.00%> (+0.04%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://app.codecov.io/gh/apache/pinot/pull/10731?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...nandpush/SegmentGenerationAndPushTaskExecutor.java](https://app.codecov.io/gh/apache/pinot/pull/10731?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3Mvc2VnbWVudGdlbmVyYXRpb25hbmRwdXNoL1NlZ21lbnRHZW5lcmF0aW9uQW5kUHVzaFRhc2tFeGVjdXRvci5qYXZh) | `26.95% <100.00%> (-41.40%)` | :arrow_down: |
... and [1636 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/10731/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jvenant commented on pull request #10731: set reader spec config and add test
Posted by "jvenant (via GitHub)" <gi...@apache.org>.
jvenant commented on PR #10731:
URL: https://github.com/apache/pinot/pull/10731#issuecomment-1702681303
Any feedback on this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jvenant commented on a diff in pull request #10731: set reader spec config and add test
Posted by "jvenant (via GitHub)" <gi...@apache.org>.
jvenant commented on code in PR #10731:
URL: https://github.com/apache/pinot/pull/10731#discussion_r1189460970
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskGeneratorTest.java:
##########
@@ -78,4 +86,52 @@ public void testRealCluster()
ControllerTest.sendPostRequest(_controllerRequestURLBuilder.forClusterConfigs(), request);
Assert.assertEquals(_generator.getNumConcurrentTasksPerInstance(), 1);
}
+
+ @Test
+ public void testGenerateTaskSpec() throws Exception {
+ URL resourcesLoc = SegmentGenerationAndPushTaskGeneratorTest.class.getClassLoader().getResource(".");
+ SegmentGenerationAndPushTaskExecutor executor = new SegmentGenerationAndPushTaskExecutor();
+ Schema schema = new Schema.SchemaBuilder().build();
+ // FieldUtils.writeField(executor, "_pinotTaskConfig", new PinotTaskConfig("",
Review Comment:
Fixed in bb90840fffa9e1f3d2f03af2f8270127bfe91787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #10731: set reader spec config and add test
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #10731:
URL: https://github.com/apache/pinot/pull/10731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org