You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/06/24 08:11:20 UTC
[GitHub] [incubator-pinot] MrNeocore opened a new issue #7090: SegmentNameGenerator: Extend interface to accept input file name
MrNeocore opened a new issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090
Following Slack discussion: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1624443889243500
**Use case:**
Some tables in Pinot are used in conjunction with [IdSet filtering](https://docs.google.com/document/d/1s6DZ9eTPqH7vaKQlPjKiWb_OBC3hkkEGICIzcd5gozc/edit#) or [Lookups](https://docs.pinot.apache.org/users/user-guide-query/lookup-udf-join) - and in some cases don't have a time column.
- Hence time-based segment name generation strategies do not allow for simple segment replacements whenever data in those "dimension" tables change.
**Proposition**
Allow segment name generation to be based on the input file names such that segments can be named following a user provided id.
E.g.
```
basedir/id1/file.parquet
basedir/id2/file.parquet
```
Would generate segments
```
<table_name>_id1.segment
<table_name>_id2.segment
```
Currently, the [SegmentNameGenerator interface](https://github.com/apache/incubator-pinot/blob/7ce8b756dff2b153151517afd552759e44d46c75/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/name/SegmentNameGenerator.java) doesn't allow input file names, therefore it is not possible to implement a strategy similar to the one presented above.
**Note**: If you known any other alternative to reach the use case goal, please feel free to provide ideas !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kkrugler commented on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-874977841
Also currently is it possible to implement a custom name generator, via java classes in some plugin dir? I guess we could support patterns to be able to define the resulting segment name based on the input file path, but that seems like it could get messy. Wondering if a first cut to just enable this, and ensure you can implement your own name generator class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kkrugler commented on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-877830221
I've dug into this a bit, and started a first pass. The problem is that the distinction between real-time (from Kafka) and batch (from input files) is erased early on, as is right and proper :) But it means that where segment generation is occurring, information about the input file path/name has been lost. So the changes are more extensive than I was expecting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kkrugler commented on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-874989183
How about being able to define things via:
``` json
"filePathPattern":".+/(.+)/.+\\.parquet",
"segmentName":"${tableName}_${filePathPattern:\\1}.parquet"
```
or, for our use case:
``` json
"filePathPattern":".+/(.+)\\.csv",
"segmentName":"${filePathPattern:\\1}"
```
We could also support formatting of the input dates via say `${minTimeValue:yyyy-MM}`, but maybe that's a bridge too far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv commented on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-874981243
> Also currently is it possible to implement a custom name generator, via java classes in some plugin dir? I guess we could support patterns to be able to define the resulting segment name based on the input file path, but that seems like it could get messy. Wondering if a first cut to just enable this, and ensure you can implement your own name generator class.
IIRC, segment name generator is not pluggable, atm. I can see that plugging in custom name generators would be useful. It might open up the door for operational overhead though (given that we rely on names to overwrite segments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] MrNeocore commented on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
MrNeocore commented on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-877809869
@kkrugler As a user, this idea seems interesting and pretty flexible :) Then, as you suggested, it would seem possible to add more advanced parsing to extend that feature further down the line.
FYI, the way I 'solved' this for now is by using the `fixed` strategy in conjunction with custom job spec templating.
But this adds business-level complexity and implies to run an ingestion job for every single segment, leading to slower ingestion in case of many segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] MrNeocore commented on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
MrNeocore commented on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-877863918
Ah that makes sense.. thanks for looking into it, let's hope there's still a nice way to do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kkrugler edited a comment on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
kkrugler edited a comment on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-874975957
Java 8 is the minimum version, right? So we could extend the `SegmentNameGenerator` interface with a default method that takes the input file path, and just calls the old version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] troywinter commented on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
troywinter commented on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-878047071
May I ask anyone is working on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv commented on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-874979849
> Java 8 is the minimum version, right? So we could extend the `SegmentNameGenerator` interface with a default method that takes the input file path, and just calls the old version.
Yes, Java8 is the minimum version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kkrugler commented on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-874975957
Java 8 is the minimum version, right? So we could extend the `SegmentNameGenerator` interface with a default method that takes the input filename, and just calls the old version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] MrNeocore edited a comment on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
MrNeocore edited a comment on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-877809869
@kkrugler As a user, this idea seems interesting and pretty flexible :) Then, as you suggested, it would seem possible to add more advanced parsing to extend that feature further down the line.
FYI, the way I 'solved' this for now is by using the `fixed` strategy in conjunction with custom job spec templating.
But this adds business-level complexity and implies to run an ingestion job for every single segment, leading to slower ingestion in case of many segments.
(Idea suggested by @Jackie-Jiang)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kkrugler commented on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-874983795
And pluggable name generators would make it harder to provide a hosted solution :) But without that, I'd need to also add support to format the input dates to be `YYYY-MM`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kkrugler commented on issue #7090: SegmentNameGenerator: Extend interface to accept input file name
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7090:
URL: https://github.com/apache/incubator-pinot/issues/7090#issuecomment-878261884
@troywinter - Yes, I've started some work (see above for my comments on why it's a bit harder than I was expecting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org