You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/05/06 18:53:34 UTC
[GitHub] [incubator-pinot] mangrrua opened a new issue #5344: Create segment file from in-memory data
mangrrua opened a new issue #5344:
URL: https://github.com/apache/incubator-pinot/issues/5344
Apache Pinot have a great abstraction layer to convert any file format to pinot segment file. But it is not useful when we want to convert in-memory data(eg: InputStream, lists etc.) to the pinot segment.
`RecordReader` waits input file path for segment creating, and `SegmentIndexCreationDriverImpl` takes RecordReader implementation. Then it read input file and create Pinot segment file. But implementation is based on input File. This is a bad limitation. I want to create Pinot segment file from in memory data.
We can expand the `RecordReader` and `SegmentIndexCreatingDriverImpl` implementations? Or how can we do anything for that? I want that, because I do not want to write any data to the temp files while converting it to pinot segment file. This is a costly operation. I want to convert data to pinot segment from in-memory data directly.
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua edited a comment on issue #5344: Create segment file from in-memory data
Posted by GitBox <gi...@apache.org>.
mangrrua edited a comment on issue #5344:
URL: https://github.com/apache/incubator-pinot/issues/5344#issuecomment-625050832
The above example looks great. That's API so useful for any input format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #5344: Create segment file from in-memory data
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #5344:
URL: https://github.com/apache/incubator-pinot/issues/5344#issuecomment-624847416
We want to have a simple API such as
```
SegmentWriter writer = ...
writer.add(GenericRow)
Segment writer.seal()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #5344: Create segment file from in-memory data
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #5344:
URL: https://github.com/apache/incubator-pinot/issues/5344#issuecomment-624846844
You are right, we have some plans on changing this.
@Jackie-Jiang @elonazoulay Alex had some thoughts on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua commented on issue #5344: Create segment file from in-memory data
Posted by GitBox <gi...@apache.org>.
mangrrua commented on issue #5344:
URL: https://github.com/apache/incubator-pinot/issues/5344#issuecomment-625050832
The above example is look great. That's API so useful for any input format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua edited a comment on issue #5344: Create segment file from in-memory data
Posted by GitBox <gi...@apache.org>.
mangrrua edited a comment on issue #5344:
URL: https://github.com/apache/incubator-pinot/issues/5344#issuecomment-625050832
The above example is looking great. That's API so useful for any input format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua closed issue #5344: Create segment file from in-memory data
Posted by GitBox <gi...@apache.org>.
mangrrua closed issue #5344:
URL: https://github.com/apache/incubator-pinot/issues/5344
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua commented on issue #5344: Create segment file from in-memory data
Posted by GitBox <gi...@apache.org>.
mangrrua commented on issue #5344:
URL: https://github.com/apache/incubator-pinot/issues/5344#issuecomment-628728018
Reopen issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua removed a comment on issue #5344: Create segment file from in-memory data
Posted by GitBox <gi...@apache.org>.
mangrrua removed a comment on issue #5344:
URL: https://github.com/apache/incubator-pinot/issues/5344#issuecomment-628728018
Reopen issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org