You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/14 13:09:39 UTC
[GitHub] [pinot] lksvenoy-r7 opened a new issue, #8889: The Pinot Flink connector does not gracefully handle failure
lksvenoy-r7 opened a new issue, #8889:
URL: https://github.com/apache/pinot/issues/8889
The current pinot flink connector does not gracefully handle errors. Due to the way the connector works, if it errors in the middle of adding segments to a table, the table ends up with an inconsistent view. Additionally, the connector does not currently support refresh tables. Refresh tables require atomic segment replacement, but the connector currently naively uploads segments as they are built.
From testing the connector in production, I've also identified a few performance issues. These have a few different causes; The AVRO serialization is not configurable, nor is the file writing configurable (for example for different block sizes).
I have written a flink connector based on this one, but with some heavy amendments. First of all, it implements WithPostCommitTopology<GenericRecord, PinotSinkCommittable> from flink, implementing a global committer. It does work in a few different stages:
1. Operator is responsible for sending serialized AVRO records directly to the sink
2. The sink writer is responsible for building and flushing segments to disk
3. The sink committer (before global commit) is responsible for uploading the segments to a location that is reachable by all nodes in the flink cluster (In my case, to S3 deep store)
4. The global sink committer executes the segment replacement protocol defined in the Pinot SDK.
This sink is currently only compatible with REFRESH type tables that want to replace all segments on every single job execution. It takes care of atomically replacing the segments for the table, and performs well due to the way it does the hard work upfront. I am open to sharing this code so that it can be merged into the pinot repository, but it does have some limitations.
- No checkpointing
- Only BATCH execution mode is supported at the moment
- Only REFRESH tables are supported at the moment (Full segment replacement)
- The connector currently bypasses certain Pinot conventions (such as using certain attributes defined in the batch config, and so on). This would need to be approached with scrutiny to ensure the code is in-line with the rest of the repository.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on issue #8889: The Pinot Flink connector does not gracefully handle failure
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #8889:
URL: https://github.com/apache/pinot/issues/8889#issuecomment-1155796503
Hi @lksvenoy-r7 - thanks for opening this issue, sounds very promising! A few comments & questions...
1. I think not supporting checkpointing is OK.
2. When you say only BATCH execution mode, are you talking about DataSets or bounded streams?
3. There's been quite a lot of discussion around how to build segments that are properly partitioned (matching Flink's hashing function) so that the resulting table can be configured as partitioned. Would your version work with a bucketing policy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] lksvenoy-r7 commented on issue #8889: The Pinot Flink connector does not gracefully handle failure
Posted by GitBox <gi...@apache.org>.
lksvenoy-r7 commented on issue #8889:
URL: https://github.com/apache/pinot/issues/8889#issuecomment-1160308891
Some additional information regarding segment creation:
I was speaking with @mayankshriv regarding extending the segment index creation driver to allow in-memory segment creation, but unfortunately the interface implements File getOutputDirectory(), meaning we are bound to using a file.
One approach which could work is to abstract away File, and use the Pinot filesystem to allow the output to be elsewhere.
A general improvement to the architecture would be to concretely return types which have to pass through PinotFS. As it stands, since PinotFS takes a generic java URI, there's no way of knowing if a specific URI has to pass through this pattern. A good inspiration for this would be to look at the way flink handles abstracting away the file system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] lksvenoy-r7 commented on issue #8889: The Pinot Flink connector does not gracefully handle failure
Posted by GitBox <gi...@apache.org>.
lksvenoy-r7 commented on issue #8889:
URL: https://github.com/apache/pinot/issues/8889#issuecomment-1160307175
Hello @kkrugler
1. 👍🏻 This might be OK for an initial version
2. I'm talking about bounded streams
3. Yes. My job is configured to allow you to add multiple sinks per partition. This allows you to create multiple buckets to scale ingestion for certain replicas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org