You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "afterincomparableyum (via GitHub)" <gi...@apache.org> on 2026/01/11 18:28:47 UTC
[PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
afterincomparableyum opened a new pull request, #3583:
URL: https://github.com/apache/celeborn/pull/3583
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
This PR implements retry support for createReader failures in the C++ client, matching the behavior of the Java implementation. The implementation includes:
- Added configuration properties:
* clientFetchMaxRetriesForEachReplica (default: 3)
* dataIoRetryWait (default: 5s)
* clientPushReplicateEnabled (default: false)
- Added peer location support methods to PartitionLocation:
* hasPeer() - Check if location has a peer replica
* getPeer() - Get the peer location
* hostAndFetchPort() - Get host:port string for logging
- Implemented retry logic in createReaderWithRetry():
* Retries up to fetchChunkMaxRetry_ times (doubled if replication enabled)[which is why I added this parameter in this PR]
* Switches to peer location on failure when available
* Sleeps between retries when both replicas tried or no peer exists
* Resets retry counter when moving to new location or on success
- Added unit tests for new functionality
### How was this patch tested?
Unit tests and compiling
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3910429377
ping @SteNicholas for review, I have resolved conflicts as well as addressed comments from CoPilot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2786044297
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +196,45 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ auto reader = createReader(*currentLocation);
+ fetchChunkRetryCnt_ = 0;
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ fetchChunkRetryCnt_++;
+
+ if (currentLocation->hasPeer()) {
+ if (fetchChunkRetryCnt_ % 2 == 0) {
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(retryWait_.count()));
+ }
+ LOG(WARNING) << "CreatePartitionReader failed " << fetchChunkRetryCnt_
Review Comment:
looking into this....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3949132332
I've addressed the comments of CoPilot @SteNicholas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2785999168
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +196,45 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ auto reader = createReader(*currentLocation);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2786045305
##########
cpp/celeborn/conf/CelebornConf.h:
##########
@@ -176,6 +185,12 @@ class CelebornConf : public BaseConf {
protocol::CompressionCodec shuffleCompressionCodec() const;
int shuffleCompressionZstdCompressLevel() const;
+
+ int clientFetchMaxRetriesForEachReplica() const;
+
+ Timeout dataIoRetryWait() const;
Review Comment:
done
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +196,45 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ auto reader = createReader(*currentLocation);
+ fetchChunkRetryCnt_ = 0;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2813488475
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +205,58 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ if (isExcluded(*currentLocation)) {
+ throw std::runtime_error(
+ "Fetch data from excluded worker! " +
+ currentLocation->hostAndFetchPort());
+ }
+ auto reader = createReader(*currentLocation);
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ excludeFailedFetchLocation(currentLocation->hostAndFetchPort(), e);
+ fetchChunkRetryCnt_++;
+
+ if (currentLocation->hasPeer() && !readSkewPartitionWithoutMapRange_) {
+ if (fetchChunkRetryCnt_ % 2 == 0) {
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(retryWait_.count()));
+ }
Review Comment:
This is a good observation, but it is not necessarily a bug. The C++ behavior here matches the Java behavior exactly. The peer relationship is one-directional by design. In Java, `setPeer` is only ever called on the primary, meaning java has the same one directional relationship.
@SteNicholas what do you think? Right now, it matches the Java behavior, meaning it will try it on the primary, switch to replica on failure, and retry on replica for remaining attempts. Should we change this behavior or keep it as is?
I say keep it as is. I am not worried about this comment by Copilot AI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2835744417
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +205,56 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ if (isExcluded(*currentLocation)) {
+ CELEBORN_FAIL(
+ "Fetch data from excluded worker! {}",
+ currentLocation->hostAndFetchPort());
+ }
+ auto reader = createReader(*currentLocation);
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ excludeFailedFetchLocation(currentLocation->hostAndFetchPort(), e);
+ fetchChunkRetryCnt_++;
+
+ if (currentLocation->hasPeer() && !readSkewPartitionWithoutMapRange_) {
+ if (fetchChunkRetryCnt_ % 2 == 0) {
+ std::this_thread::sleep_for(retryWait_);
+ }
+ LOG(WARNING) << "CreatePartitionReader failed " << fetchChunkRetryCnt_
+ << "/" << fetchChunkMaxRetry_ << " times for location "
+ << currentLocation->hostAndFetchPort()
+ << ", change to peer. Error: " << e.what();
+ // TODO: When stream handlers are supported, send BUFFER_STREAM_END
+ // to close the active stream before switching to peer, matching the
+ // Java CelebornInputStream behavior. Currently, the C++ client does
+ // not have pre-opened stream handlers, so stream cleanup is handled
+ // by WorkerPartitionReader's destructor.
+ currentLocation = currentLocation->getPeer();
+ } else {
+ LOG(WARNING) << "CreatePartitionReader failed " << fetchChunkRetryCnt_
+ << "/" << fetchChunkMaxRetry_ << " times for location "
+ << currentLocation->hostAndFetchPort()
+ << ", retry the same location. Error: " << e.what();
+ std::this_thread::sleep_for(retryWait_);
+ }
+ }
+ }
+
+ // Max retries exceeded, rethrow the last exception wrapped with context
+ throw utils::CelebornRuntimeError(
+ lastException,
+ "createPartitionReader failed after " +
+ std::to_string(fetchChunkRetryCnt_) + " retries for location " +
+ location.hostAndFetchPort(),
+ false);
Review Comment:
The final wrapped exception message always reports the original `location.hostAndFetchPort()`, even if the last failure happened after switching to the peer. This can be misleading when debugging. Consider including the last attempted location (e.g., currentLocation->hostAndFetchPort()) or both original+current in the error context.
```suggestion
std::string errorMessage =
"createPartitionReader failed after " +
std::to_string(fetchChunkRetryCnt_) +
" retries. Original location: " + location.hostAndFetchPort() +
", last attempted location: " + currentLocation->hostAndFetchPort();
throw utils::CelebornRuntimeError(lastException, errorMessage, false);
```
##########
cpp/celeborn/client/ShuffleClient.h:
##########
@@ -70,6 +72,10 @@ class ShuffleClient {
int endMapIndex,
bool needCompression) = 0;
+ virtual void excludeFailedFetchLocation(
+ const std::string& hostAndFetchPort,
+ const std::exception& e) = 0;
+
Review Comment:
The new ShuffleClient::excludeFailedFetchLocation API (and its implementation in ShuffleClientImpl) appears unused in the current codebase (CelebornInputStream maintains exclusion internally instead). If this isn’t intended for external callers, consider removing it to avoid expanding the public interface; otherwise, consider wiring CelebornInputStream to call through ShuffleClient so there is a single source of truth for exclusion behavior.
```suggestion
```
##########
cpp/celeborn/conf/CelebornConf.h:
##########
@@ -129,6 +129,25 @@ class CelebornConf : public BaseConf {
static constexpr std::string_view kShuffleCompressionZstdCompressLevel{
"celeborn.client.shuffle.compression.zstd.level"};
+ static constexpr std::string_view kClientFetchMaxRetriesForEachReplica{
+ "celeborn.client.fetch.maxRetriesForEachReplica"};
+
+ static constexpr std::string_view kNetworkIoRetryWait{
+ "celeborn.data.io.retryWait"};
+
+ static constexpr std::string_view kClientPushReplicateEnabled{
+ "celeborn.client.push.replicate.enabled"};
+
+ static constexpr std::string_view kClientFetchExcludeWorkerOnFailureEnabled{
+ "celeborn.client.fetch.excludeWorkerOnFailure.enabled"};
+
+ static constexpr std::string_view kClientFetchExcludedWorkerExpireTimeout{
+ "celeborn.client.fetch.excludedWorker.expireTimeout"};
+
+ static constexpr std::string_view
+ kClientAdaptiveOptimizeSkewedPartitionReadEnabled{
+ "celeborn.client.adaptive.optimizeSkewedPartitionRead.enabled"};
Review Comment:
PR description lists 3 new config properties, but this change also introduces additional fetch-related configs (excludeWorkerOnFailure, excludedWorker expireTimeout, optimizeSkewedPartitionRead). Please update the PR description (or drop these configs if out of scope) so reviewers/users can understand the full config surface change.
##########
cpp/celeborn/client/ShuffleClient.h:
##########
Review Comment:
ShuffleClient.h uses utils::ConcurrentHashMap in several type aliases/member declarations, but it does not include the header that defines it (celeborn/utils/CelebornUtils.h). It currently compiles only because CelebornInputStream.h happens to include CelebornUtils.h; add the direct include here to avoid fragile transitive dependencies.
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
Review Comment:
CelebornInputStream.cpp uses std::this_thread::sleep_for(retryWait_) later in the file but does not include <thread> (and the header no longer provides it). This can fail to compile depending on transitive includes; add an explicit #include <thread> in this .cpp.
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +205,56 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ if (isExcluded(*currentLocation)) {
+ CELEBORN_FAIL(
+ "Fetch data from excluded worker! {}",
+ currentLocation->hostAndFetchPort());
+ }
Review Comment:
When a location is already in the exclusion list, the code throws and then retries/sleeps (especially in the no-peer branch). Since isExcluded(*currentLocation) will keep returning true until the exclusion expires, these retries are guaranteed to fail and just add delay. Consider failing fast (no sleep/retry) or skipping to another available location/peer when the current location is excluded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2813973306
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +205,58 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
Review Comment:
added target tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2813496196
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +205,58 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ if (isExcluded(*currentLocation)) {
+ throw std::runtime_error(
+ "Fetch data from excluded worker! " +
+ currentLocation->hostAndFetchPort());
+ }
+ auto reader = createReader(*currentLocation);
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ excludeFailedFetchLocation(currentLocation->hostAndFetchPort(), e);
+ fetchChunkRetryCnt_++;
+
+ if (currentLocation->hasPeer() && !readSkewPartitionWithoutMapRange_) {
+ if (fetchChunkRetryCnt_ % 2 == 0) {
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(retryWait_.count()));
+ }
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2706583335
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +196,45 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ auto reader = createReader(*currentLocation);
+ fetchChunkRetryCnt_ = 0;
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ fetchChunkRetryCnt_++;
Review Comment:
The shuffle client should exclude failed fetch location.
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +196,45 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ auto reader = createReader(*currentLocation);
Review Comment:
This should check whether the partition location is excluded, which aligns with the logic of `CelebornInputStream#createReaderWithRetry`.
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +196,45 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ auto reader = createReader(*currentLocation);
+ fetchChunkRetryCnt_ = 0;
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ fetchChunkRetryCnt_++;
+
+ if (currentLocation->hasPeer()) {
+ if (fetchChunkRetryCnt_ % 2 == 0) {
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(retryWait_.count()));
+ }
+ LOG(WARNING) << "CreatePartitionReader failed " << fetchChunkRetryCnt_
Review Comment:
Does this aligin with the failure handling of `CelebornInputStream#createReaderWithRetry`?
##########
cpp/celeborn/conf/CelebornConf.h:
##########
@@ -176,6 +185,12 @@ class CelebornConf : public BaseConf {
protocol::CompressionCodec shuffleCompressionCodec() const;
int shuffleCompressionZstdCompressLevel() const;
+
+ int clientFetchMaxRetriesForEachReplica() const;
+
+ Timeout dataIoRetryWait() const;
Review Comment:
Could this method name align with `CelebornConf#networkIoRetryWaitMs` for io wait conf of all modules?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2813499579
##########
cpp/celeborn/client/reader/CelebornInputStream.h:
##########
@@ -17,14 +17,19 @@
#pragma once
+#include <thread>
+
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3868499040
Sorry for delay on this, will push a couple commits within next couple of days
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2844511387
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +205,56 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ if (isExcluded(*currentLocation)) {
+ CELEBORN_FAIL(
+ "Fetch data from excluded worker! {}",
+ currentLocation->hostAndFetchPort());
+ }
Review Comment:
I will leave a TODO comment for further investigation on this. for now, I just want it to match java client behavior. and if needed, I can add this optimization to both c++ and java client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3902304468
@SteNicholas I have addressed the comments, can you please take a look again. thank you for your review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2813506137
##########
cpp/celeborn/utils/CelebornUtils.cpp:
##########
@@ -19,6 +19,22 @@
namespace celeborn {
namespace utils {
+
+bool isCriticalCauseForFetch(const std::exception& e) {
+ if (dynamic_cast<const std::system_error*>(&e)) {
+ return true;
+ }
+
+ std::string msg = e.what();
+ if (msg.find("Connecting to") != std::string::npos ||
+ msg.find("Failed to") != std::string::npos ||
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2813521705
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +205,58 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
Review Comment:
good suggestion, I will add targeted tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2844458098
##########
cpp/celeborn/client/ShuffleClient.h:
##########
@@ -70,6 +72,10 @@ class ShuffleClient {
int endMapIndex,
bool needCompression) = 0;
+ virtual void excludeFailedFetchLocation(
+ const std::string& hostAndFetchPort,
+ const std::exception& e) = 0;
+
Review Comment:
it's likely best to match java client and use the ShuffleClient::excludeFailedFetchLocation API. Will refactor code accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2813512127
##########
cpp/celeborn/client/ShuffleClient.cpp:
##########
@@ -286,7 +290,19 @@ std::unique_ptr<CelebornInputStream> ShuffleClientImpl::readPartition(
attemptNumber,
startMapIndex,
endMapIndex,
- needCompression);
+ needCompression,
+ fetchExcludedWorkers_);
+}
+
+void ShuffleClientImpl::excludeFailedFetchLocation(
+ const std::string& hostAndFetchPort,
+ const std::exception& e) {
+ if (pushReplicateEnabled_ && fetchExcludeWorkerOnFailureEnabled_) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3770836920
## [Codecov](https://app.codecov.io/gh/apache/celeborn/pull/3583?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 67.04%. Comparing base ([`2dd1b7a`](https://app.codecov.io/gh/apache/celeborn/commit/2dd1b7aac4d488c0496da1fc77cab9c79bccd69b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5d32d94`](https://app.codecov.io/gh/apache/celeborn/commit/5d32d94425248738900df7545823cbc95b9f08a9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #3583 +/- ##
==========================================
- Coverage 67.13% 67.04% -0.09%
==========================================
Files 357 357
Lines 21860 21924 +64
Branches 1943 1949 +6
==========================================
+ Hits 14674 14696 +22
- Misses 6166 6213 +47
+ Partials 1020 1015 -5
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/celeborn/pull/3583?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3904864709
Ping @RexXiong @FMX @HolyLow for review. Thank you for your review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2786046406
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +196,45 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ auto reader = createReader(*currentLocation);
+ fetchChunkRetryCnt_ = 0;
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ fetchChunkRetryCnt_++;
+
+ if (currentLocation->hasPeer()) {
+ if (fetchChunkRetryCnt_ % 2 == 0) {
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(retryWait_.count()));
+ }
+ LOG(WARNING) << "CreatePartitionReader failed " << fetchChunkRetryCnt_
+ << "/" << fetchChunkMaxRetry_ << " times for location "
+ << currentLocation->hostAndFetchPort()
+ << ", change to peer. Error: " << e.what();
+ currentLocation = currentLocation->getPeer();
+ } else {
+ LOG(WARNING) << "CreatePartitionReader failed " << fetchChunkRetryCnt_
+ << "/" << fetchChunkMaxRetry_ << " times for location "
+ << currentLocation->hostAndFetchPort()
+ << ", retry the same location. Error: " << e.what();
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(retryWait_.count()));
+ }
+ }
+ }
+
+ CELEBORN_FAIL(
+ "createPartitionReader failed after " +
+ std::to_string(fetchChunkRetryCnt_) + " retries for location " +
+ location.hostAndFetchPort());
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2810957648
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +205,58 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ if (isExcluded(*currentLocation)) {
+ throw std::runtime_error(
+ "Fetch data from excluded worker! " +
+ currentLocation->hostAndFetchPort());
+ }
+ auto reader = createReader(*currentLocation);
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ excludeFailedFetchLocation(currentLocation->hostAndFetchPort(), e);
+ fetchChunkRetryCnt_++;
+
+ if (currentLocation->hasPeer() && !readSkewPartitionWithoutMapRange_) {
+ if (fetchChunkRetryCnt_ % 2 == 0) {
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(retryWait_.count()));
+ }
Review Comment:
`createReaderWithRetry` switches to `currentLocation->getPeer()`, but in this codebase only PRIMARY locations own `replicaPeer` (REPLICA locations don't have a peer). After the first switch to the replica, `currentLocation->hasPeer()` will be false and the retry loop will never switch back to the primary, so the intended primary<->replica alternation (and the `retryCnt % 2` sleep semantics) won’t work. Consider keeping explicit pointers to both primary and replica (e.g., `primary=&location` and `replica=location.getPeer()`) and toggling between them on each failure instead of relying on `hasPeer()` on the current location.
##########
cpp/celeborn/client/ShuffleClient.cpp:
##########
@@ -286,7 +290,19 @@ std::unique_ptr<CelebornInputStream> ShuffleClientImpl::readPartition(
attemptNumber,
startMapIndex,
endMapIndex,
- needCompression);
+ needCompression,
+ fetchExcludedWorkers_);
+}
+
+void ShuffleClientImpl::excludeFailedFetchLocation(
+ const std::string& hostAndFetchPort,
+ const std::exception& e) {
+ if (pushReplicateEnabled_ && fetchExcludeWorkerOnFailureEnabled_) {
Review Comment:
`ShuffleClientImpl::excludeFailedFetchLocation` ignores the passed exception (`e`) and will exclude the worker for any failure when the flags are enabled. This differs from the Java behavior (and from `CelebornInputStream::excludeFailedFetchLocation`) which only excludes on critical fetch causes. Use `utils::isCriticalCauseForFetch(e)` (or otherwise align the filtering logic) to avoid excluding workers on non-critical/read-corruption errors, and to avoid an unused-parameter warning for `e`.
```suggestion
if (pushReplicateEnabled_ && fetchExcludeWorkerOnFailureEnabled_ &&
utils::isCriticalCauseForFetch(e)) {
```
##########
cpp/celeborn/client/reader/CelebornInputStream.h:
##########
@@ -17,14 +17,19 @@
#pragma once
+#include <thread>
+
Review Comment:
`<thread>` is included in this header but the header doesn’t use any threading types; it’s only needed in the .cpp for `std::this_thread::sleep_for`. Moving this include to `CelebornInputStream.cpp` would reduce transitive compile-time overhead for all includers of this header.
```suggestion
```
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +205,58 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
Review Comment:
The new retry / peer-switching / exclusion behavior in `createReaderWithRetry` is substantial but there are no unit tests covering it in the C++ client test suite (there are tests for other reader components like `WorkerPartitionReader`). Adding targeted tests (e.g., a fake `PartitionReader`/factory that fails N times, verifies alternation between primary/replica, and verifies exclusion timestamps) would help prevent regressions and ensure parity with the Java implementation.
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +205,58 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ if (isExcluded(*currentLocation)) {
+ throw std::runtime_error(
+ "Fetch data from excluded worker! " +
+ currentLocation->hostAndFetchPort());
+ }
+ auto reader = createReader(*currentLocation);
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ excludeFailedFetchLocation(currentLocation->hostAndFetchPort(), e);
+ fetchChunkRetryCnt_++;
+
+ if (currentLocation->hasPeer() && !readSkewPartitionWithoutMapRange_) {
+ if (fetchChunkRetryCnt_ % 2 == 0) {
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(retryWait_.count()));
+ }
Review Comment:
This retry sleep can be simplified and made less error-prone by sleeping directly on `retryWait_` (it’s already a `std::chrono::milliseconds`) instead of re-wrapping `retryWait_.count()` into another `milliseconds` duration.
##########
cpp/celeborn/utils/CelebornUtils.cpp:
##########
@@ -19,6 +19,22 @@
namespace celeborn {
namespace utils {
+
+bool isCriticalCauseForFetch(const std::exception& e) {
+ if (dynamic_cast<const std::system_error*>(&e)) {
+ return true;
+ }
+
+ std::string msg = e.what();
+ if (msg.find("Connecting to") != std::string::npos ||
+ msg.find("Failed to") != std::string::npos ||
Review Comment:
`isCriticalCauseForFetch` claims to match Java/Scala behavior, but the string checks use `find` (substring match) rather than the Java `startsWith` semantics for "Connecting to" / "Failed to". Using a prefix check here would better match upstream behavior and avoid classifying unrelated messages that merely contain these phrases as "critical" (which can lead to unnecessary worker exclusion).
```suggestion
if (msg.find("Connecting to") == 0 ||
msg.find("Failed to") == 0 ||
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3906786871
@afterincomparableyum, could you please rebase the main branch to resolve the conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3925566753
@afterincomparableyum, could you please rebase the main branch to resolve the conflict?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3929189061
@SteNicholas I have rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3942228145
I will address the comments of CoPilot by tomorrow @SteNicholas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2844435446
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +205,56 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ if (isExcluded(*currentLocation)) {
+ CELEBORN_FAIL(
+ "Fetch data from excluded worker! {}",
+ currentLocation->hostAndFetchPort());
+ }
+ auto reader = createReader(*currentLocation);
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ excludeFailedFetchLocation(currentLocation->hostAndFetchPort(), e);
+ fetchChunkRetryCnt_++;
+
+ if (currentLocation->hasPeer() && !readSkewPartitionWithoutMapRange_) {
+ if (fetchChunkRetryCnt_ % 2 == 0) {
+ std::this_thread::sleep_for(retryWait_);
+ }
+ LOG(WARNING) << "CreatePartitionReader failed " << fetchChunkRetryCnt_
+ << "/" << fetchChunkMaxRetry_ << " times for location "
+ << currentLocation->hostAndFetchPort()
+ << ", change to peer. Error: " << e.what();
+ // TODO: When stream handlers are supported, send BUFFER_STREAM_END
+ // to close the active stream before switching to peer, matching the
+ // Java CelebornInputStream behavior. Currently, the C++ client does
+ // not have pre-opened stream handlers, so stream cleanup is handled
+ // by WorkerPartitionReader's destructor.
+ currentLocation = currentLocation->getPeer();
+ } else {
+ LOG(WARNING) << "CreatePartitionReader failed " << fetchChunkRetryCnt_
+ << "/" << fetchChunkMaxRetry_ << " times for location "
+ << currentLocation->hostAndFetchPort()
+ << ", retry the same location. Error: " << e.what();
+ std::this_thread::sleep_for(retryWait_);
+ }
+ }
+ }
+
+ // Max retries exceeded, rethrow the last exception wrapped with context
+ throw utils::CelebornRuntimeError(
+ lastException,
+ "createPartitionReader failed after " +
+ std::to_string(fetchChunkRetryCnt_) + " retries for location " +
+ location.hostAndFetchPort(),
+ false);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3938023365
@afterincomparableyum, thanks for update. Could you take a look at the review comment of Copilot?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3797400145
Thank you for your comments @SteNicholas , I will take a look over the next couple of days. I suspect some refactoring may need to be done to this PR, I will notify you once done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2731656758
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +196,45 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ auto reader = createReader(*currentLocation);
+ fetchChunkRetryCnt_ = 0;
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ fetchChunkRetryCnt_++;
+
+ if (currentLocation->hasPeer()) {
+ if (fetchChunkRetryCnt_ % 2 == 0) {
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(retryWait_.count()));
+ }
+ LOG(WARNING) << "CreatePartitionReader failed " << fetchChunkRetryCnt_
+ << "/" << fetchChunkMaxRetry_ << " times for location "
+ << currentLocation->hostAndFetchPort()
+ << ", change to peer. Error: " << e.what();
+ currentLocation = currentLocation->getPeer();
+ } else {
+ LOG(WARNING) << "CreatePartitionReader failed " << fetchChunkRetryCnt_
+ << "/" << fetchChunkMaxRetry_ << " times for location "
+ << currentLocation->hostAndFetchPort()
+ << ", retry the same location. Error: " << e.what();
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(retryWait_.count()));
+ }
+ }
+ }
+
+ CELEBORN_FAIL(
+ "createPartitionReader failed after " +
+ std::to_string(fetchChunkRetryCnt_) + " retries for location " +
+ location.hostAndFetchPort());
Review Comment:
The captured lastException should be rethrown instead of using CELEBORN_FAIL. The Java implementation throws CelebornIOException with the lastException as the cause. In C++, you should use std::rethrow_exception(lastException) to preserve the original exception information, which is critical for debugging. If you want to add context, you could wrap it in a CelebornRuntimeError similar to the pattern seen in CelebornException.cpp line 35-36.
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +196,45 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ auto reader = createReader(*currentLocation);
+ fetchChunkRetryCnt_ = 0;
Review Comment:
Resetting fetchChunkRetryCnt_ to 0 on successful reader creation (line 207) is redundant since it's already reset at line 187 in moveToNextReader() before this function is called. While not harmful, removing this reset would make the code cleaner and align better with the Java implementation which doesn't reset the counter after successful reader creation.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2785999778
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +196,45 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ auto reader = createReader(*currentLocation);
+ fetchChunkRetryCnt_ = 0;
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ fetchChunkRetryCnt_++;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#issuecomment-3735321034
@HolyLow @SteNicholas @FMX @RexXiong Could you please help review this PR? Appreciate your help in improving this as needed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2222][CIP-14] Support Retrying when createReader failed for CelebornInputStream in CppClient [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3583:
URL: https://github.com/apache/celeborn/pull/3583#discussion_r2807764659
##########
cpp/celeborn/client/reader/CelebornInputStream.cpp:
##########
@@ -189,9 +196,45 @@ void CelebornInputStream::moveToNextReader() {
std::shared_ptr<PartitionReader> CelebornInputStream::createReaderWithRetry(
const protocol::PartitionLocation& location) {
- // TODO: support retrying when createReader failed. Maybe switch to peer
- // location?
- return createReader(location);
+ const protocol::PartitionLocation* currentLocation = &location;
+ std::exception_ptr lastException;
+
+ while (fetchChunkRetryCnt_ < fetchChunkMaxRetry_) {
+ try {
+ VLOG(1) << "Create reader for location " << currentLocation->host << ":"
+ << currentLocation->fetchPort;
+ auto reader = createReader(*currentLocation);
+ fetchChunkRetryCnt_ = 0;
+ return reader;
+ } catch (const std::exception& e) {
+ lastException = std::current_exception();
+ fetchChunkRetryCnt_++;
+
+ if (currentLocation->hasPeer()) {
+ if (fetchChunkRetryCnt_ % 2 == 0) {
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(retryWait_.count()));
+ }
+ LOG(WARNING) << "CreatePartitionReader failed " << fetchChunkRetryCnt_
Review Comment:
resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org