You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "jeqo (via GitHub)" <gi...@apache.org> on 2023/06/28 09:57:35 UTC
[GitHub] [kafka] jeqo opened a new pull request, #13923: KAFKA-15131: Improve RemoteStorageManager exception handling
jeqo opened a new pull request, #13923:
URL: https://github.com/apache/kafka/pull/13923
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jeqo commented on pull request #13923: KAFKA-15131: Improve RemoteStorageManager exception handling documentation
Posted by "jeqo (via GitHub)" <gi...@apache.org>.
jeqo commented on PR #13923:
URL: https://github.com/apache/kafka/pull/13923#issuecomment-1611590682
Sure, make sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jeqo commented on pull request #13923: KAFKA-15131: Improve RemoteStorageManager exception handling documentation
Posted by "jeqo (via GitHub)" <gi...@apache.org>.
jeqo commented on PR #13923:
URL: https://github.com/apache/kafka/pull/13923#issuecomment-1611480623
@divijvaidya adding the following changes to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=97554472&selectedPageVersions=363&selectedPageVersions=362
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on a diff in pull request #13923: KAFKA-15131: Improve RemoteStorageManager exception handling documentation
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #13923:
URL: https://github.com/apache/kafka/pull/13923#discussion_r1245118305
##########
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java:
##########
@@ -107,20 +110,22 @@ InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
* @param endPosition end position of log segment to be read, inclusive.
* @return input stream of the requested log segment data.
* @throws RemoteStorageException if there are any errors while fetching the desired segment.
- * @throws RemoteResourceNotFoundException when there are no resources associated with the given remoteLogSegmentMetadata.
+ * @throws RemoteResourceNotFoundException the requested log segment is not found in the remote storage.
*/
InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
int startPosition,
int endPosition) throws RemoteStorageException;
/**
* Returns the index for the respective log segment of {@link RemoteLogSegmentMetadata}.
+ * <p>
+ * If the index is not present (e.g. Transaction index may not exist), throws {@link RemoteResourceNotFoundException}
Review Comment:
Perhaps, we can be more descriptive here. My suggestion is to add the following (free free to change the wording):
`e.g. Transaction index may not exist because segments create prior to version 2.8.0 will not have transaction index associated with them.`
`@throws RemoteResourceNotFoundException the requested index is not found in the remote storage. The caller of this function are encouraged to re-create the indexes from the segment as the suggested way of handling this error.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jeqo commented on a diff in pull request #13923: KAFKA-15131: Improve RemoteStorageManager exception handling documentation
Posted by "jeqo (via GitHub)" <gi...@apache.org>.
jeqo commented on code in PR #13923:
URL: https://github.com/apache/kafka/pull/13923#discussion_r1245252224
##########
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java:
##########
@@ -107,20 +110,22 @@ InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
* @param endPosition end position of log segment to be read, inclusive.
* @return input stream of the requested log segment data.
* @throws RemoteStorageException if there are any errors while fetching the desired segment.
- * @throws RemoteResourceNotFoundException when there are no resources associated with the given remoteLogSegmentMetadata.
+ * @throws RemoteResourceNotFoundException the requested log segment is not found in the remote storage.
*/
InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
int startPosition,
int endPosition) throws RemoteStorageException;
/**
* Returns the index for the respective log segment of {@link RemoteLogSegmentMetadata}.
+ * <p>
+ * If the index is not present (e.g. Transaction index may not exist), throws {@link RemoteResourceNotFoundException}
Review Comment:
thanks @divijvaidya, applying this suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #13923: KAFKA-15131: Improve RemoteStorageManager exception handling documentation
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #13923:
URL: https://github.com/apache/kafka/pull/13923#issuecomment-1611315037
I'll take a look this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jeqo commented on pull request #13923: KAFKA-15131: Improve RemoteStorageManager exception handling documentation
Posted by "jeqo (via GitHub)" <gi...@apache.org>.
jeqo commented on PR #13923:
URL: https://github.com/apache/kafka/pull/13923#issuecomment-1614397965
@showuon I've rephrased the comments, lmk how it looks now. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13923: KAFKA-15131: Improve RemoteStorageManager exception handling documentation
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13923:
URL: https://github.com/apache/kafka/pull/13923#discussion_r1247695189
##########
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java:
##########
@@ -120,7 +123,10 @@ InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
* @param indexType type of the index to be fetched for the segment.
* @return input stream of the requested index.
* @throws RemoteStorageException if there are any errors while fetching the index.
- * @throws RemoteResourceNotFoundException when there are no resources associated with the given remoteLogSegmentMetadata.
+ * @throws RemoteResourceNotFoundException the requested index is not found in the remote storage
+ * (e.g. Transaction index may not exist because segments create prior to version 2.8.0 will not have transaction index associated with them.).
Review Comment:
minor typo: `segments create prior to version` -> `segments created prior to version`
##########
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java:
##########
@@ -75,6 +75,9 @@ enum IndexType {
* <p>
* Invoker of this API should always send a unique id as part of {@link RemoteLogSegmentMetadata#remoteLogSegmentId()}
* even when it retries to invoke this method for the same log segment data.
+ * <p>
+ * This operation is expected to be idempotent. If a copy operation is retried and there is existing content already written,
Review Comment:
Newly added statement gives more clarity on idempotency.
The given unique id sent as part of `RemoteLogSegmentMetadata` can be used to avoid overwrites in RSM implementation. Caller makes sure it sends unique id for multiple invocations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on pull request #13923: KAFKA-15131: Improve RemoteStorageManager exception handling documentation
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #13923:
URL: https://github.com/apache/kafka/pull/13923#issuecomment-1611589151
Thanks @jeqo. Looks good to me but we will wait for Luke and/or Satish to look into this before we merge this one in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #13923: KAFKA-15131: Improve RemoteStorageManager exception handling documentation
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #13923:
URL: https://github.com/apache/kafka/pull/13923#discussion_r1247591766
##########
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java:
##########
@@ -107,20 +110,26 @@ InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
* @param endPosition end position of log segment to be read, inclusive.
* @return input stream of the requested log segment data.
* @throws RemoteStorageException if there are any errors while fetching the desired segment.
- * @throws RemoteResourceNotFoundException when there are no resources associated with the given remoteLogSegmentMetadata.
+ * @throws RemoteResourceNotFoundException the requested log segment is not found in the remote storage.
*/
InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
int startPosition,
int endPosition) throws RemoteStorageException;
/**
* Returns the index for the respective log segment of {@link RemoteLogSegmentMetadata}.
+ * <p>
+ * If the index is not present (e.g. Transaction index may not exist because segments create prior to
+ * version 2.8.0 will not have transaction index associated with them.),
+ * throws {@link RemoteResourceNotFoundException}
*
* @param remoteLogSegmentMetadata metadata about the remote log segment.
* @param indexType type of the index to be fetched for the segment.
* @return input stream of the requested index.
* @throws RemoteStorageException if there are any errors while fetching the index.
- * @throws RemoteResourceNotFoundException when there are no resources associated with the given remoteLogSegmentMetadata.
+ * @throws RemoteResourceNotFoundException the requested index is not found in the remote storage.
+ * The caller of this function are encouraged to re-create the indexes from the segment
+ * as the suggested way of handling this error.
Review Comment:
This is confused because transactionIndex should be expected to be non-existed. Maybe we can put it this way:
`The caller of this function are encouraged to re-create the indexes from the segment as the suggested way of handling this error if the index is expected to be existed.`
WDYT?
##########
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java:
##########
@@ -107,20 +110,26 @@ InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
* @param endPosition end position of log segment to be read, inclusive.
* @return input stream of the requested log segment data.
* @throws RemoteStorageException if there are any errors while fetching the desired segment.
- * @throws RemoteResourceNotFoundException when there are no resources associated with the given remoteLogSegmentMetadata.
+ * @throws RemoteResourceNotFoundException the requested log segment is not found in the remote storage.
*/
InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata,
int startPosition,
int endPosition) throws RemoteStorageException;
/**
* Returns the index for the respective log segment of {@link RemoteLogSegmentMetadata}.
+ * <p>
+ * If the index is not present (e.g. Transaction index may not exist because segments create prior to
+ * version 2.8.0 will not have transaction index associated with them.),
Review Comment:
I think this line should be put below:
`* @throws RemoteResourceNotFoundException the requested index is not found in the remote storage (e.g. Transaction index may not exist because segments create prior to version 2.8.0 will not have transaction index associated with them.).`
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13923: KAFKA-15131: Improve RemoteStorageManager exception handling documentation
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13923:
URL: https://github.com/apache/kafka/pull/13923#discussion_r1247690846
##########
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java:
##########
@@ -75,6 +75,9 @@ enum IndexType {
* <p>
* Invoker of this API should always send a unique id as part of {@link RemoteLogSegmentMetadata#remoteLogSegmentId()}
* even when it retries to invoke this method for the same log segment data.
+ * <p>
+ * This operation is expected to be idempotent. If a copy operation is retried and there is existing content already written,
Review Comment:
Newly added statement gives more clarity on idempotency.
The unique id sent as part of `RemoteLogSegmentMetadata` can be used to avoid overwrites in RSM implementation. Caller makes sure it sends unique id for multiple invocations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya merged pull request #13923: KAFKA-15131: Improve RemoteStorageManager exception handling documentation
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya merged PR #13923:
URL: https://github.com/apache/kafka/pull/13923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org