You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "charleenklang (via GitHub)" <gi...@apache.org> on 2023/09/01 07:59:09 UTC
[GitHub] [camel-kafka-connector] charleenklang opened a new issue, #1561: How to backup a topic with partition and offset data using camel-minio-sink plugin
charleenklang opened a new issue, #1561:
URL: https://github.com/apache/camel-kafka-connector/issues/1561
Hi,
I am trying to backup Kafka topics to an s3 bucket and restore the data back to a Kafka topic.
What I am observing is that the offset will be restored but all messages will be written to the same partition in the restored topic, even if they were in the original topic in different partitions.
For example:
If we consume from the original topic with kcat:
```
% Reached end of topic kafkatopic-sample [0] at offset 4
% Reached end of topic kafkatopic-sample [2] at offset 0
% Reached end of topic kafkatopic-sample [1] at offset 9
```
If we backup the data and restore it back to a new topic:
```
% Reached end of topic kafkatopic-restore-test [1] at offset 13
% Reached end of topic kafkatopic-restore-test [0] at offset 0
% Reached end of topic kafkatopic-restore-test [2] at offset 0
```
Is it possible to backup the partition and offset data as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] valdar commented on issue #1561: How to backup a topic with partition and offset data using camel-minio-sink plugin
Posted by "valdar (via GitHub)" <gi...@apache.org>.
valdar commented on issue #1561:
URL: https://github.com/apache/camel-kafka-connector/issues/1561#issuecomment-1702635759
Well in theory you could save the partition of each message in s3 then reuse it to send it to a specific partition. Mind that I am not sure there are all the bits accessible already to achieve this. It sounds like a potentially interesting use case though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] charleenklang commented on issue #1561: How to backup a topic with partition and offset data using camel-minio-sink plugin
Posted by "charleenklang (via GitHub)" <gi...@apache.org>.
charleenklang commented on issue #1561:
URL: https://github.com/apache/camel-kafka-connector/issues/1561#issuecomment-1702735547
I agree that this is an interesting use case. Do you know if this is part of the road map?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] charleenklang commented on issue #1561: How to backup a topic with partition and offset data using camel-minio-sink plugin
Posted by "charleenklang (via GitHub)" <gi...@apache.org>.
charleenklang commented on issue #1561:
URL: https://github.com/apache/camel-kafka-connector/issues/1561#issuecomment-1702409581
Thanks for the answer :)
What happens with the messages that are not sent with the message key and are assigned based on round robin to a partition? In that case it will not be possible to restore the data to the original partition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] charleenklang commented on issue #1561: How to backup a topic with partition and offset data using camel-minio-sink plugin
Posted by "charleenklang (via GitHub)" <gi...@apache.org>.
charleenklang commented on issue #1561:
URL: https://github.com/apache/camel-kafka-connector/issues/1561#issuecomment-1705450664
I have partition and key in the data I want to safe to s3. But it seems to me that only the message (value) will be stored in the s3 objects. And restoring will result in publishing the messages to the same partition without any keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] valdar commented on issue #1561: How to backup a topic with partition and offset data using camel-minio-sink plugin
Posted by "valdar (via GitHub)" <gi...@apache.org>.
valdar commented on issue #1561:
URL: https://github.com/apache/camel-kafka-connector/issues/1561#issuecomment-1702369323
If the partition was originally assigned from the message key, you need to restore the messages using the same key. That requires some extra work that might or not be possible doing ootb depending on the fact that is possible to easily extract the key from s3 saved data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] valdar commented on issue #1561: How to backup a topic with partition and offset data using camel-minio-sink plugin
Posted by "valdar (via GitHub)" <gi...@apache.org>.
valdar commented on issue #1561:
URL: https://github.com/apache/camel-kafka-connector/issues/1561#issuecomment-1704385151
Well there is a part that is specific to your data that is not easy to generalize: do you have partition and/or key in the data you save to s3?
Then we can start to think about how to restore it using the source connector. As I said something might be already in place that we can use for this purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] charleenklang commented on issue #1561: How to backup a topic with partition and offset data using camel-minio-sink plugin
Posted by "charleenklang (via GitHub)" <gi...@apache.org>.
charleenklang commented on issue #1561:
URL: https://github.com/apache/camel-kafka-connector/issues/1561#issuecomment-1705531039
For example, there is a message in `topic-1` like this:
```txt
Key (1 bytes): test-key
Value (4 bytes): test
Timestamp: 1693840308197 Partition: 1 Offset: 4
```
The object in the s3 bucket:
```txt
$ mc cat mybucket/myobject
test
```
After restoring the message to a new topic it looks like this:
```test
Key (-1 bytes):
Value (4 bytes): test
Timestamp: 1693840947125 Partition: 2 Offset: 5
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org