You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ajreid21 (via GitHub)" <gi...@apache.org> on 2026/07/14 14:31:33 UTC
[PR] Kafka Connect: Set engine metadata in EnvironmentContext [iceberg]
ajreid21 opened a new pull request, #17195:
URL: https://github.com/apache/iceberg/pull/17195
Set Kafka Connect engine name and version during connector and task startup so commits include the Kafka runtime context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Set engine metadata in EnvironmentContext [iceberg]
Posted by "AnatolyPopov (via GitHub)" <gi...@apache.org>.
AnatolyPopov commented on PR #17195:
URL: https://github.com/apache/iceberg/pull/17195#issuecomment-4978105085
Hi @ajreid21, just curious, what is the use case for this? The main one I can think of is when something breaks after a Kafka Connect runtime upgrade and you need to know which version was used for a particular commit. That feels like a pretty rare case, while iceberg-version already provides most of the release-level traceability for the connector. Is the motivation mainly consistency with Spark and Flink, or am I missing another use case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Set engine metadata in EnvironmentContext [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #17195:
URL: https://github.com/apache/iceberg/pull/17195#issuecomment-5028677992
+1 to additionally keeping the engine metadata in the environment context. I think it's a good idea to get as many clients setting this as possible just so it's at least a bit more reliable for implementations to leverage. It's also pretty lightweight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Set engine metadata in EnvironmentContext [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar merged PR #17195:
URL: https://github.com/apache/iceberg/pull/17195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Set engine metadata in EnvironmentContext [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #17195:
URL: https://github.com/apache/iceberg/pull/17195#issuecomment-5036486355
Thanks @ajreid21 and @AnatolyPopov @singhpk234 for reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Set engine metadata in EnvironmentContext [iceberg]
Posted by "ajreid21 (via GitHub)" <gi...@apache.org>.
ajreid21 commented on PR #17195:
URL: https://github.com/apache/iceberg/pull/17195#issuecomment-4982606729
Yeah, there is already precedent in Spark and Flink. This is helpful for IRC catalog services to quickly understand what external engines are writing to what tables (currently, there is not another good way to get this context as there is no other defined header for this and user-agent is usually too generic). It's also helpful for gathering as much context as possible for when trying to debug an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Set engine metadata in EnvironmentContext [iceberg]
Posted by "ajreid21 (via GitHub)" <gi...@apache.org>.
ajreid21 commented on PR #17195:
URL: https://github.com/apache/iceberg/pull/17195#issuecomment-5023620363
@singhpk234 Yeah, the flink state properties are in the snapshot summary already. I think explicitly setting the engine-name property makes the downstream consumer logic simpler and there is less worry that this property name ([referenced in the spec](https://github.com/apache/iceberg/blob/1ec15051dc09ef6a0d61fa332eb8f2503f3170f8/format/spec.md?plain=1#L2112)) may change, etc. with potentially new engine version/implementations (plus you can't get engine version from the flink state properties - although I agree version is less important).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org