[GitHub] [iceberg] kbendick commented on issue #4977: Support Kafka Connect within Iceberg

[GitBox <gi...@apache.org>]

kbendick commented on issue #4977:
URL: https://github.com/apache/iceberg/issues/4977#issuecomment-1154099316

   Thanks for opening this @kondamudikarthik!
   
   @youngxinler: first, thanks for your interest in this discussions. It’s not entirely true that writing to Iceberg tables needs to be achieved via a computing engine. It can absolutely be achieved with just the plain Java API (or the new Python refactor). While I generally recommend that a compute engine be used, as it’s an easier experience, my understanding is that Kafka Connect is a framework for map-side only / per-record transforms for writing data from Kafka to some other location (in this case Iceberg tables).
   
   Near-real time data is very often written as-is or with only a handful of per-record transforms. For example, upstream producers will stage the data as they want it written, and then the folks who maintain the data lake might add some metadata to the record (ingestion timestamp etc) but otherwise write the data out as it’s been received, while ensuring that the data has consistent schema.
   
   There are many scenarios in which Kafka Connect would be beneficial to end users and if there’s interest from the community then it’s something we should definitely consider supporting.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org