You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Randall Hauch (Jira)" <ji...@apache.org> on 2021/10/05 18:03:00 UTC
[jira] [Assigned] (KAFKA-10865) Improve trace-logging for
Transformations (including Predicates)
[ https://issues.apache.org/jira/browse/KAFKA-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Randall Hauch reassigned KAFKA-10865:
-------------------------------------
Assignee: Tiago Martins (was: Govinda Sakhare)
> Improve trace-logging for Transformations (including Predicates)
> ----------------------------------------------------------------
>
> Key: KAFKA-10865
> URL: https://issues.apache.org/jira/browse/KAFKA-10865
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Robin Moffatt
> Assignee: Tiago Martins
> Priority: Major
> Labels: newbie
>
> I've been spending [a bunch of time poking around SMTs|https://rmoff.net/categories/twelvedaysofsmt/] recently, and one common challenge I've had is being able to debug when things don't behave as I expect.
>
> I know that there is the {{TransformationChain}} logger, but this only gives (IIUC) the input record
> {code:java}
> [2020-12-17 09:38:58,057] TRACE [sink-simulator-day12-00|task-0] Applying transformation io.confluent.connect.transforms.Filter$Value to SinkRecord{kafkaOffset=10551, timestampType=CreateTime} ConnectRecord{topic='day12-sys01', kafkaPartition=0, key=2c2ceb9b-8b31-4ade-a757-886ebfb7a398, keySchema=Schema{STRING}, value=Struct{units=16,product=Founders Breakfast Stout,amount=30.41,txn_date=Sat Dec 12 18:21:18 GMT 2020,source=SYS01}, valueSchema=Schema{io.mdrogalis.Gen0:STRUCT}, timestamp=1608197938054, headers=ConnectHeaders(headers=)} (org.apache.kafka.connect.runtime.TransformationChain:47)
> {code}
>
> I think it would be really useful to also have trace level logging that included:
> - the _output_ of *each* transform
> - the evaluation and result of any `predicate`s
> I have been using {{com.github.jcustenborder.kafka.connect.simulator.SimulatorSinkConnector}} which is really useful for seeing the final record:
> {code:java}
> [2020-12-17 09:38:58,057] INFO [sink-simulator-day12-00|task-0] record.value=Struct{units=16,product=Founders Breakfast Stout,amount=30.41,txn_date=Sat Dec 12 18:21:18 GMT 2020,source=SYS01} (com.github.jcustenborder.kafka.connect.simulator.SimulatorSinkTask:50)
> {code}
>
> But doesn't include things like topic name (which is often changed by common SMTs)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)