You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/03/29 15:42:58 UTC

[GitHub] [camel-kafka-connector] PLCyr opened a new issue #1128: Connector Kafka Idempotency Crash Resilience

PLCyr opened a new issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128


   Hi, 
   
   I'm new to Camel Kafka Connector. I configured a SFTP Connector in Kubernetes with download=false and the Kafka Topic idempotency feature enable. The idempotency work fine with the FileName as long as the Kafka Connect Pod is running. However after a connector restart or pod crash, all the files that are still on the SFTP server have newly generated events when the connector is up again. I was expecting the idempotency feature to re-consume the idempotency Kafka topic at offset 0 so it could have some kind of crash resilience. Same behavior with FTP / FTPS / SFTP connector.
   
   Is it the expected behavior or a bug ?
   
   **Repo steps**
   For my test, I uploaded a file called "Test_2.txt" in my SFTP Server. The SFTP connector saw it and resulted in one event in Topic and IdempotencyTopic. I deleted my Kafka Connect pod to simulate a crash. Upon restart, my Topic and IdempotencyTopic each have a second event which is a duplicate of the Test_2.txt.
   
   **Setup** 
   Windows 10 Pro with Kubernetes provided by Docker Desktop
   Confluent Kafka running in Kubernetes
   Camel Kafka Connector SFTP Version 0.8.0 running in Kubernetes
   
   **SFTP Connector Config**
   "name": "sftp-connector",
   "connector.class": "org.apache.camel.kafkaconnector.sftp.CamelSftpSourceConnector",
   "key.converter": "org.apache.kafka.connect.storage.StringConverter",
   "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
   "transforms": "RemoteTransformer",
   "transforms.RemoteTransformer.type": "org.apache.camel.kafkaconnector.sftp.transformers.SftpRemoteFileTransforms",
   "camel.idempotency.enabled": "true",
   "camel.idempotency.expression.type": "header",
   "camel.idempotency.expression.header": "CamelFileNameConsumed",
   "camel.idempotency.repository.type": "kafka",
   "camel.idempotency.kafka.bootstrap.servers": "confluent-kafka-cp-kafka:9092",
   "camel.idempotency.kafka.topic": "SFTP-Persistance-Topic",
   "camel.source.endpoint.recursive": "true",
   "camel.source.path.host": "sftp.sftp",
   "camel.source.path.port": "22",
   "camel.source.endpoint.username": "foo",
   "camel.source.endpoint.password": "pass",
   "camel.source.endpoint.include": ".*.txt",
   "camel.source.path.directoryName": "data/",
   "camel.source.endpoint.download": "false",
   "camel.source.endpoint.noop": "true",
   "camel.source.endpoint.binary": "false",
   "camel.source.contentLogLevel": "INFO",
   "camel.source.endpoint.runLoggingLevel": "INFO",
   "topics": "SFTP-Topic"
   
   **Log at connector restart**
   [2021-03-29 15:03:27,822] INFO Creating Camel route from(sftp:sftp.sftp:22/data/?binary=false&download=false&include=.*.txt&noop=true&password=RAW(pass)&recursive=true&runLoggingLevel=INFO&username=RAW(foo)) (org.apache.camel.kafkaconnector.utils.CamelKafkaConnectMain)
   [2021-03-29 15:03:27,824] INFO idempotentConsumer(header(expressionHeader), MemoryIdempotentRepository.memoryIdempotentRepository(100)).to(direct:end?pollingConsumerQueueSize=1000&pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true) (org.apache.camel.kafkaconnector.utils.CamelKafkaConnectMain)
   [2021-03-29 15:03:27,943] INFO Endpoint is configured with noop=true so forcing endpoint to be idempotent as well (org.apache.camel.component.file.remote.RemoteFileEndpoint)
   [2021-03-29 15:03:27,944] INFO Using default memory based idempotent repository with cache max size: 1000 (org.apache.camel.component.file.remote.RemoteFileEndpoint)
   [2021-03-29 15:03:27,954] INFO ConsumerConfig values:
           allow.auto.create.topics = true
           auto.commit.interval.ms = 5000
           auto.offset.reset = latest
           bootstrap.servers = [confluent-kafka-cp-kafka:9092]
           check.crcs = true
           client.dns.lookup = use_all_dns_ips
           client.id = consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4
           client.rack =
           connections.max.idle.ms = 540000
           default.api.timeout.ms = 60000
           enable.auto.commit = true
           exclude.internal.topics = true
           fetch.max.bytes = 52428800
           fetch.max.wait.ms = 500
           fetch.min.bytes = 1
           group.id = 3efa9c9e-3694-451d-a70b-9690bba7fa4f
           group.instance.id = null
           heartbeat.interval.ms = 3000
           interceptor.classes = []
           internal.leave.group.on.close = true
           internal.throw.on.fetch.stable.offset.unsupported = false
           isolation.level = read_uncommitted
           key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
           max.partition.fetch.bytes = 1048576
           max.poll.interval.ms = 300000
           max.poll.records = 500
           metadata.max.age.ms = 300000
           metric.reporters = []
           metrics.num.samples = 2
           metrics.recording.level = INFO
           metrics.sample.window.ms = 30000
           partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
           receive.buffer.bytes = 65536
           reconnect.backoff.max.ms = 1000
           reconnect.backoff.ms = 50
           request.timeout.ms = 30000
           retry.backoff.ms = 100
           sasl.client.callback.handler.class = null
           sasl.jaas.config = null
           sasl.kerberos.kinit.cmd = /usr/bin/kinit
           sasl.kerberos.min.time.before.relogin = 60000
           sasl.kerberos.service.name = null
           sasl.kerberos.ticket.renew.jitter = 0.05
           sasl.kerberos.ticket.renew.window.factor = 0.8
           sasl.login.callback.handler.class = null
           sasl.login.class = null
           sasl.login.refresh.buffer.seconds = 300
           sasl.login.refresh.min.period.seconds = 60
           sasl.login.refresh.window.factor = 0.8
           sasl.login.refresh.window.jitter = 0.05
           sasl.mechanism = GSSAPI
           security.protocol = PLAINTEXT
           security.providers = null
           send.buffer.bytes = 131072
           session.timeout.ms = 10000
           socket.connection.setup.timeout.max.ms = 127000
           socket.connection.setup.timeout.ms = 10000
           ssl.cipher.suites = null
           ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
           ssl.endpoint.identification.algorithm = https
           ssl.engine.factory.class = null
           ssl.key.password = null
           ssl.keymanager.algorithm = SunX509
           ssl.keystore.certificate.chain = null
           ssl.keystore.key = null
           ssl.keystore.location = null
           ssl.keystore.password = null
           ssl.keystore.type = JKS
           ssl.protocol = TLSv1.3
           ssl.provider = null
           ssl.secure.random.implementation = null
           ssl.trustmanager.algorithm = PKIX
           ssl.truststore.certificates = null
           ssl.truststore.location = null
           ssl.truststore.password = null
           ssl.truststore.type = JKS
           value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
    (org.apache.kafka.clients.consumer.ConsumerConfig)
   [2021-03-29 15:03:27,975] INFO Kafka version: 6.1.0-ccs (org.apache.kafka.common.utils.AppInfoParser)
   [2021-03-29 15:03:27,975] INFO Kafka commitId: 5496d92defc9bbe4 (org.apache.kafka.common.utils.AppInfoParser)
   [2021-03-29 15:03:27,975] INFO Kafka startTimeMs: 1617030207975 (org.apache.kafka.common.utils.AppInfoParser)
   [2021-03-29 15:03:27,975] INFO ProducerConfig values:
           acks = 1
           batch.size = 0
           bootstrap.servers = [confluent-kafka-cp-kafka:9092]
           buffer.memory = 33554432
           client.dns.lookup = use_all_dns_ips
           client.id = producer-4
           compression.type = none
           connections.max.idle.ms = 540000
           delivery.timeout.ms = 120000
           enable.idempotence = false
           interceptor.classes = []
           internal.auto.downgrade.txn.commit = false
           key.serializer = class org.apache.kafka.common.serialization.StringSerializer
           linger.ms = 0
           max.block.ms = 60000
           max.in.flight.requests.per.connection = 5
           max.request.size = 1048576
           metadata.max.age.ms = 300000
           metadata.max.idle.ms = 300000
           metric.reporters = []
           metrics.num.samples = 2
           metrics.recording.level = INFO
           metrics.sample.window.ms = 30000
           partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
           receive.buffer.bytes = 32768
           reconnect.backoff.max.ms = 1000
           reconnect.backoff.ms = 50
           request.timeout.ms = 30000
           retries = 2147483647
           retry.backoff.ms = 100
           sasl.client.callback.handler.class = null
           sasl.jaas.config = null
           sasl.kerberos.kinit.cmd = /usr/bin/kinit
           sasl.kerberos.min.time.before.relogin = 60000
           sasl.kerberos.service.name = null
           sasl.kerberos.ticket.renew.jitter = 0.05
           sasl.kerberos.ticket.renew.window.factor = 0.8
           sasl.login.callback.handler.class = null
           sasl.login.class = null
           sasl.login.refresh.buffer.seconds = 300
           sasl.login.refresh.min.period.seconds = 60
           sasl.login.refresh.window.factor = 0.8
           sasl.login.refresh.window.jitter = 0.05
           sasl.mechanism = GSSAPI
           security.protocol = PLAINTEXT
           security.providers = null
           send.buffer.bytes = 131072
           socket.connection.setup.timeout.max.ms = 127000
           socket.connection.setup.timeout.ms = 10000
           ssl.cipher.suites = null
           ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
           ssl.endpoint.identification.algorithm = https
           ssl.engine.factory.class = null
           ssl.key.password = null
           ssl.keymanager.algorithm = SunX509
           ssl.keystore.certificate.chain = null
           ssl.keystore.key = null
           ssl.keystore.location = null
           ssl.keystore.password = null
           ssl.keystore.type = JKS
           ssl.protocol = TLSv1.3
           ssl.provider = null
           ssl.secure.random.implementation = null
           ssl.trustmanager.algorithm = PKIX
           ssl.truststore.certificates = null
           ssl.truststore.location = null
           ssl.truststore.password = null
           ssl.truststore.type = JKS
           transaction.timeout.ms = 60000
           transactional.id = null
           value.serializer = class org.apache.kafka.common.serialization.StringSerializer
    (org.apache.kafka.clients.producer.ProducerConfig)
   [2021-03-29 15:03:27,979] INFO Kafka version: 6.1.0-ccs (org.apache.kafka.common.utils.AppInfoParser)
   [2021-03-29 15:03:27,980] INFO Kafka commitId: 5496d92defc9bbe4 (org.apache.kafka.common.utils.AppInfoParser)
   [2021-03-29 15:03:27,980] INFO Kafka startTimeMs: 1617030207979 (org.apache.kafka.common.utils.AppInfoParser)
   [2021-03-29 15:03:27,984] INFO SourceConnectorConfig values:
           config.action.reload = restart
           connector.class = org.apache.camel.kafkaconnector.sftp.CamelSftpSourceConnector
           errors.log.enable = false
           errors.log.include.messages = false
           errors.retry.delay.max.ms = 60000
           errors.retry.timeout = 0
           errors.tolerance = none
           header.converter = null
           key.converter = class org.apache.kafka.connect.storage.StringConverter
           name = sftp-connector
           predicates = []
           tasks.max = 1
           topic.creation.groups = []
           transforms = [RemoteTransformer]
           value.converter = class org.apache.kafka.connect.converters.ByteArrayConverter
    (org.apache.kafka.connect.runtime.SourceConnectorConfig)
   [2021-03-29 15:03:27,986] INFO Warming up cache from topic SFTP-Persistance-Topic (org.apache.camel.processor.idempotent.kafka.KafkaIdempotentRepository)
   [2021-03-29 15:03:27,986] INFO EnrichedConnectorConfig values:
           config.action.reload = restart
           connector.class = org.apache.camel.kafkaconnector.sftp.CamelSftpSourceConnector
           errors.log.enable = false
           errors.log.include.messages = false
           errors.retry.delay.max.ms = 60000
           errors.retry.timeout = 0
           errors.tolerance = none
           header.converter = null
           key.converter = class org.apache.kafka.connect.storage.StringConverter
           name = sftp-connector
           predicates = []
           tasks.max = 1
           topic.creation.groups = []
           transforms = [RemoteTransformer]
           transforms.RemoteTransformer.key = null
           transforms.RemoteTransformer.negate = false
           transforms.RemoteTransformer.predicate =
           transforms.RemoteTransformer.type = class org.apache.camel.kafkaconnector.sftp.transformers.SftpRemoteFileTransforms
           value.converter = class org.apache.kafka.connect.converters.ByteArrayConverter
    (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig)
   [2021-03-29 15:03:27,987] INFO Setting task configurations for 1 workers. (org.apache.camel.kafkaconnector.CamelSourceConnector)
   [2021-03-29 15:03:27,990] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] Subscribed to topic(s): SFTP-Persistance-Topic (org.apache.kafka.clients.consumer.KafkaConsumer)
   [2021-03-29 15:03:28,001] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] Cluster ID: fFQtkPXrT92v270crRLidQ (org.apache.kafka.clients.Metadata)
   [2021-03-29 15:03:28,001] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] Discovered group coordinator confluent-kafka-cp-kafka-2.confluent-kafka-cp-kafka-headless.kafka:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
   [2021-03-29 15:03:28,004] INFO [Producer clientId=producer-4] Cluster ID: fFQtkPXrT92v270crRLidQ (org.apache.kafka.clients.Metadata)
   [2021-03-29 15:03:28,006] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
   [2021-03-29 15:03:28,074] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
   [2021-03-29 15:03:31,096] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] Successfully joined group with generation Generation{generationId=1, memberId='consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4-124f20c5-6776-424e-abfe-ec07081e230a', protocol='range'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
   [2021-03-29 15:03:31,100] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] Finished assignment for group at generation 1: {consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4-124f20c5-6776-424e-abfe-ec07081e230a=Assignment(partitions=[SFTP-Persistance-Topic-0])} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
   [2021-03-29 15:03:31,142] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] Successfully synced group in generation Generation{generationId=1, memberId='consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4-124f20c5-6776-424e-abfe-ec07081e230a', protocol='range'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
   [2021-03-29 15:03:31,142] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] Notifying assignor about the new Assignment(partitions=[SFTP-Persistance-Topic-0]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
   [2021-03-29 15:03:31,142] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] Adding newly assigned partitions: SFTP-Persistance-Topic-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
   [2021-03-29 15:03:31,149] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] Found no committed offset for partition SFTP-Persistance-Topic-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
   [2021-03-29 15:03:31,153] INFO Cache OK (org.apache.camel.processor.idempotent.kafka.KafkaIdempotentRepository)
   [2021-03-29 15:03:31,164] INFO Known host file not configured, using user known host file: /home/appuser/.ssh/known_hosts (org.apache.camel.component.file.remote.SftpOperations)
   [2021-03-29 15:03:31,166] INFO [Consumer clientId=consumer-3efa9c9e-3694-451d-a70b-9690bba7fa4f-4, groupId=3efa9c9e-3694-451d-a70b-9690bba7fa4f] Resetting offset for partition SFTP-Persistance-Topic-0 to position FetchPosition{offset=1, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[confluent-kafka-cp-kafka-1.confluent-kafka-cp-kafka-headless.kafka:9092 (id: 1 rack: null)], epoch=0}}. (org.apache.kafka.clients.consumer.internals.SubscriptionState)
   [2021-03-29 15:03:31,713] WARN JSCH -> Permanently added 'sftp.sftp' (RSA) to the list of known hosts. (org.apache.camel.component.file.remote.SftpOperations)
   [2021-03-29 15:03:31,713] WARN Server asks for confirmation (yes|no): /home/appuser/.ssh/known_hosts does not exist.
   Are you sure you want to create it?. Camel will answer no. (org.apache.camel.component.file.remote.SftpOperations)
   [2021-03-29 15:03:32,037] INFO Routes startup summary (total:1 started:1) (org.apache.camel.impl.engine.AbstractCamelContext)
   [2021-03-29 15:03:32,038] INFO  Started route1 (sftp://sftp.sftp:22/data/) (org.apache.camel.impl.engine.AbstractCamelContext)
   [2021-03-29 15:03:32,038] INFO Apache Camel 3.8.0 (camel-1) started in 4s306ms (build:98ms init:117ms start:4s91ms) (org.apache.camel.impl.engine.AbstractCamelContext)
   [2021-03-29 15:03:32,039] INFO CamelSourceTask connector task started (org.apache.camel.kafkaconnector.CamelSourceTask)
   [2021-03-29 15:03:32,040] INFO WorkerSourceTask{id=sftp-connector-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask)
   [2021-03-29 15:03:33,035] INFO Scheduled task started on:   sftp://sftp.sftp:22/data/?binary=false&download=false&include=.*.txt&noop=true&password=xxxxxx (org.apache.camel.support.ScheduledPollConsumer)
   [2021-03-29 15:03:33,079] INFO SourceRecord{sourcePartition={filename=sftp://sftp.sftp:22/data/?binary=false&download=false&include=.*.txt&noop=true&password=xxxxxx}, sourceOffset={position=9F81987AE7F9AE2-0000000000000000}} ConnectRecord{topic='SFTP-Topic', kafkaPartition=null, key=null, keySchema=null, value=null, valueSchema=null, timestamp=1617030213076, headers=ConnectHeaders(headers=[ConnectHeader(key=CamelHeader.CamelFileAbsolute, value=false, schema=Schema{BOOLEAN}), ConnectHeader(key=CamelHeader.CamelFileAbsolutePath, value=data/Test_2.txt, schema=Schema{STRING}), ConnectHeader(key=CamelHeader.CamelFileHost, value=sftp.sftp, schema=Schema{STRING}), ConnectHeader(key=CamelHeader.CamelFileLastModified, value=1616182842000, schema=Schema{INT64}), ConnectHeader(key=CamelHeader.CamelFileLength, value=72, schema=Schema{INT64}), ConnectHeader(key=CamelHeader.CamelFileName, value=Test_2.txt, schema=Schema{STRING}), ConnectHeader(key=CamelHeader.CamelFileNameConsumed, value=Test_2
 .txt, schema=Schema{STRING}), ConnectHeader(key=CamelHeader.CamelFileNameOnly, value=Test_2.txt, schema=Schema{STRING}), ConnectHeader(key=CamelHeader.CamelFileParent, value=data, schema=Schema{STRING}), ConnectHeader(key=CamelHeader.CamelFilePath, value=data//Test_2.txt, schema=Schema{STRING}), ConnectHeader(key=CamelHeader.CamelFileRelativePath, value=Test_2.txt, schema=Schema{STRING}), ConnectHeader(key=CamelProperty.CamelBatchSize, value=1, schema=Schema{INT32}), ConnectHeader(key=CamelProperty.CamelUnitOfWorkProcessSync, value=true, schema=Schema{BOOLEAN}), ConnectHeader(key=CamelProperty.CamelBatchComplete, value=true, schema=Schema{BOOLEAN}), ConnectHeader(key=CamelProperty.CamelBatchIndex, value=0, schema=Schema{INT32}), ConnectHeader(key=CamelProperty.CamelToEndpoint, value=direct://end?pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true&pollingConsumerQueueSize=1000, schema=Schema{STRING})])} (org.apache.camel.kafkaconnector.CamelSourceTask)
   [2021-03-29 15:03:33,080] INFO Scheduled task completed on: sftp://sftp.sftp:22/data/?binary=false&download=false&include=.*.txt&noop=true&password=xxxxxx (org.apache.camel.support.ScheduledPollConsumer)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] PLCyr commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
PLCyr commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-812127643


   I did a test with camel-kafka-connector 0.7.3 and the behavior is good. The idempotent topic is re-read from offset 0 after a connector crash (pod crash) or connector configuration change. These lines in the log show the difference
   
   [2021-04-01 19:20:00,722] INFO [Consumer clientId=consumer-473c5e5a-e3e9-438e-8f92-8ae811cd2214-5, groupId=473c5e5a-e3e9-438e-8f92-8ae811cd2214] **Seeking to EARLIEST offset** of partition SFTP-Persistance-Topic-0 (org.apache.kafka.clients.consumer.internals.SubscriptionState)
   [2021-04-01 19:20:00,729] INFO [Consumer clientId=consumer-473c5e5a-e3e9-438e-8f92-8ae811cd2214-5, groupId=473c5e5a-e3e9-438e-8f92-8ae811cd2214] Resetting offset for partition SFTP-Persistance-Topic-0 to position FetchPosition{**offset=0**, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[confluent-kafka-cp-kafka-2.confluent-kafka-cp-kafka-headless.kafka:9092 (id: 2 rack: null)], epoch=0}}. (org.apache.kafka.clients.consumer.internals.SubscriptionState)
   [2021-04-01 19:20:00,836] INFO Cache OK (org.apache.camel.processor.idempotent.kafka.KafkaIdempotentRepository)
   
   It's a good thing this fix will be include in the incoming camel-kafka-connector 0.9.0 !
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] oscerd commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-809936873


   Can you please try adding the camel.source.endpoint.initialDelay to a bigger value? like for example 30000 or 40000? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] PLCyr commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
PLCyr commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-809569901


   Sorry, I had misread "camel.source.endpoint.idempotent" for "camel.idempotency.enabled".
   Just tested  "camel.source.endpoint.idempotent" to false and noop = true, same behavior
   
   The log still show a hint that Memory Idempotent Repo is used
   [2021-03-29 17:27:47,943] INFO Setting initial properties in Camel context: [[connector.class=org.apache.camel.kafkaconnector.sftp.CamelSftpSourceConnector, camel.source.endpoint.runLoggingLevel=INFO, camel.idempotency.enabled=true, transforms=RemoteTransformer, camel.idempotency.expression.header=CamelFileNameConsumed, camel.source.endpoint.download=false, camel.source.endpoint.noop=true, key.converter=org.apache.kafka.connect.storage.StringConverter, camel.source.contentLogLevel=INFO, transforms.RemoteTransformer.type=org.apache.camel.kafkaconnector.sftp.transformers.SftpRemoteFileTransforms, camel.source.endpoint.include=.*.txt, camel.source.endpoint.password=pass, name=sftp-connector, camel.source.path.directoryName=data/, camel.idempotency.kafka.topic=SFTP-Persistance-Topic, camel.source.endpoint.username=foo, camel.source.component=sftp, camel.source.path.host=sftp.sftp, camel.idempotency.repository.type=kafka, camel.idempotency.kafka.bootstrap.servers=confluent-kafka-cp-kaf
 ka:9092, value.converter=org.apache.kafka.connect.converters.ByteArrayConverter, camel.idempotency.expression.type=header, camel.source.endpoint.recursive=true, topics=SFTP-Topic, camel.source.path.port=22, camel.source.endpoint.idempotent=false, task.class=org.apache.camel.kafkaconnector.sftp.CamelSftpSourceTask, camel.source.endpoint.binary=false]] (org.apache.camel.kafkaconnector.utils.CamelKafkaConnectMain)
   [2021-03-29 17:27:47,944] INFO [Producer clientId=connector-producer-sftp-connector-0] Cluster ID: fFQtkPXrT92v270crRLidQ (org.apache.kafka.clients.Metadata)
   [2021-03-29 17:27:47,961] INFO Creating Camel route from(sftp:sftp.sftp:22/data/?binary=false&download=false&idempotent=false&include=.*.txt&noop=true&password=RAW(pass)&recursive=true&runLoggingLevel=INFO&username=RAW(foo)) (org.apache.camel.kafkaconnector.utils.CamelKafkaConnectMain)
   [2021-03-29 17:27:47,963] INFO idempotentConsumer(header(expressionHeader), MemoryIdempotentRepository.memoryIdempotentRepository(100)).to(direct:end?pollingConsumerQueueSize=1000&pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true) (org.apache.camel.kafkaconnector.utils.CamelKafkaConnectMain)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] PLCyr commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
PLCyr commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-810285644


   Yes, got the same behavior ... 
   The most interesting line in the log I think is this one 
   
   [2021-03-30 13:29:56,087] INFO [Consumer clientId=consumer-30bd6ecd-fd2f-4f07-b02f-6eb94c44a7de-6, groupId=30bd6ecd-fd2f-4f07-b02f-6eb94c44a7de] Resetting offset for partition SFTP-Persistance-Topic-0 to position FetchPosition{**offset=10**, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[confluent-kafka-cp-kafka-1.confluent-kafka-cp-kafka-headless.kafka:9092 (id: 1 rack: null)], epoch=2}}. (org.apache.kafka.clients.consumer.internals.SubscriptionState)
   
   Shouldn't the offset be resetted to 0 since it's a new consumer group id each time the connector restart ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] valdar commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
valdar commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-811938937


   @PLCyr it turned out possibly the solution is much simpler than envisioned: the [fix](https://github.com/apache/camel/pull/5066) we are talking about is not in `camel 3.8.0` that `camel-kafka-connecotr 0.8.0` is using. We are in the process of releasing `camel-kafka-connector 0.9.0` based on `camel 3.9.0` that should contain the fix.
   
   It should be ready in a couple of days. I will ping you and you can try if the new version solves your problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] oscerd commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-810323111


   Now I noticed this line in your description
   
   "I was expecting the idempotency feature to re-consume the idempotency Kafka topic at offset 0 so it could have some kind of crash resilience."
   
   Basically the idempotency we are using is a camel concept. When you stop a consumer (camel consumer) with a kafka idempotent repository, the offset will be saved. When you restart the connector it will restart from that offset. This is mainly because the sftp component and the others works with just one consumer. In camel to support mutliple consumers we need to do some work on single component and it's not feasible on all the components, another possibility is to use some kind of lock mechanism with camel-master component.
   
   Sorry I didn't notice your sentence there. The behavior is normal from a Camel perspective.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] valdar commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
valdar commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-811807955


   @PLCyr what you are experiencing is not the intended behavior.
   We have reviewed a recent fix on that part and done some additional testing and the problem is we can not reproduce your issue yet.
   
   Can you give us some details on you environment? kafka version, how is deployed on k8s, how is the pod defined and how you restart it? 
   
   In addition a log with `org.apache.camel.processor.idempotent.kafka` logging level set to `TRACE` would help as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] oscerd commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-812376454


   No, it's not possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] oscerd commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-812129167


   Yeah, this was something we didn't consider in the beginning. We didn't consider the LTS.
   
   You mean combining multiple headers in an expression?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] valdar closed issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
valdar closed issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] PLCyr commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
PLCyr commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-810343946


   That was my main question ! 
   
   From my perspective, if the connector could re-read all event in the idempotence topic after a crash, it'll be more resilient. If not, in event of a crash we are exposed to duplicate events if we use noop=true without renaming the files in the SFTP server.
   
   You can confirm that Camel Kafka Connector are not expected to have this kind of resilience ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] valdar edited a comment on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
valdar edited a comment on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-811807955


   @PLCyr what you are experiencing is not the intended behavior.
   We have reviewed a recent [fix](https://github.com/apache/camel/pull/5066) on an [issue](https://issues.apache.org/jira/browse/CAMEL-16181) that part and done some additional testing and unfortunately we can not reproduce your issue yet.
   
   Can you give us some details on you environment? kafka version, how is deployed on k8s, how is the pod defined and how you restart it? 
   
   In addition a log with `org.apache.camel.processor.idempotent.kafka` logging level set to `TRACE` would help as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] PLCyr commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
PLCyr commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-812128669


   Another related question, is the "camel.idempotency.expression.header" can support a composite header expression ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] oscerd commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-809560969


   You need to set it to false. In that way the kafka idempotency Will be used. Otherwise the component in memory idempotent repo will be used. Just set the option I said to false and it should work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] PLCyr commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
PLCyr commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-809551702


   I do want to use Kafka idempotence, so I test to set noop=false.
   If I retry to kill the pod, the connector isn't rebuilding is idempotence from the idempoten topic in Kafka.
   It could be the expected behavior or a bug, just want someone to confirm it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] valdar edited a comment on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
valdar edited a comment on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-811807955


   @PLCyr what you are experiencing is not the intended behavior.
   We have reviewed a recent fix on that part and done some additional testing and unfortunately we can not reproduce your issue yet.
   
   Can you give us some details on you environment? kafka version, how is deployed on k8s, how is the pod defined and how you restart it? 
   
   In addition a log with `org.apache.camel.processor.idempotent.kafka` logging level set to `TRACE` would help as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] PLCyr commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
PLCyr commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-812130091


   Yes multiple header, like combining the CamelHeader.CamelFileAbsolutePath and CamelHeader.CamelFileLastModified for the idempotent expression check


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] oscerd commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-809538109


   Try to set "camel.source.endpoint.idempotent" to false, because the noop = true implies:
   
   If true, the file is not moved or deleted in any way. This option is good for readonly data, or for ETL type requirements. If noop=true, Camel will set idempotent=true as well, to avoid consuming the same files over and over again.
   
   So the memory idempotent take the precedence. Leave the configuration as is and add  "camel.source.endpoint.idempotent" to false


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] valdar commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
valdar commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-812443765


   @PLCyr I am closing this issue. Feel free to open another one to discuss the "combine header expressions" feature if you like.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] valdar commented on issue #1128: Connector Kafka Idempotency Crash Resilience

Posted by GitBox <gi...@apache.org>.
valdar commented on issue #1128:
URL: https://github.com/apache/camel-kafka-connector/issues/1128#issuecomment-811969286


   @PLCyr actually the fix should be included already in `camel-kafka-connecotr 0.7.3` (it my seems counter intuitive but due to how camel version are intended to be maintained makes sense) sftp connector is here: https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-sftp-kafka-connector/0.7.3/ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org