You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Sagar Sumit (Jira)" <ji...@apache.org> on 2022/07/29 06:56:00 UTC
[jira] [Updated] (HUDI-3113) Kafka Connect create Multiple Embedded Timeline Services
[ https://issues.apache.org/jira/browse/HUDI-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sagar Sumit updated HUDI-3113:
------------------------------
Fix Version/s: 0.13.0
(was: 0.12.0)
> Kafka Connect create Multiple Embedded Timeline Services
> --------------------------------------------------------
>
> Key: HUDI-3113
> URL: https://issues.apache.org/jira/browse/HUDI-3113
> Project: Apache Hudi
> Issue Type: Improvement
> Components: kafka-connect
> Reporter: cdmikechen
> Assignee: Ethan Guo
> Priority: Blocker
> Fix For: 0.13.0
>
>
> After Kafka Connect started, I've found that hudi will create not one Embedded Timeline Service.
> {code}
> [2021-12-28 07:52:57,154] INFO Starting Timeline service !! (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> [2021-12-28 07:52:57,155] WARN Unable to find driver bind address from spark config (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> [2021-12-28 07:52:57,169] INFO Creating View Manager with storage type :MEMORY (org.apache.hudi.common.table.view.FileSystemViewManager)
> [2021-12-28 07:52:57,170] INFO Creating in-memory based Table View (org.apache.hudi.common.table.view.FileSystemViewManager)
> [2021-12-28 07:52:57,184] INFO Logging initialized @27658ms to org.apache.hudi.org.eclipse.jetty.util.log.Slf4jLog (org.apache.hudi.org.eclipse.jetty.util.log)
> [2021-12-28 07:52:57,502] INFO
> __ __ _
> / /____ _ _ __ ____ _ / /(_)____
> __ / // __ `/| | / // __ `// // // __ \
> / /_/ // /_/ / | |/ // /_/ // // // / / /
> \____/ \__,_/ |___/ \__,_//_//_//_/ /_/
> https://javalin.io/documentation
> (io.javalin.Javalin)
> [2021-12-28 07:52:57,504] INFO Starting Javalin ... (io.javalin.Javalin)
> [2021-12-28 07:52:57,650] INFO Listening on http://localhost:43691/ (io.javalin.Javalin)
> [2021-12-28 07:52:57,650] INFO Javalin started in 151ms \o/ (io.javalin.Javalin)
> [2021-12-28 07:52:57,650] INFO Starting Timeline server on port :43691 (org.apache.hudi.timeline.service.TimelineService)
> [2021-12-28 07:52:57,650] INFO Started embedded timeline server at 172.17.0.7:43691 (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> [2021-12-28 07:52:57,661] INFO Start Transaction Coordinator for topic hudi-test-topic partition 0 (org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
> [2021-12-28 07:52:57,690] INFO Loaded instants upto : Option\{val=[==>20211228075022280__commit__INFLIGHT]} (org.apache.hudi.common.table.timeline.HoodieActiveTimeline)
> [2021-12-28 07:52:57,822] INFO Retrieved Raw Kafka offsets from Hudi Commit File 0=100 (org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
> [2021-12-28 07:52:57,823] INFO Initialized the kafka offset commits \{0=100} (org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
> [2021-12-28 07:52:57,858] INFO The value of hoodie.datasource.write.keygenerator.type is empty, using SIMPLE (org.apache.hudi.keygen.factory.HoodieAvroKeyGeneratorFactory)
> [2021-12-28 07:52:57,928] INFO AdminClientConfig values:
> bootstrap.servers = [10.3.101.60:9092]
> client.dns.lookup = use_all_dns_ips
> client.id =
> connections.max.idle.ms = 300000
> default.api.timeout.ms = 60000
> metadata.max.age.ms = 300000
> metric.reporters = []
> metrics.num.samples = 2
> metrics.recording.level = INFO
> metrics.sample.window.ms = 30000
> receive.buffer.bytes = 65536
> reconnect.backoff.max.ms = 1000
> reconnect.backoff.ms = 50
> request.timeout.ms = 30000
> retries = 2147483647
> retry.backoff.ms = 100
> sasl.client.callback.handler.class = null
> sasl.jaas.config = null
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.min.time.before.relogin = 60000
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> sasl.kerberos.ticket.renew.window.factor = 0.8
> sasl.login.callback.handler.class = null
> sasl.login.class = null
> sasl.login.refresh.buffer.seconds = 300
> sasl.login.refresh.min.period.seconds = 60
> sasl.login.refresh.window.factor = 0.8
> sasl.login.refresh.window.jitter = 0.05
> sasl.mechanism = GSSAPI
> security.protocol = PLAINTEXT
> security.providers = null
> send.buffer.bytes = 131072
> socket.connection.setup.timeout.max.ms = 127000
> socket.connection.setup.timeout.ms = 10000
> ssl.cipher.suites = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
> ssl.endpoint.identification.algorithm = https
> ssl.engine.factory.class = null
> ssl.key.password = null
> ssl.keymanager.algorithm = SunX509
> ssl.keystore.certificate.chain = null
> ssl.keystore.key = null
> ssl.keystore.location = null
> ssl.keystore.password = null
> ssl.keystore.type = JKS
> ssl.protocol = TLSv1.3
> ssl.provider = null
> ssl.secure.random.implementation = null
> ssl.trustmanager.algorithm = PKIX
> ssl.truststore.certificates = null
> ssl.truststore.location = null
> ssl.truststore.password = null
> ssl.truststore.type = JKS
> (org.apache.kafka.clients.admin.AdminClientConfig)
> [2021-12-28 07:52:57,930] INFO Kafka version: 6.1.1-ccs (org.apache.kafka.common.utils.AppInfoParser)
> [2021-12-28 07:52:57,930] INFO Kafka commitId: c209f70c6c2e52ae (org.apache.kafka.common.utils.AppInfoParser)
> [2021-12-28 07:52:57,930] INFO Kafka startTimeMs: 1640677977929 (org.apache.kafka.common.utils.AppInfoParser)
> [2021-12-28 07:52:57,937] INFO Starting Timeline service !! (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> [2021-12-28 07:52:57,938] WARN Unable to find driver bind address from spark config (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> [2021-12-28 07:52:57,938] INFO Creating View Manager with storage type :MEMORY (org.apache.hudi.common.table.view.FileSystemViewManager)
> [2021-12-28 07:52:57,938] INFO Creating in-memory based Table View (org.apache.hudi.common.table.view.FileSystemViewManager)
> [2021-12-28 07:52:57,943] INFO Latest number of partitions for topic hudi-test-topic is 1 (org.apache.hudi.connect.utils.KafkaConnectUtils)
> [2021-12-28 07:52:57,947] INFO
> __ __ _
> / /____ _ _ __ ____ _ / /(_)____
> __ / // __ `/| | / // __ `// // // __ \
> / /_/ // /_/ / | |/ // /_/ // // // / / /
> \____/ \__,_/ |___/ \__,_//_//_//_/ /_/
> https://javalin.io/documentation
> (io.javalin.Javalin)
> [2021-12-28 07:52:57,947] INFO Starting Javalin ... (io.javalin.Javalin)
> [2021-12-28 07:52:57,959] INFO Loading HoodieTableMetaClient from hdfs://hdp-syzh-cluster/hive/warehouse/default.db/hudi_test_topic (org.apache.hudi.common.table.HoodieTableMetaClient)
> [2021-12-28 07:52:57,960] INFO Listening on http://localhost:41798/ (io.javalin.Javalin)
> [2021-12-28 07:52:57,960] INFO Javalin started in 14ms \o/ (io.javalin.Javalin)
> [2021-12-28 07:52:57,960] INFO Starting Timeline server on port :41798 (org.apache.hudi.timeline.service.TimelineService)
> [2021-12-28 07:52:57,962] INFO Started embedded timeline server at 172.17.0.7:41798 (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> {code}
>
> I think a Kafka Connect should only start one Embedded Timeline Service in the current environment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)