You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by John Casey via user <us...@beam.apache.org> on 2022/11/01 14:12:33 UTC
Re: SSL issue: Kafka Avro write with Schema Registry (GCP)
This issue indicates that your certificates don't have a path between the
worker and the schema registry. You need to make sure your certs know that
certification path, though I don't actually know how to ensure that.
On Mon, Oct 31, 2022 at 6:13 PM Ahmet Altay <al...@google.com> wrote:
> (moving this to the user list, dev list to bcc.)
>
> Adding relevant people: @John Casey <jo...@google.com>.
>
> (Keshav, for Dataflow issues you could also reach out to Dataflow support:
> https://cloud.google.com/dataflow/docs/support)
>
> On Mon, Oct 31, 2022 at 1:23 PM Chennakeshavlu Maddela <
> Chennakeshavlu.Maddela@davita.com> wrote:
>
>> Hi Team,
>>
>>
>>
>> We are setting up avro write to a kafka topic with confluent schema
>> registry on SSL, its throwing (below) error.
>>
>>
>>
>> We are using SASL_SSL with PEM certificate for connecting Kafka broker,
>> which is working fine with non-avro kafka topics. Can you please help us
>> with configuring SSL for schema registry? (we are using dataflow runner)
>>
>>
>>
>> Please let me know if you need more details.
>>
>>
>>
>> Thank you,
>>
>> Keshav
>>
>>
>>
>> *Exception:*
>>
>> Failed to send HTTP request to endpoint:
>> https://confluent-schemaregistry-xxxx.com/subjects/topic-value?deleted=false
>>
>>
>>
>> javax.net.ssl.SSLHandshakeException: PKIX path building failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>> valid certification path to requested target
>>
>> at
>> java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
>>
>> at
>> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:350)
>>
>> at
>> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:293)
>>
>> at
>> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:288)
>>
>> at
>> java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1356)
>>
>> at
>> java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1231)
>>
>> at
>> java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1174)
>>
>> at
>> java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
>>
>> at
>> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
>>
>> at
>> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
>>
>> at
>> java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
>>
>> at
>> java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
>>
>> at
>> java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1408)
>>
>> at
>> java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1314)
>>
>> at
>> java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
>>
>> at
>> java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
>>
>> at
>> java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
>>
>> at
>> java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
>>
>> at
>> java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367)
>>
>> at
>> java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342)
>>
>> at
>> java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246)
>>
>> at
>> io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:199)
>>
>> at
>> io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:256)
>>
>> at
>> io.confluent.kafka.schemaregistry.client.rest.RestService.lookUpSubjectVersion(RestService.java:323)
>>
>> at
>> io.confluent.kafka.schemaregistry.client.rest.RestService.lookUpSubjectVersion(RestService.java:311)
>>
>> at
>> io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getIdFromRegistry(CachedSchemaRegistryClient.java:191)
>>
>> at
>> io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getId(CachedSchemaRegistryClient.java:323)
>>
>> at
>> io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:73)
>>
>> at
>> io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:53)
>>
>> at
>> org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
>>
>> at
>> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:952)
>>
>> at
>> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:912)
>>
>> at
>> com.davita.cwow.pmt.transformations.PmtAvroKafkaWriter$KafkaWriteEvaluationFn.processElement(PmtAvroKafkaWriter.java:86)
>>
>> at
>> com.davita.cwow.pmt.transformations.PmtAvroKafkaWriter$KafkaWriteEvaluationFn$DoFnInvoker.invokeProcessElement(Unknown
>> Source)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:188)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:285)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:275)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:85)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:423)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:411)
>>
>> at
>> com.davita.cwow.pmt.transformations.PmtAvroKafkaWriter$EvaluationAvroToProducerRecordFn.processElement(PmtAvroKafkaWriter.java:67)
>>
>> at
>> com.davita.cwow.pmt.transformations.PmtAvroKafkaWriter$EvaluationAvroToProducerRecordFn$DoFnInvoker.invokeProcessElement(Unknown
>> Source)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:188)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:285)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:275)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:85)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:423)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:411)
>>
>> at
>> com.davita.cwow.pmt.transformations.BatchEvaluationTransform$2.process(BatchEvaluationTransform.java:100)
>>
>> at
>> com.davita.cwow.pmt.transformations.BatchEvaluationTransform$2$DoFnInvoker.invokeProcessElement(Unknown
>> Source)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:188)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:285)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:275)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:85)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:423)
>>
>> at
>> com.davita.cwow.pmt.transformations.PatientProtocolEvaluationResultMutationTransform$ToEntityFn.processElement(PatientProtocolEvaluationResultMutationTransform.java:194)
>>
>> at
>> com.davita.cwow.pmt.transformations.PatientProtocolEvaluationResultMutationTransform$ToEntityFn$DoFnInvoker.invokeProcessElement(Unknown
>> Source)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:185)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:285)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:275)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:85)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:423)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:411)
>>
>> at
>> java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
>>
>> at
>> java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
>>
>> at
>> com.davita.cwow.pmt.transformations.ProtocolEvaluatorTransform$5.process(ProtocolEvaluatorTransform.java:248)
>>
>> at
>> com.davita.cwow.pmt.transformations.ProtocolEvaluatorTransform$5$DoFnInvoker.invokeProcessElement(Unknown
>> Source)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:188)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:285)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:275)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:85)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:423)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:411)
>>
>> at
>> org.apache.beam.sdk.transforms.join.CoGroupByKey$ConstructCoGbkResultFn.processElement(CoGroupByKey.java:192)
>>
>> at
>> org.apache.beam.sdk.transforms.join.CoGroupByKey$ConstructCoGbkResultFn$DoFnInvoker.invokeProcessElement(Unknown
>> Source)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:188)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:185)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:108)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:128)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:56)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:121)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:117)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:218)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:169)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:83)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:420)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:389)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
>>
>> at
>> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
>>
>> at
>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>
>> at java.base/java.lang.Thread.run(Thread.java:834)
>>
>> Caused by: sun.security.validator.ValidatorException: PKIX path building
>> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
>> to find valid certification path to requested target
>>
>> at
>> java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
>>
>> at
>> java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
>>
>> at
>> java.base/sun.security.validator.Validator.validate(Validator.java:264)
>>
>> at
>> java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
>>
>> at
>> java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
>>
>> at
>> java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
>>
>> at
>> java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1340)
>>
>> ... 117 more
>>
>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>> unable to find valid certification path to requested target
>>
>> at
>> java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
>>
>> at
>> java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
>>
>> at
>> java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
>>
>> at
>> java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
>>
>> ... 123 more
>>
>> CONFIDENTIALITY NOTICE: THIS MESSAGE IS CONFIDENTIAL, INTENDED FOR THE
>> NAMED RECIPIENT(S) AND MAY CONTAIN INFORMATION THAT IS (I) PROPRIETARY TO
>> THE SENDER, AND/OR, (II) PRIVILEGED, CONFIDENTIAL, AND/OR OTHERWISE EXEMPT
>> FROM DISCLOSURE UNDER APPLICABLE STATE AND FEDERAL LAW, INCLUDING, BUT NOT
>> LIMITED TO, PRIVACY STANDARDS IMPOSED PURSUANT TO THE FEDERAL HEALTH
>> INSURANCE PORTABILITY AND ACCOUNTABILITY ACT OF 1996 ("HIPAA"). IF YOU ARE
>> NOT THE INTENDED RECIPIENT, OR THE EMPLOYEE OR AGENT RESPONSIBLE FOR
>> DELIVERING THE MESSAGE TO THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED
>> THAT ANY DISSEMINATION, DISTRIBUTION OR COPYING OF THIS COMMUNICATION IS
>> STRICTLY PROHIBITED. IF YOU HAVE RECEIVED THIS TRANSMISSION IN ERROR,
>> PLEASE (I) NOTIFY US IMMEDIATELY BY REPLY E-MAIL OR BY TELEPHONE AT
>> (855.472.9822 <(855)%20472-9822>), (II) REMOVE IT FROM YOUR SYSTEM, AND
>> (III) DESTROY THE ORIGINAL TRANSMISSION AND ITS ATTACHMENTS WITHOUT READING
>> OR SAVING THEM. THANK YOU.
>>
>> -DaVita Inc-
>>
>
Re: SSL issue: Kafka Avro write with Schema Registry (GCP)
Posted by John Casey via user <us...@beam.apache.org>.
Unfortunately, I don't have a working sample for this, apologies
On Tue, Nov 1, 2022 at 1:00 PM MURUGASUNDARAM ANNAMALAI <
MURUGASUNDARAM.ANNAMALAI@davita.com> wrote:
> Yes, Truststore is a JKS file and we are not sure how to inject path based
> files in Google dataflow as how dataflow can identify the path and how the
> truststore JKS is copied across. We are not using JDK default truststore as
> we have our own truststore that we use in microservices that we want to use
> in dataflow also
>
> Is there a sample code you can provide on Dtaflow approach file based
> injections
>
>
>
> One workaround we are trying is creating the truststore file dynamically
> using API from PEM string but that needs lot of code and testing
>
>
>
> *From: *Chennakeshavlu Maddela <Ch...@davita.com>
> *Date: *Tuesday, November 1, 2022 at 12:10 PM
> *To: *John Casey <jo...@google.com>, Ahmet Altay <al...@google.com>
> *Cc: *"user@beam.apache.org" <us...@beam.apache.org>, MURUGASUNDARAM
> ANNAMALAI <MU...@davita.com>
> *Subject: *Re: SSL issue: Kafka Avro write with Schema Registry (GCP)
>
>
>
> Thank you for quick response. We are currently PEM cert to connect kafka
> broker and working fine, but same config is not working with schema
> registry (in Apache beam).
>
>
>
> We have spring boot micro service docker container as producer with key
> store injected, its working with schema registry. Not sure how to implement
> same with apache beam to inject/provide for dataflow workers.
>
>
>
> Adding my kafka teammate in cc.
>
>
>
>
>
> Thank you,
>
> Keshav
>
> *From: *John Casey <jo...@google.com>
> *Date: *Tuesday, November 1, 2022 at 9:12 AM
> *To: *Ahmet Altay <al...@google.com>
> *Cc: *"user@beam.apache.org" <us...@beam.apache.org>, Chennakeshavlu
> Maddela <Ch...@davita.com>
> *Subject: *Re: SSL issue: Kafka Avro write with Schema Registry (GCP)
>
>
>
> WARNING: This email originated outside of DaVita. Even if this looks like
> a DaVita email, it is not.
>
> DO NOT provide your username, password, or any other personal information
> in response to this or any other email.
>
> DAVITA WILL NEVER ask you for your username or password via email.
>
> DO NOT CLICK links or attachments unless you are positive the content is
> safe.
> IF IN DOUBT about the safety of this message, use the Report Phishing
> button.
>
> This issue indicates that your certificates don't have a path between the
> worker and the schema registry. You need to make sure your certs know that
> certification path, though I don't actually know how to ensure that.
>
>
>
> On Mon, Oct 31, 2022 at 6:13 PM Ahmet Altay <al...@google.com> wrote:
>
> (moving this to the user list, dev list to bcc.)
>
>
>
> Adding relevant people: @John Casey <jo...@google.com>.
>
>
>
> (Keshav, for Dataflow issues you could also reach out to Dataflow support:
> https://cloud.google.com/dataflow/docs/support)
>
>
>
> On Mon, Oct 31, 2022 at 1:23 PM Chennakeshavlu Maddela <
> Chennakeshavlu.Maddela@davita.com> wrote:
>
> Hi Team,
>
>
>
> We are setting up avro write to a kafka topic with confluent schema
> registry on SSL, its throwing (below) error.
>
>
>
> We are using SASL_SSL with PEM certificate for connecting Kafka broker,
> which is working fine with non-avro kafka topics. Can you please help us
> with configuring SSL for schema registry? (we are using dataflow runner)
>
>
>
> Please let me know if you need more details.
>
>
>
> Thank you,
>
> Keshav
>
>
>
> *Exception:*
>
> Failed to send HTTP request to endpoint:
> https://confluent-schemaregistry-xxxx.com/subjects/topic-value?deleted=false
>
>
>
> javax.net.ssl.SSLHandshakeException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
>
> at
> java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
>
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:350)
>
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:293)
>
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:288)
>
> at
> java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1356)
>
> at
> java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1231)
>
> at
> java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1174)
>
> at
> java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
>
> at
> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
>
> at
> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
>
> at
> java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
>
> at
> java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
>
> at
> java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1408)
>
> at
> java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1314)
>
> at
> java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
>
> at
> java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
>
> at
> java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
>
> at
> java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
>
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367)
>
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342)
>
> at
> java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246)
>
> at
> io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:199)
>
> at
> io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:256)
>
> at
> io.confluent.kafka.schemaregistry.client.rest.RestService.lookUpSubjectVersion(RestService.java:323)
>
> at
> io.confluent.kafka.schemaregistry.client.rest.RestService.lookUpSubjectVersion(RestService.java:311)
>
> at
> io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getIdFromRegistry(CachedSchemaRegistryClient.java:191)
>
> at
> io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getId(CachedSchemaRegistryClient.java:323)
>
> at
> io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:73)
>
> at
> io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:53)
>
> at
> org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
>
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:952)
>
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:912)
>
> at
> com.davita.cwow.pmt.transformations.PmtAvroKafkaWriter$KafkaWriteEvaluationFn.processElement(PmtAvroKafkaWriter.java:86)
>
> at
> com.davita.cwow.pmt.transformations.PmtAvroKafkaWriter$KafkaWriteEvaluationFn$DoFnInvoker.invokeProcessElement(Unknown
> Source)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:188)
>
> at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>
> at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:285)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:275)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:85)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:423)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:411)
>
> at
> com.davita.cwow.pmt.transformations.PmtAvroKafkaWriter$EvaluationAvroToProducerRecordFn.processElement(PmtAvroKafkaWriter.java:67)
>
> at
> com.davita.cwow.pmt.transformations.PmtAvroKafkaWriter$EvaluationAvroToProducerRecordFn$DoFnInvoker.invokeProcessElement(Unknown
> Source)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:188)
>
> at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>
> at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:285)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:275)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:85)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:423)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:411)
>
> at
> com.davita.cwow.pmt.transformations.BatchEvaluationTransform$2.process(BatchEvaluationTransform.java:100)
>
> at
> com.davita.cwow.pmt.transformations.BatchEvaluationTransform$2$DoFnInvoker.invokeProcessElement(Unknown
> Source)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:188)
>
> at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>
> at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:285)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:275)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:85)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:423)
>
> at
> com.davita.cwow.pmt.transformations.PatientProtocolEvaluationResultMutationTransform$ToEntityFn.processElement(PatientProtocolEvaluationResultMutationTransform.java:194)
>
> at
> com.davita.cwow.pmt.transformations.PatientProtocolEvaluationResultMutationTransform$ToEntityFn$DoFnInvoker.invokeProcessElement(Unknown
> Source)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:185)
>
> at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>
> at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:285)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:275)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:85)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:423)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:411)
>
> at
> java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
>
> at
> java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
>
> at
> com.davita.cwow.pmt.transformations.ProtocolEvaluatorTransform$5.process(ProtocolEvaluatorTransform.java:248)
>
> at
> com.davita.cwow.pmt.transformations.ProtocolEvaluatorTransform$5$DoFnInvoker.invokeProcessElement(Unknown
> Source)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:188)
>
> at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>
> at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:285)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:275)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:85)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:423)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:411)
>
> at
> org.apache.beam.sdk.transforms.join.CoGroupByKey$ConstructCoGbkResultFn.processElement(CoGroupByKey.java:192)
>
> at
> org.apache.beam.sdk.transforms.join.CoGroupByKey$ConstructCoGbkResultFn$DoFnInvoker.invokeProcessElement(Unknown
> Source)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
>
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:188)
>
> at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:340)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>
> at
> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:185)
>
> at
> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:108)
>
> at
> org.apache.beam.runners.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:128)
>
> at
> org.apache.beam.runners.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:56)
>
> at
> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:121)
>
> at
> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
>
> at
> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:117)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:218)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:169)
>
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:83)
>
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:420)
>
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:389)
>
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314)
>
> at
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
>
> at
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
>
> at
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
>
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:834)
>
> Caused by: sun.security.validator.ValidatorException: PKIX path building
> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
> to find valid certification path to requested target
>
> at
> java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
>
> at
> java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
>
> at
> java.base/sun.security.validator.Validator.validate(Validator.java:264)
>
> at
> java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
>
> at
> java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
>
> at
> java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
>
> at
> java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1340)
>
> ... 117 more
>
> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
> unable to find valid certification path to requested target
>
> at
> java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
>
> at
> java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
>
> at
> java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
>
> at
> java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
>
> ... 123 more
>
> CONFIDENTIALITY NOTICE: THIS MESSAGE IS CONFIDENTIAL, INTENDED FOR THE
> NAMED RECIPIENT(S) AND MAY CONTAIN INFORMATION THAT IS (I) PROPRIETARY TO
> THE SENDER, AND/OR, (II) PRIVILEGED, CONFIDENTIAL, AND/OR OTHERWISE EXEMPT
> FROM DISCLOSURE UNDER APPLICABLE STATE AND FEDERAL LAW, INCLUDING, BUT NOT
> LIMITED TO, PRIVACY STANDARDS IMPOSED PURSUANT TO THE FEDERAL HEALTH
> INSURANCE PORTABILITY AND ACCOUNTABILITY ACT OF 1996 ("HIPAA"). IF YOU ARE
> NOT THE INTENDED RECIPIENT, OR THE EMPLOYEE OR AGENT RESPONSIBLE FOR
> DELIVERING THE MESSAGE TO THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED
> THAT ANY DISSEMINATION, DISTRIBUTION OR COPYING OF THIS COMMUNICATION IS
> STRICTLY PROHIBITED. IF YOU HAVE RECEIVED THIS TRANSMISSION IN ERROR,
> PLEASE (I) NOTIFY US IMMEDIATELY BY REPLY E-MAIL OR BY TELEPHONE AT
> (855.472.9822 <(855)%20472-9822>), (II) REMOVE IT FROM YOUR SYSTEM, AND
> (III) DESTROY THE ORIGINAL TRANSMISSION AND ITS ATTACHMENTS WITHOUT READING
> OR SAVING THEM. THANK YOU.
>
> -DaVita Inc-
>
> CONFIDENTIALITY NOTICE: THIS MESSAGE IS CONFIDENTIAL, INTENDED FOR THE
> NAMED RECIPIENT(S) AND MAY CONTAIN INFORMATION THAT IS (I) PROPRIETARY TO
> THE SENDER, AND/OR, (II) PRIVILEGED, CONFIDENTIAL, AND/OR OTHERWISE EXEMPT
> FROM DISCLOSURE UNDER APPLICABLE STATE AND FEDERAL LAW, INCLUDING, BUT NOT
> LIMITED TO, PRIVACY STANDARDS IMPOSED PURSUANT TO THE FEDERAL HEALTH
> INSURANCE PORTABILITY AND ACCOUNTABILITY ACT OF 1996 ("HIPAA"). IF YOU ARE
> NOT THE INTENDED RECIPIENT, OR THE EMPLOYEE OR AGENT RESPONSIBLE FOR
> DELIVERING THE MESSAGE TO THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED
> THAT ANY DISSEMINATION, DISTRIBUTION OR COPYING OF THIS COMMUNICATION IS
> STRICTLY PROHIBITED. IF YOU HAVE RECEIVED THIS TRANSMISSION IN ERROR,
> PLEASE (I) NOTIFY US IMMEDIATELY BY REPLY E-MAIL OR BY TELEPHONE AT
> (855.472.9822 <(855)%20472-9822>), (II) REMOVE IT FROM YOUR SYSTEM, AND
> (III) DESTROY THE ORIGINAL TRANSMISSION AND ITS ATTACHMENTS WITHOUT READING
> OR SAVING THEM. THANK YOU.
>
> -DaVita Inc-
>