You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by shzshi <sh...@discover.com> on 2019/01/07 10:42:48 UTC
Re: Spark Kinesis Connector SSL issue
Hi team,
please help , we are kind of blocked here.
Cheers,
Shashi
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
RE: RE: Re: Spark Kinesis Connector SSL issue
Posted by Shashikant Bangera <sh...@discover.com>.
Hi,
The issue is that the KCL inside the Spark Streaming connector does not provide a way to pass KCL configuration in, which means we can’t supply configuration to disable SSL cert checks. In a typical (non-Spark Streaming) KCL app, we can instantiate the KCL via:
Worker worker = new Worker.Builder()
.config(kclConfig) , //we can specify to disable SSL certs here.
.kinesisClient(kinesisClient)
.recordProcessorFactory(processorFactory)
.build();
However with the Kinesis Spark Streaming consumer, we do not have the ability to set this. Our only interface with Kinesis is:
KinesisUtils.createStream(context, appName, streamName, serviceEndpoint, regionName,
InitialPositionInStream.TRIM_HORIZON, checkpoint, StorageLevel.MEMORY_AND_DISK_2())
and so we can’t pass config anywhere for the KCL to read.
Regards,
Ben
Shashikant Bangera | DevOps Engineer
Payment Services DevOps Engineering
Email: shashikantbangera@discover.com<ma...@discover.com>
Group email: eppdevops@discover.com<ma...@discover.com>
Tel: +44 (0)
Mob: +44 (0) 7440783885
From: Ben Watson
Sent: 07 January 2019 12:32
To: yeikel valdes <em...@yeikel.com>; Shashikant Bangera <sh...@discover.com>
Cc: user@spark.apache.org
Subject: RE: [EXTERNAL] RE: Re: Spark Kinesis Connector SSL issue
Hi,
The issue is that the KCL inside the Spark Streaming connector does not provide a way to pass KCL configuration in, which means we can’t supply configuration to disable SSL cert checks. In a typical (non-Spark Streaming) KCL app, we can instantiate the KCL via:
Worker worker = new Worker.Builder()
.config(kclConfig) , //we can specify to disable SSL certs here.
.kinesisClient(kinesisClient)
.recordProcessorFactory(processorFactory)
.build();
However with the Kinesis Spark Streaming consumer, we do not have the ability to set this. Our only interface with Kinesis is:
KinesisUtils.createStream(context, appName, streamName, serviceEndpoint, regionName,
InitialPositionInStream.TRIM_HORIZON, checkpoint, StorageLevel.MEMORY_AND_DISK_2())
and so we can’t pass config anywhere for the KCL to read.
Regards,
Ben
From: yeikel valdes [mailto:email@yeikel.com]
Sent: 07 January 2019 12:21
To: Shashikant Bangera <sh...@discover.com>
Cc: user@spark.apache.org; Ben Watson <be...@discover.com>
Subject: [EXTERNAL] RE: Re: Spark Kinesis Connector SSL issue
CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected emails.
Any chance you can share a minimum example to replicate the issue?
---- On Mon, 07 Jan 2019 04:17:44 -0800 shashikantbangera@discover.com wrote ----
Hi Valdes,
Thank you for your response, to answer to your question. yes I can
@ben : correct me if I am wrong.
Cheers,
Shashi
Shashikant Bangera | DevOps Engineer
Payment Services DevOps Engineering
Email: shashikantbangera@discover.com<ma...@discover.com>
Group email: eppdevops@discover.com<ma...@discover.com>
Tel: +44 (0)
Mob: +44 (0) 7440783885
From: yeikel valdes [mailto:email@yeikel.com<ma...@yeikel.com>]
Sent: 07 January 2019 12:15
To: Shashikant Bangera <sh...@discover.com>>
Cc: user@spark.apache.org<ma...@spark.apache.org>
Subject: [EXTERNAL] Re: Spark Kinesis Connector SSL issue
CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected emails.
Can you call this service with regular code(No Spark)?
---- On Mon, 07 Jan 2019 02:42:48 -0800 shashikantbangera@discover.com<ma...@discover.com> wrote ----
Hi team,
please help , we are kind of blocked here.
Cheers,
Shashi
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Duser-2Dlist.1001560.n3.nabble.com_&d=DwMFaQ&c=aGHIJeafFG1OvdAmHquqHk8LwduOIVr390d9RxG_Ii0&r=5iagzyAhXr_lW4Qru-HH2EiLU8DXPfEhqvrYe8cl44A&m=bruN0LP0xvrh5SboalWPOu1Iws5Ia1RONxxCEzzYQ9g&s=h1CVgmdCzA6yS9NqRH9XMVXrD4aeTZsKJMxauzIFDq4&e=>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
RE: Re: Spark Kinesis Connector SSL issue
Posted by yeikel valdes <em...@yeikel.com>.
Any chance you can share a minimum example to replicate the issue?
---- On Mon, 07 Jan 2019 04:17:44 -0800 shashikantbangera@discover.com wrote ----
Hi Valdes,
Thank you for your response, to answer to your question. yes I can
@ben : correct me if I am wrong.
Cheers,
Shashi
Shashikant Bangera | DevOps Engineer
Payment Services DevOps Engineering
Email: shashikantbangera@discover.com
Group email: eppdevops@discover.com
Tel: +44 (0)
Mob: +44 (0) 7440783885
From: yeikel valdes [mailto:email@yeikel.com]
Sent: 07 January 2019 12:15
To: Shashikant Bangera <sh...@discover.com>
Cc: user@spark.apache.org
Subject: [EXTERNAL] Re: Spark Kinesis Connector SSL issue
CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected emails.
Can you call this service with regular code(No Spark)?
---- On Mon, 07 Jan 2019 02:42:48 -0800 shashikantbangera@discover.com wrote ----
Hi team,
please help , we are kind of blocked here.
Cheers,
Shashi
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
RE: Re: Spark Kinesis Connector SSL issue
Posted by Shashikant Bangera <sh...@discover.com>.
Hi Valdes,
Thank you for your response, to answer to your question. yes I can
@ben : correct me if I am wrong.
Cheers,
Shashi
Shashikant Bangera | DevOps Engineer
Payment Services DevOps Engineering
Email: shashikantbangera@discover.com<ma...@discover.com>
Group email: eppdevops@discover.com<ma...@discover.com>
Tel: +44 (0)
Mob: +44 (0) 7440783885
From: yeikel valdes [mailto:email@yeikel.com]
Sent: 07 January 2019 12:15
To: Shashikant Bangera <sh...@discover.com>
Cc: user@spark.apache.org
Subject: [EXTERNAL] Re: Spark Kinesis Connector SSL issue
CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected emails.
Can you call this service with regular code(No Spark)?
---- On Mon, 07 Jan 2019 02:42:48 -0800 shashikantbangera@discover.com wrote ----
Hi team,
please help , we are kind of blocked here.
Cheers,
Shashi
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Duser-2Dlist.1001560.n3.nabble.com_&d=DwMFaQ&c=aGHIJeafFG1OvdAmHquqHk8LwduOIVr390d9RxG_Ii0&r=5iagzyAhXr_lW4Qru-HH2EiLU8DXPfEhqvrYe8cl44A&m=bruN0LP0xvrh5SboalWPOu1Iws5Ia1RONxxCEzzYQ9g&s=h1CVgmdCzA6yS9NqRH9XMVXrD4aeTZsKJMxauzIFDq4&e=>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
Re: Spark Kinesis Connector SSL issue
Posted by yeikel valdes <em...@yeikel.com>.
Can you call this service with regular code(No Spark)?
---- On Mon, 07 Jan 2019 02:42:48 -0800 shashikantbangera@discover.com wrote ----
Hi team,
please help , we are kind of blocked here.
Cheers,
Shashi
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org