You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by shzshi <sh...@discover.com> on 2019/01/07 10:42:48 UTC

Re: Spark Kinesis Connector SSL issue

Hi team, 

please help , we are kind of blocked here. 

Cheers,
Shashi



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


RE: RE: Re: Spark Kinesis Connector SSL issue

Posted by Shashikant Bangera <sh...@discover.com>.
Hi,

The issue is that the KCL inside the Spark Streaming connector does not provide a way to pass KCL configuration in, which means we can’t supply configuration to disable SSL cert checks. In a typical (non-Spark Streaming) KCL app, we can instantiate the KCL via:

       Worker worker = new Worker.Builder()
                .config(kclConfig) , //we can specify to disable SSL certs here.
                .kinesisClient(kinesisClient)
                .recordProcessorFactory(processorFactory)
                .build();

However with the Kinesis Spark Streaming consumer, we do not have the ability to set this. Our only interface with Kinesis is:

KinesisUtils.createStream(context, appName, streamName, serviceEndpoint, regionName,
        InitialPositionInStream.TRIM_HORIZON, checkpoint, StorageLevel.MEMORY_AND_DISK_2())

and so we can’t pass config anywhere for the KCL to read.

Regards,

Ben


Shashikant Bangera | DevOps Engineer
Payment Services DevOps Engineering
Email: shashikantbangera@discover.com<ma...@discover.com>
Group email: eppdevops@discover.com<ma...@discover.com>
Tel: +44 (0)
Mob: +44 (0) 7440783885


From: Ben Watson
Sent: 07 January 2019 12:32
To: yeikel valdes <em...@yeikel.com>; Shashikant Bangera <sh...@discover.com>
Cc: user@spark.apache.org
Subject: RE: [EXTERNAL] RE: Re: Spark Kinesis Connector SSL issue

Hi,

The issue is that the KCL inside the Spark Streaming connector does not provide a way to pass KCL configuration in, which means we can’t supply configuration to disable SSL cert checks. In a typical (non-Spark Streaming) KCL app, we can instantiate the KCL via:

       Worker worker = new Worker.Builder()
                .config(kclConfig) , //we can specify to disable SSL certs here.
                .kinesisClient(kinesisClient)
                .recordProcessorFactory(processorFactory)
                .build();

However with the Kinesis Spark Streaming consumer, we do not have the ability to set this. Our only interface with Kinesis is:

KinesisUtils.createStream(context, appName, streamName, serviceEndpoint, regionName,
        InitialPositionInStream.TRIM_HORIZON, checkpoint, StorageLevel.MEMORY_AND_DISK_2())

and so we can’t pass config anywhere for the KCL to read.

Regards,

Ben

From: yeikel valdes [mailto:email@yeikel.com]
Sent: 07 January 2019 12:21
To: Shashikant Bangera <sh...@discover.com>
Cc: user@spark.apache.org; Ben Watson <be...@discover.com>
Subject: [EXTERNAL] RE: Re: Spark Kinesis Connector SSL issue

CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected emails.


Any chance you can share a minimum example to replicate the issue?


---- On Mon, 07 Jan 2019 04:17:44 -0800 shashikantbangera@discover.com wrote ----
Hi Valdes,

Thank you for your response, to answer to your question. yes I can

@ben : correct me if I am wrong.

Cheers,
Shashi

Shashikant Bangera | DevOps Engineer
Payment Services DevOps Engineering
Email: shashikantbangera@discover.com<ma...@discover.com>
Group email: eppdevops@discover.com<ma...@discover.com>
Tel: +44 (0)
Mob: +44 (0) 7440783885


From: yeikel valdes [mailto:email@yeikel.com<ma...@yeikel.com>]
Sent: 07 January 2019 12:15
To: Shashikant Bangera <sh...@discover.com>>
Cc: user@spark.apache.org<ma...@spark.apache.org>
Subject: [EXTERNAL] Re: Spark Kinesis Connector SSL issue

CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected emails.


Can you call this service with regular code(No Spark)?


---- On Mon, 07 Jan 2019 02:42:48 -0800 shashikantbangera@discover.com<ma...@discover.com> wrote ----
Hi team,

please help , we are kind of blocked here.

Cheers,
Shashi



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Duser-2Dlist.1001560.n3.nabble.com_&d=DwMFaQ&c=aGHIJeafFG1OvdAmHquqHk8LwduOIVr390d9RxG_Ii0&r=5iagzyAhXr_lW4Qru-HH2EiLU8DXPfEhqvrYe8cl44A&m=bruN0LP0xvrh5SboalWPOu1Iws5Ia1RONxxCEzzYQ9g&s=h1CVgmdCzA6yS9NqRH9XMVXrD4aeTZsKJMxauzIFDq4&e=>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>



RE: Re: Spark Kinesis Connector SSL issue

Posted by yeikel valdes <em...@yeikel.com>.
Any chance you can share a minimum example to replicate the issue?

---- On Mon, 07 Jan 2019 04:17:44 -0800 shashikantbangera@discover.com wrote ----

Hi Valdes,

 

Thank you for your response, to answer to your question. yes I can

 

@ben : correct me if I am wrong.

 

Cheers,

Shashi

 

Shashikant Bangera | DevOps Engineer

Payment Services DevOps Engineering

Email: shashikantbangera@discover.com

Group email: eppdevops@discover.com

Tel: +44 (0)

Mob: +44 (0) 7440783885

 

 

From: yeikel valdes [mailto:email@yeikel.com] 
Sent: 07 January 2019 12:15
To: Shashikant Bangera <sh...@discover.com>
Cc: user@spark.apache.org
Subject: [EXTERNAL] Re: Spark Kinesis Connector SSL issue

 

CAUTION EXTERNAL EMAIL 
DO NOT open attachments or click on links from unknown senders or unexpected emails.

 

Can you call this service with regular code(No Spark)?

 


---- On Mon, 07 Jan 2019 02:42:48 -0800 shashikantbangera@discover.com wrote ----

Hi team, 

please help , we are kind of blocked here. 

Cheers, 
Shashi 



-- 
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ 

--------------------------------------------------------------------- 
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

 

RE: Re: Spark Kinesis Connector SSL issue

Posted by Shashikant Bangera <sh...@discover.com>.
Hi Valdes,

Thank you for your response, to answer to your question. yes I can

@ben : correct me if I am wrong.

Cheers,
Shashi

Shashikant Bangera | DevOps Engineer
Payment Services DevOps Engineering
Email: shashikantbangera@discover.com<ma...@discover.com>
Group email: eppdevops@discover.com<ma...@discover.com>
Tel: +44 (0)
Mob: +44 (0) 7440783885


From: yeikel valdes [mailto:email@yeikel.com]
Sent: 07 January 2019 12:15
To: Shashikant Bangera <sh...@discover.com>
Cc: user@spark.apache.org
Subject: [EXTERNAL] Re: Spark Kinesis Connector SSL issue

CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected emails.


Can you call this service with regular code(No Spark)?


---- On Mon, 07 Jan 2019 02:42:48 -0800 shashikantbangera@discover.com wrote ----
Hi team,

please help , we are kind of blocked here.

Cheers,
Shashi



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Duser-2Dlist.1001560.n3.nabble.com_&d=DwMFaQ&c=aGHIJeafFG1OvdAmHquqHk8LwduOIVr390d9RxG_Ii0&r=5iagzyAhXr_lW4Qru-HH2EiLU8DXPfEhqvrYe8cl44A&m=bruN0LP0xvrh5SboalWPOu1Iws5Ia1RONxxCEzzYQ9g&s=h1CVgmdCzA6yS9NqRH9XMVXrD4aeTZsKJMxauzIFDq4&e=>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>


Re: Spark Kinesis Connector SSL issue

Posted by yeikel valdes <em...@yeikel.com>.
Can you call this service with regular code(No Spark)?

---- On Mon, 07 Jan 2019 02:42:48 -0800 shashikantbangera@discover.com wrote ----

Hi team, 

please help , we are kind of blocked here. 

Cheers, 
Shashi 



-- 
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ 

--------------------------------------------------------------------- 
To unsubscribe e-mail: user-unsubscribe@spark.apache.org