Problem with OAuth login from within SpringContextProcessor using Camel

[Matt Foley <mf...@hortonworks.com>]

Hello,
I'm working on using Apache Camel in a SpringContextProcessor to send a SOQL query to Salesforce, and fetch the response into NiFi.  To make sure I got all the pieces tied together, I went through the following steps:

1. used Camel in a Spring context to read from a local file, and write a file, entirely within a Camel route (not in NiFi) -- works, shows basic use of Camel.
2. used Camel in a SpringContextProcessor to read from a file, and route it to a PutFile processor with a NiFi route -- works, shows basic use of SpringContextProcessor, including correct use of "toNiFi" channel from Camel.
3. used Camel in a Spring context to send a SOQL query to Salesforce, and write the result to a file, entirely within a Camel route -- works, and shows successful Salesforce login, and general correct use of Camel's Salesforce queries in Spring.
4. use Camel in a SpringContextProcessor to send a SOQL query to Salesforce, and route the result to a PutFile processor with a NiFi route -- doesn't work; see below.

I have 1, 2, and 3 working, which as noted above indicates correct use of most of the building blocks.

However, when I try #4, the login to Salesforce is attempted, but fails to connect, and times out after 1 minute.  (Normal response time from SF is ~1 sec.)  I know it is reading my credentials correctly, because two of the five credential strings show up in the logs.

Do I need further configuration to allow an SSL OAuth connection from a SpringContextProcessor in NiFi?  Or do you have other ideas for what could be wrong?  Suggestions for debugging methods or enabling relevant logs would also be welcome.  I'm using a Mac for my dev platform.

Nifi-app.log fragments and Spring context files for the four cases are available for viewing at https://github.com/mattf-horton/Spring-experiments , if anyone would be kind enough to look at them.

Thanks in advance for any advice or direction, or clarifying questions.
--Matt​

Re: Problem with OAuth login from within SpringContextProcessor using Camel

[Oleg Zhurakousky <oz...@hortonworks.com>]

Matt

Sorry for the late reply

Let me start by saying that I am happy to see that 3 out of 4 scenarios are working well for you and that Spring integration is getting some traction. Now to the problem. . .
SpringContextProcessor was designed to provide an ability to send, receive and/or both FlowFiles as Spring Messages to/from any application that is bootstrapped via Spring Application Context regardless of what such application may do. The important part here is that once FlowFile was converted to Spring Message it essentially is considered “out of NiFi” (the same would apply to Camel). So I am going to call it Spring application for simplicity for now.

So the question may need to be phrased a bit different. What I am thinking and picking up from your description is that something that you need to pass to your Spring application is missing. If so that could be due to misconfiguration in Spring or you may need something that needs to be passed (integrated with) from NiFi to Spring.

The way I would approach this is I would create a simple and totally isolated Spring application that works and then when it does we can bootstrap it into NiFi and see what’s wring if anything.

Does it make sense?

Cheers
Oleg

> On Sep 15, 2016, at 7:00 PM, Matt Foley <mf...@hortonworks.com> wrote:
> 
> 
> Hello,
> I'm working on using Apache Camel in a SpringContextProcessor to send a SOQL query to Salesforce, and fetch the response into NiFi.  To make sure I got all the pieces tied together, I went through the following steps:
> 
> 1. used Camel in a Spring context to read from a local file, and write a file, entirely within a Camel route (not in NiFi) -- works, shows basic use of Camel.
> 2. used Camel in a SpringContextProcessor to read from a file, and route it to a PutFile processor with a NiFi route -- works, shows basic use of SpringContextProcessor, including correct use of "toNiFi" channel from Camel.
> 3. used Camel in a Spring context to send a SOQL query to Salesforce, and write the result to a file, entirely within a Camel route -- works, and shows successful Salesforce login, and general correct use of Camel's Salesforce queries in Spring.
> 4. use Camel in a SpringContextProcessor to send a SOQL query to Salesforce, and route the result to a PutFile processor with a NiFi route -- doesn't work; see below.
> 
> I have 1, 2, and 3 working, which as noted above indicates correct use of most of the building blocks.
> 
> However, when I try #4, the login to Salesforce is attempted, but fails to connect, and times out after 1 minute.  (Normal response time from SF is ~1 sec.)  I know it is reading my credentials correctly, because two of the five credential strings show up in the logs.
> 
> Do I need further configuration to allow an SSL OAuth connection from a SpringContextProcessor in NiFi?  Or do you have other ideas for what could be wrong?  Suggestions for debugging methods or enabling relevant logs would also be welcome.  I'm using a Mac for my dev platform.
> 
> Nifi-app.log fragments and Spring context files for the four cases are available for viewing at https://github.com/mattf-horton/Spring-experiments , if anyone would be kind enough to look at them.
> 
> Thanks in advance for any advice or direction, or clarifying questions.
> --Matt​
> 
>