You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Craig Insley <cr...@gmail.com> on 2019/07/03 12:56:13 UTC
Re: Nifi Cross Account Download With A Profile Flag
Hello,
This is an old thread but I'm having the same issue on 1.9.2. I have cross
account access with an assumed role working just fine with the aws command
line using a profile tag. However, no matter how I configure
my AWSCredentialsProviderControllerService I get errors.
Using the profile tag only I get a 403.
Using a combination of the profile tag for credentials and the Assume Role
settings, I get "The security token included in the request is invalid."
Setting the credentials and Assume Role settings explicitly in the
controller service I get the same "The security token included in the
request is invalid."
Using a credentials file, I get the same "The security token included in
the request is invalid."
I do have an External ID. I really hope I am just missing something. Any
help would be appreciated!
Thanks,
Craig
On Thu, Sep 1, 2016 at 2:45 PM James Wing <jv...@gmail.com> wrote:
> Keren,
>
> I'm certain cross-account access can work in 0.6.0, I've done it.
>
> The timeout error calling sts:AssumeRole is not the same error you started
> with, right? Earlier, the error was 403 "Access Denied", which would have
> been farther in the auth process. Any idea what changed? Are you using
> the HTTP proxy settings on FetchS3Object?
>
>
> Thanks,
>
> James
>
> On Thu, Sep 1, 2016 at 10:30 AM, Tseytlin, Keren <
> Keren.Tseytlin@capitalone.com> wrote:
>
> > Hey James,
> >
> > No problem. I’ve tried running a bunch of different ways to do it
> manually
> > to try and avoid the CLI profile flag. But it seems like that is the only
> > it works locally (without Nifi). My cross account role doesn’t require an
> > External ID, and that feature doesn’t exist in 0.6.0 either.
> >
> > An small portion of the logs are below. The logs show that it starts to
> > attempt to get the S3 object, it times out on assuming the
> > role/credentials, and then fails.
> >
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.startSession(STS
> > AssumeRoleSessionCredentialsProvider.java:272)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.getCredentials(S
> > TSAssumeRoleSessionCredentialsProvider.java:247)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.getCredentials(S
> > TSAssumeRoleSessionCredentialsProvider.java:34)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.
> > invoke(A
> > WSSecurityTokenServiceClient.java:1098) [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.
> > assumeRo
> > le(AWSSecurityTokenServiceClient.java:1000)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.startSession(STS
> > AssumeRoleSessionCredentialsProvider.java:272)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.getCredentials(S
> > TSAssumeRoleSessionCredentialsProvider.java:247)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.getCredentials(S
> > TSAssumeRoleSessionCredentialsProvider.java:34)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3589)
> > [aws-java-sdk-s3-1.10.32.jar:na]
> > at
> > com.amazonaws.services.s3.AmazonS3Client.getObject(
> > AmazonS3Client.java:1116
> > ) [aws-java-sdk-s3-1.10.32.jar:na]
> > at
> > org.apache.nifi.processors.aws.s3.FetchS3Object.
> > onTrigger(FetchS3Object.jav
> > a:105) [nifi-aws-processors-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
> > at
> > org.apache.nifi.processor.AbstractProcessor.onTrigger(
> > AbstractProcessor.jav
> > a:27) [nifi-api-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
> >
> >
> > If you (or anyone scanning the thread) can think of a way to do this
> > without upgrading that’d be awesome. Otherwise, I’ll start motivating the
> > masses.
> >
> > Best,
> > Keren
> >
> > On 9/1/16, 12:17 PM, "James Wing" <jv...@gmail.com> wrote:
> >
> > >Keren,
> > >
> > >I'm sorry if my advice is a bit confusing, there have been some changes
> to
> > >AWS credentials over the last few versions. NiFi 0.6.0 does not have
> the
> > >option to use a CLI profile in AWSCredentialsProviderControllerService,
> I
> > >think that was introduced in 0.7.0.
> > >
> > >Would it be possible to share some of the log entries? Was there a
> stack
> > >trace associated with the timeout?
> > >
> > >Also, does your cross-account Role require an External ID? I do not
> > >believe that is supported in 0.6.0, but can be required to assume some
> > >roles.
> > >
> > >
> > >Thanks,
> > >
> > >James
> > >
> > >On Thu, Sep 1, 2016 at 7:57 AM, Tseytlin, Keren <
> > >Keren.Tseytlin@capitalone.com> wrote:
> > >
> > >> Thanks for your responses!
> > >>
> > >> @James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0.
> > >>
> > >> I¹ve set up debugging, and it shows me that it¹s trying to connect,
> but
> > >>it
> > >> times out on connecting. It would be awesome if it would also return
> the
> > >> account ID of the credentials it is trying to use.
> > >>
> > >> Is there any way to see the exact keys/tokens that Nifi is trying to
> use
> > >> to get the S3 object? I¹m not seeing it in the logs.
> > >>
> > >> I tried to set the Profile in Nifi, but it complains that it¹s not a
> > >>valid
> > >> property.
> > >>
> > >> Best,
> > >> Keren
> > >>
> > >> On 8/31/16, 6:24 PM, "Andrew Grande" <ap...@gmail.com> wrote:
> > >>
> > >> >Debug logging can be set in a processor itself in the UI, too.
> > >> >
> > >> >On Wed, Aug 31, 2016, 5:34 PM James Wing <jv...@gmail.com> wrote:
> > >> >
> > >> >> Keren,
> > >> >>
> > >> >> Which version of NiFi are you using?
> > >> >>
> > >> >> One thing I noticed in your configuration of FetchS3Object is you
> are
> > >> >> setting both the Access Key and Secret Key properties with the AWS
> > >> >> Credentials Provider. When you are using the AWS Credentials
> > >>Provider
> > >> >> Service, you should not specify keys.
> > >> >>
> > >> >> A more certainly helpful thing to do is enable debug logging for
> the
> > >>AWS
> > >> >> processor package by adding a line like the following to
> > >> >>conf/logback.xml:
> > >> >>
> > >> >> <logger name="org.apache.nifi.processors.aws" level="DEBUG"/>
> > >> >>
> > >> >> With the debug logging enabled, there are messages indicating which
> > >> >> credential type is being attempted. Your settings for the AWS
> > >> >>Credentials
> > >> >> Provider look appropriate. The controller service is indeed
> > >>designed to
> > >> >> refresh the STS token automagically using the AWS SDK classes for
> > >> >>temporary
> > >> >> credentials.
> > >> >>
> > >> >> Last, you might experiment with configuring
> > >> >> AWSCredentialsProviderControllerService to use your named CLI
> > profile
> > >> >> "crossaccountrole", which should also work.
> > >> >>
> > >> >> Thanks,
> > >> >>
> > >> >> James
> > >> >>
> > >> >> On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren <
> > >> >> Keren.Tseytlin@capitalone.com> wrote:
> > >> >>
> > >> >> > Hi All!
> > >> >> >
> > >> >> > Looking for some help on enabling Cross Account communication
> > >>within
> > >> >> Nifi!
> > >> >> >
> > >> >> > My goal: There are files stored from CloudTrail in an S3 bucket
> in
> > >> >>VPC B.
> > >> >> > My Nifi machines are in VPC A. I want Nifi to be able to get
> those
> > >> >>files
> > >> >> > from VPC B. VPC A and VPC B need to be communicating in the
> > >> >>FetchS3Object
> > >> >> > component.
> > >> >> >
> > >> >> > See this link for some additional info:
> > http://docs.aws.amazon.com/
> > >> >> >
> > >>
> > >>>>awscloudtrail/latest/userguide/cloudtrail-sharing-
> > logs-assume-role.html
> > >> >> >
> > >> >> > I have communication working manually on the Nifi machines in
> VPC A
> > >> >>when
> > >> >> I
> > >> >> > use the AWS CLI. The process is as follows:
> > >> >> >
> > >> >> > 1. Run sts -assume-role on my Nifi machine (VPC A) to assume
> a
> > >> >>role
> > >> >> > I've created in VPC B that is configured to have access to the S3
> > >> >>bucket
> > >> >> in
> > >> >> > VPC B.
> > >> >> >
> > >> >> > 2. This will generate temporary keys that need to be
> refreshed
> > >> >>every
> > >> >> > hour. There is no way to have assume role create permanent keys.
> > >> >>Export
> > >> >> the
> > >> >> > keys as environment variables.
> > >> >> >
> > >> >> > 3. Set up ~/.aws/config to have a profile "crossaccountrole"
> > >>that
> > >> >> > connects to the arn of the role created in VPC B.
> > >> >> >
> > >> >> > 4. Run the following command à "aws s3 cp s3://<object path>
> > >> >><object
> > >> >> > name locally> --profile crossaccountrole"
> > >> >> >
> > >> >> > Most importantly, if I ever try to run this without the --profile
> > >> >>flag,
> > >> >> > then it will not allow me to download the file. It seems like
> > >> >>perhaps to
> > >> >> > get it to work with Nifi I need a place to pass in the profile
> that
> > >> >>needs
> > >> >> > to be used in order for the communication to work.
> > >> >> >
> > >> >> > I've been trying to implement this in Nifi. Within the
> > >>FetchS3Object,
> > >> >>I
> > >> >> > have created an AWSCredentialsProviderService which has the
> > >>following
> > >> >> > properties:
> > >> >> >
> > >> >> > · Access Key: VPC A access key
> > >> >> >
> > >> >> > · Secret Key: VPC A secret key
> > >> >> >
> > >> >> > · Assume Role ARN: VPC B role
> > >> >> >
> > >> >> > · Assume Role Session Name: crossaccountrole
> > >> >> >
> > >> >> > · Session Time: 3600
> > >> >> > The general properties in the FetchS3Object are as follows:
> > >> >> >
> > >> >> > · Bucket: VPC B bucket name
> > >> >> >
> > >> >> > · Object: Filename of VPC B bucket object
> > >> >> >
> > >> >> > · Access Key: VPC A access key
> > >> >> >
> > >> >> > · Secret Key: VPC A secret key
> > >> >> >
> > >> >> > · AWS Credentials Provider Service: <see above>
> > >> >> >
> > >> >> > However, when this tries to run I get Access Denied. I've been
> > >>going
> > >> >> > through the source code for Nifi and I'm not sure if short-lived
> > >> >>tokens
> > >> >> get
> > >> >> > passed through. Can anyone please provide me some guidance or
> > >> >>suggestions
> > >> >> > on how to get this to work? J
> > >> >> >
> > >> >> > Best,
> > >> >> > Keren
> > >> >> > ________________________________________________________
> > >> >> >
> > >> >> > The information contained in this e-mail is confidential and/or
> > >> >> > proprietary to Capital One and/or its affiliates and may only be
> > >>used
> > >> >> > solely in performance of work or services for Capital One. The
> > >> >> information
> > >> >> > transmitted herewith is intended only for use by the individual
> or
> > >> >>entity
> > >> >> > to which it is addressed. If the reader of this message is not
> the
> > >> >> intended
> > >> >> > recipient, you are hereby notified that any review,
> retransmission,
> > >> >> > dissemination, distribution, copying or other use of, or taking
> of
> > >>any
> > >> >> > action in reliance upon this information is strictly prohibited.
> If
> > >> >>you
> > >> >> > have received this communication in error, please contact the
> > >>sender
> > >> >>and
> > >> >> > delete the material from your computer.
> > >> >> >
> > >> >>
> > >>
> > >> ________________________________________________________
> > >>
> > >> The information contained in this e-mail is confidential and/or
> > >> proprietary to Capital One and/or its affiliates and may only be used
> > >> solely in performance of work or services for Capital One. The
> > >>information
> > >> transmitted herewith is intended only for use by the individual or
> > >>entity
> > >> to which it is addressed. If the reader of this message is not the
> > >>intended
> > >> recipient, you are hereby notified that any review, retransmission,
> > >> dissemination, distribution, copying or other use of, or taking of any
> > >> action in reliance upon this information is strictly prohibited. If
> you
> > >> have received this communication in error, please contact the sender
> and
> > >> delete the material from your computer.
> > >>
> > >>
> >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
>