Airflow & AWS Lambda Tutorials?

[Andrew Maguire <an...@gmail.com>]

Hi all,

Does anyone know of any tutorials or examples of triggering an aws lambda
function from airflow?

I googled a bit bit could not find much. I do see the aws_lambda_hook in
there but just a little unsure how to use it.

Cheers,
Andy

Re: Airflow & AWS Lambda Tutorials?

[Taylor Edmiston <te...@gmail.com>]

Hey Andrew -

My suggestions are similar to Marcel's.

I favor invoking the Lambda function with the invoke
<https://boto3.readthedocs.io/en/latest/reference/services/lambda.html#Lambda.Client.invoke>
call in boto3 in a PythonOperator, or using the AwsHook if that abstraction
adds value here vs calling boto directly.

I second that using the SimpleHttpOperator in Airflow to hit the Lambda
function like a REST API is a reasonable alternative.

Personally, I would also recommend using a serverless framework such as
Zappa or Serverless as in my experience, there's a surprising amount of
boilerplate involved in writing, deploying, updating, and maintaining
serverless functions (+ API Gateway etc).

If you're considering Lambda alternatives, I think the Zeit Now Serverless
Docker Beta <https://zeit.co/blog/serverless-docker> is a pretty cool and
very easy way to run serverless functions.  There's a Hello World Flask
example <https://github.com/zeit/now-examples/tree/master/python-flask>
that shows how it all works in just a few lines.

*Taylor Edmiston*
Blog <https://blog.tedmiston.com/> | CV
<https://stackoverflow.com/cv/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor> | Stack Overflow
<https://stackoverflow.com/users/149428/taylor-edmiston>



On Wed, Jul 11, 2018 at 3:07 PM Hamade, Marcel (GfK) <Ma...@gfk.com>
wrote:

> The aws_lambda_hook it self uses the AWS_hook which is a wrapper arount
> the boto3 library(the standard way to interact with AWS via python). If you
> add the AWS connections correctly you can use the hook in one of your self
> written operators to trigger your lambda functions
>
> Ano way would be to trigger Lambda functions via AWS API Gateway
> https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-with-lambda-integration.html
> For that you can use the HTTP-Operator as a starting point but of course
> you have to handle HTTP-Authorisation.
>
> Cheers,
> Marcel
>
> ________________________________________
> Von: Andrew Maguire <an...@gmail.com>
> Gesendet: Mittwoch, 11. Juli 2018 13:33
> An: dev@airflow.incubator.apache.org
> Betreff: Airflow & AWS Lambda Tutorials?
>
> Hi all,
>
> Does anyone know of any tutorials or examples of triggering an aws lambda
> function from airflow?
>
> I googled a bit bit could not find much. I do see the aws_lambda_hook in
> there but just a little unsure how to use it.
>
> Cheers,
> Andy
>
> ________________________________
>
> GfK SE, Nuremberg, Germany, commercial register at the local court
> Amtsgericht Nuremberg HRB 25014; Management Board: Peter Feld (CEO),
> Christian Bigatà Joseph (CFO). Supervisory Board Chairman: Ralf
> Klein-Bölting. This email and any attachments may contain confidential or
> privileged information. Please note that unauthorized copying, disclosure
> or distribution of the material in this email is not permitted.
>
>

AW: Airflow & AWS Lambda Tutorials?

["Hamade, Marcel (GfK)" <Ma...@gfk.com>]

The aws_lambda_hook it self uses the AWS_hook which is a wrapper arount the boto3 library(the standard way to interact with AWS via python). If you add the AWS connections correctly you can use the hook in one of your self written operators to trigger your lambda functions

Ano way would be to trigger Lambda functions via AWS API Gateway https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-with-lambda-integration.html
For that you can use the HTTP-Operator as a starting point but of course you have to handle HTTP-Authorisation.

Cheers,
Marcel

________________________________________
Von: Andrew Maguire <an...@gmail.com>
Gesendet: Mittwoch, 11. Juli 2018 13:33
An: dev@airflow.incubator.apache.org
Betreff: Airflow & AWS Lambda Tutorials?

Hi all,

Does anyone know of any tutorials or examples of triggering an aws lambda
function from airflow?

I googled a bit bit could not find much. I do see the aws_lambda_hook in
there but just a little unsure how to use it.

Cheers,
Andy

________________________________

GfK SE, Nuremberg, Germany, commercial register at the local court Amtsgericht Nuremberg HRB 25014; Management Board: Peter Feld (CEO), Christian Bigatà Joseph (CFO). Supervisory Board Chairman: Ralf Klein-Bölting. This email and any attachments may contain confidential or privileged information. Please note that unauthorized copying, disclosure or distribution of the material in this email is not permitted.