Re: Airflow & AWS Lambda Tutorials?

[Taylor Edmiston <te...@gmail.com>]

Hey Andrew -

My suggestions are similar to Marcel's.

I favor invoking the Lambda function with the invoke
<https://boto3.readthedocs.io/en/latest/reference/services/lambda.html#Lambda.Client.invoke>
call in boto3 in a PythonOperator, or using the AwsHook if that abstraction
adds value here vs calling boto directly.

I second that using the SimpleHttpOperator in Airflow to hit the Lambda
function like a REST API is a reasonable alternative.

Personally, I would also recommend using a serverless framework such as
Zappa or Serverless as in my experience, there's a surprising amount of
boilerplate involved in writing, deploying, updating, and maintaining
serverless functions (+ API Gateway etc).

If you're considering Lambda alternatives, I think the Zeit Now Serverless
Docker Beta <https://zeit.co/blog/serverless-docker> is a pretty cool and
very easy way to run serverless functions.  There's a Hello World Flask
example <https://github.com/zeit/now-examples/tree/master/python-flask>
that shows how it all works in just a few lines.

*Taylor Edmiston*
Blog <https://blog.tedmiston.com/> | CV
<https://stackoverflow.com/cv/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor> | Stack Overflow
<https://stackoverflow.com/users/149428/taylor-edmiston>



On Wed, Jul 11, 2018 at 3:07 PM Hamade, Marcel (GfK) <Ma...@gfk.com>
wrote:

> The aws_lambda_hook it self uses the AWS_hook which is a wrapper arount
> the boto3 library(the standard way to interact with AWS via python). If you
> add the AWS connections correctly you can use the hook in one of your self
> written operators to trigger your lambda functions
>
> Ano way would be to trigger Lambda functions via AWS API Gateway
> https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-with-lambda-integration.html
> For that you can use the HTTP-Operator as a starting point but of course
> you have to handle HTTP-Authorisation.
>
> Cheers,
> Marcel
>
> ________________________________________
> Von: Andrew Maguire <an...@gmail.com>
> Gesendet: Mittwoch, 11. Juli 2018 13:33
> An: dev@airflow.incubator.apache.org
> Betreff: Airflow & AWS Lambda Tutorials?
>
> Hi all,
>
> Does anyone know of any tutorials or examples of triggering an aws lambda
> function from airflow?
>
> I googled a bit bit could not find much. I do see the aws_lambda_hook in
> there but just a little unsure how to use it.
>
> Cheers,
> Andy
>
> ________________________________
>
> GfK SE, Nuremberg, Germany, commercial register at the local court
> Amtsgericht Nuremberg HRB 25014; Management Board: Peter Feld (CEO),
> Christian Bigatà Joseph (CFO). Supervisory Board Chairman: Ralf
> Klein-Bölting. This email and any attachments may contain confidential or
> privileged information. Please note that unauthorized copying, disclosure
> or distribution of the material in this email is not permitted.
>
>