You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Hanneli Tavante <ha...@gmail.com> on 2017/01/18 23:18:29 UTC
Question - HttpHook
Hi! Could anybody help me?
I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8;
Databricks API 2.0 and Python 2.7.6 )
Is there any example with that? It looks like I can't authenticate. I
created an HTTP connection on 'Connections', inserted my credentials and
pointed to the port 443, but I did not succeed.
Thanks!
Re: Question - HttpHook
Posted by Hanneli Tavante <ha...@gmail.com>.
Thank you very much, Matt!
On Wed, Jan 18, 2017 at 5:11 PM Matt Hickman <ma...@gmail.com> wrote:
> Couple things:
> - You shouldn’t need to set port 443
> - Make sure the url starts with https:// in the connection settings
> - To check your auth is working properly try something like `print
> api_hook.get_conn(None).auth`
> - My connection in airflow has a trailing / and when I run your code
> below with the endpoint beginning with a “/“ I also get some html in the
> response. Try changing `api_hook.run('/api/2.0/jobs/list’)` to
> `api_hook.run('api/2.0/jobs/list’)`.
>
> Cheers,
> Matt
>
> On January 18, 2017 at 4:48:19 PM, Hanneli Tavante (hannelita@gmail.com)
> wrote:
>
> Thank you very much for the reply!
> I set these parameters using the web interface. I tried several combos for
> an HTTP type:
> - setting host, login, password only
> - setting schema field with https + setting host, login, password only
> - setting port 443 + setting host, login, password only
> - setting port 443 + schema field https + setting host, login, password
> only
>
> for a certain my_dag.py, I run a PythonOperator calling a method that does
> the following:
> ``` api_hook =
> HttpHook(http_conn_id='databricks_dcs_convert_json_to_parquet_raw_workflow_job_id',
> method='GET')
>
> response = api_hook.run('/api/2.0/jobs/list')
> ```
>
> I also tried a complete version of run method:
> ```
> response = api_hook.run('/api/2.0/jobs/list', None, 'application/json',
> None)
>
> ```
> The problem is the same - I end up on the Login page.
>
> (I know my credentials are fine since I can perform a plain curl)
>
> It is odd since I succeed for other API integrations.
>
> Thank you very much for the help!
>
>
> On Wed, Jan 18, 2017 at 4:38 PM Matt Hickman <ma...@gmail.com>
> wrote:
>
> Hanneli,
>
> We hit the data bricks api from Airflow. The connection info for our data
> bricks instance is pretty simple:
>
> conn type: http
> host: https://<cloud_id>.cloud.databricks.com/
> login: <your username>
> password: <your password>
>
> Knowing what error you get would be helpful! Also if you provide the code
> your using for your hook.
>
> Cheers,
> Matt
>
> On January 18, 2017 at 4:33:07 PM, Hanneli Tavante (hannelita@gmail.com)
> wrote:
>
> Hi! Could anybody help me?
> I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8;
> Databricks API 2.0 and Python 2.7.6 )
> Is there any example with that? It looks like I can't authenticate. I
> created an HTTP connection on 'Connections', inserted my credentials and
> pointed to the port 443, but I did not succeed.
> Thanks!
>
>
Re: Question - HttpHook
Posted by Matt Hickman <ma...@gmail.com>.
Couple things:
- You shouldn’t need to set port 443
- Make sure the url starts with https:// in the connection settings
- To check your auth is working properly try something like `print
api_hook.get_conn(None).auth`
- My connection in airflow has a trailing / and when I run your code
below with the endpoint beginning with a “/“ I also get some html in the
response. Try changing `api_hook.run('/api/2.0/jobs/list’)` to
`api_hook.run('api/2.0/jobs/list’)`.
Cheers,
Matt
On January 18, 2017 at 4:48:19 PM, Hanneli Tavante (hannelita@gmail.com)
wrote:
Thank you very much for the reply!
I set these parameters using the web interface. I tried several combos for
an HTTP type:
- setting host, login, password only
- setting schema field with https + setting host, login, password only
- setting port 443 + setting host, login, password only
- setting port 443 + schema field https + setting host, login, password
only
for a certain my_dag.py, I run a PythonOperator calling a method that does
the following:
``` api_hook =
HttpHook(http_conn_id='databricks_dcs_convert_json_to_parquet_raw_workflow_job_id',
method='GET')
response = api_hook.run('/api/2.0/jobs/list')
```
I also tried a complete version of run method:
```
response = api_hook.run('/api/2.0/jobs/list', None, 'application/json',
None)
```
The problem is the same - I end up on the Login page.
(I know my credentials are fine since I can perform a plain curl)
It is odd since I succeed for other API integrations.
Thank you very much for the help!
On Wed, Jan 18, 2017 at 4:38 PM Matt Hickman <ma...@gmail.com> wrote:
Hanneli,
We hit the data bricks api from Airflow. The connection info for our data
bricks instance is pretty simple:
conn type: http
host: https://<cloud_id>.cloud.databricks.com/
login: <your username>
password: <your password>
Knowing what error you get would be helpful! Also if you provide the code
your using for your hook.
Cheers,
Matt
On January 18, 2017 at 4:33:07 PM, Hanneli Tavante (hannelita@gmail.com)
wrote:
Hi! Could anybody help me?
I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8;
Databricks API 2.0 and Python 2.7.6 )
Is there any example with that? It looks like I can't authenticate. I
created an HTTP connection on 'Connections', inserted my credentials and
pointed to the port 443, but I did not succeed.
Thanks!
Re: Question - HttpHook
Posted by Hanneli Tavante <ha...@gmail.com>.
Thank you very much for the reply!
I set these parameters using the web interface. I tried several combos for
an HTTP type:
- setting host, login, password only
- setting schema field with https + setting host, login, password only
- setting port 443 + setting host, login, password only
- setting port 443 + schema field https + setting host, login, password
only
for a certain my_dag.py, I run a PythonOperator calling a method that does
the following:
``` api_hook =
HttpHook(http_conn_id='databricks_dcs_convert_json_to_parquet_raw_workflow_job_id',
method='GET')
response = api_hook.run('/api/2.0/jobs/list')
```
I also tried a complete version of run method:
```
response = api_hook.run('/api/2.0/jobs/list', None, 'application/json',
None)
```
The problem is the same - I end up on the Login page.
(I know my credentials are fine since I can perform a plain curl)
It is odd since I succeed for other API integrations.
Thank you very much for the help!
On Wed, Jan 18, 2017 at 4:38 PM Matt Hickman <ma...@gmail.com> wrote:
Hanneli,
We hit the data bricks api from Airflow. The connection info for our data
bricks instance is pretty simple:
conn type: http
host: https://<cloud_id>.cloud.databricks.com/
login: <your username>
password: <your password>
Knowing what error you get would be helpful! Also if you provide the code
your using for your hook.
Cheers,
Matt
On January 18, 2017 at 4:33:07 PM, Hanneli Tavante (hannelita@gmail.com)
wrote:
Hi! Could anybody help me?
I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8;
Databricks API 2.0 and Python 2.7.6 )
Is there any example with that? It looks like I can't authenticate. I
created an HTTP connection on 'Connections', inserted my credentials and
pointed to the port 443, but I did not succeed.
Thanks!
Re: Question - HttpHook
Posted by Matt Hickman <ma...@gmail.com>.
Hanneli,
We hit the data bricks api from Airflow. The connection info for our data
bricks instance is pretty simple:
conn type: http
host: https://<cloud_id>.cloud.databricks.com/
login: <your username>
password: <your password>
Knowing what error you get would be helpful! Also if you provide the code
your using for your hook.
Cheers,
Matt
On January 18, 2017 at 4:33:07 PM, Hanneli Tavante (hannelita@gmail.com)
wrote:
Hi! Could anybody help me?
I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8;
Databricks API 2.0 and Python 2.7.6 )
Is there any example with that? It looks like I can't authenticate. I
created an HTTP connection on 'Connections', inserted my credentials and
pointed to the port 443, but I did not succeed.
Thanks!