You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/17 19:00:33 UTC
[GitHub] [airflow] DRavikanth opened a new issue #14282: Airflow task fails to submit a query to the Presto backend submitted using get_records/run
DRavikanth opened a new issue #14282:
URL: https://github.com/apache/airflow/issues/14282
**Apache Airflow version**: v2.0.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.17
**Environment**:
- **Cloud provider or hardware configuration**: Self Managed via Argo
- **OS** (e.g. from /etc/os-release):
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
- **Kernel** (e.g. `uname -a`): Linux airflow-idev-worker-0 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 GNU/Linux
- **Install tools**: "apache-airflow[presto]"
**What happened**:
Using PrestoSQL v341
I have a presto task in my dag with the following content.
```
def talk_to_presto():
ph = PrestoHook(presto_conn_id='presto_dum')
get_query = "select yyyymmdd, id from my_schema.my_table limit 1;"
# Fetch Data
results = ph.get_records(get_query)
# Below line is successfully displaying the results retrieved from the DB
logging.info(results)
# Insert Data into destination table
ph.insert_rows(table='dummy', rows=results)
```
Above task runs fine with no errors in Airflow task logs. However, the results that get_query returns is not being inserted into the dummy table in the insert_rows PrestoHook call. Any obvious mistake that I am doing here? Only following line is notably found in the Airflow task logs:
`[2021-02-16 21:09:35,822] {presto.py:185} INFO - Transactions are not enable in presto connection. Please use the isolation_level property to enable it. Falling back to insert all rows in one transaction.`
I also tried the following (observe run method usage):
```
def talk_to_presto():
ph = PrestoHook(presto_conn_id='presto_dum')
# Query PrestoDB
insert_query = "insert into scratch.test values(1, 'test');";
# Fetch Data
results = ph.run(insert_query, autocommit=True)
```
Presto Connection String in Airflow: {"protocol": "https"}
**What you expected to happen**:
When called insert_rows/run with the right set of parameters the records/dataframe should get successfully inserted into Presto
**How to reproduce it**:
1. Create a simple connection to Presto
2. create a simple table with one column
3. Insert into that table via Airflow PrestoHook using insert_rows/run function.
please note that I am able to work around this issue when using get_records(insert query)
How often does this problem occur? Once? Every time etc?
Always
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #14282: Airflow task fails to submit a query to the Presto backend submitted using get_records/run
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #14282:
URL: https://github.com/apache/airflow/issues/14282
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14282: Airflow task fails to submit a query to the Presto backend submitted using get_records/run
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14282:
URL: https://github.com/apache/airflow/issues/14282#issuecomment-780786355
Can you please open a https://github.com/apache/airflow/discussions issue instead or discuss it in Slack? This seems like more of a troubleshooting rather than bug report. Not sure how many of the users are using presto, but asking the question in #slack might bring better answers.
I am closing it for now but please copy it to "dicsussions" if you would like to follow it in GitHub instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org