You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/31 11:32:08 UTC
[GitHub] [airflow] Davide95 opened a new issue #15108: PostgresHook do not use the specified cursor
Davide95 opened a new issue #15108:
URL: https://github.com/apache/airflow/issues/15108
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Apache Airflow version**: 2.0.1
**What happened**:
From the docs it is reported that "you can choose cursor as {"cursor": "dictcursor"}." (https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/_api/airflow/providers/postgres/hooks/postgres/index.html).
I was not able to use this feature, it may be a bug.
Exception: `ERROR - tuple indices must be integers or slices, not str`
**What you expected to happen**:
I should be able to use the correct cursor while executing the queries
**How to reproduce it**:
```
hook = PostgresHook(
postgres_conn_id='conn-id',
extra=json.dump({'cursor': 'dictcursor'})
)
row = hook.get_first(
sql="""select * from filestatus """
)
logging.info(row['parquetfile'])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] scouvreur commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
scouvreur commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-979344848
Hey @Davide95 - are you still looking into addressing this issue ? I have a working dev environment with Airflow and Postgres is that's helpful I would be happy to open a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] scouvreur commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
scouvreur commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-979901385
@Davide95 @uranusjr do you think it's worth turning this issue into a feature request ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] scouvreur edited a comment on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
scouvreur edited a comment on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-979344848
Hey @Davide95 - are you still looking into addressing this issue ? I have a working dev environment with Airflow and Postgres if that's helpful I would be happy to open a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-811300433
> Does this mean that if you want to use different cursor types you have to create different connections?
Yes, as the implementation currently stands.
> Isn't this behavior particularly tricky?
It is definitely a bit inconvinient, but why would you need to mix both kinds of cursors in the first place? It’s a nice feature to have, but *particluarly tricky* wouldn’t be my description to the situation.
But nontheless, it would still be a good feature request and contribution if the interface can be designed reasonably.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Davide95 commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
Davide95 commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-979474652
Hi @scouvreur
Nope, I had some issues configuring the dev env in my machine, it seems that there were issues with btrfs:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-812220739
@uranusjr and @Davide95, Can this issue be closed now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-811268206
That said, it is definitely a bit confusing the hook just throws away unused arguments like this. Maybe it should throw an error instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-810993008
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-811267577
I believe the documentation is saying you need to set `extra` on your *Connection*, not the hook.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr edited a comment on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-811268206
That said, it is definitely a bit confusing the hook just throws away unused arguments like this. Maybe it should raise an exception instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Davide95 closed issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
Davide95 closed issue #15108:
URL: https://github.com/apache/airflow/issues/15108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-812294111
I think this depends on whether @Davide95 wants to turn this into a feature request (instead of a bug report) on setting the cursor class (and persuambly other extras) on a per-hook (per-operator?) basis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-979944570
Feel free to raise one yourself if you think it’s useful!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Davide95 commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
Davide95 commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-811295403
Hi @uranusjr
> I believe the documentation is saying you need to set extra on your Connection, not the hook.
Does this mean that if you want to use different cursor types you have to create different connections? Isn't this behavior particularly tricky?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Davide95 commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
Davide95 commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-812695742
I have some issues regarding breeze. I've already posted a message on Slack to get some help
If I will be able to have a working env, it would be great to contribute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Davide95 commented on issue #15108: PostgresHook do not use the specified cursor
Posted by GitBox <gi...@apache.org>.
Davide95 commented on issue #15108:
URL: https://github.com/apache/airflow/issues/15108#issuecomment-811302486
> Yes, as the implementation currently stands.
Ok, thanks for the clarification
> It is definitely a bit inconvinient, but why would you need to mix both kinds of cursors in the first place? It’s a nice feature to have, but _particluarly tricky_ wouldn’t be my description to the situation.
I'm inheriting a large and old codebase and migrating everything could be an issue. Integrating different DAGs with different settings would be a nice feature to have
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org