You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mobley-trent (via GitHub)" <gi...@apache.org> on 2023/10/16 14:07:25 UTC
[I] Bug: PostgreSql integration [iceberg-python]
mobley-trent opened a new issue, #78:
URL: https://github.com/apache/iceberg-python/issues/78
### Apache Iceberg version
0.5.0 (latest release)
### Please describe the bug 🐞
Python = 3.11
PostgreSql = v16
I'm having issues setting up the initial connection to postgresql. I'm trying to run the code in the docs but I'm getting this error:
```
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "iceberg_tables" does not exist
LINE 2: FROM iceberg_tables
^
```
This is my code:
```python
catalog = load_catalog(
"demo",
**{
"type": "sql",
"uri": "postgresql+psycopg2://[username]:[passwd]@localhost/demo"
}
)
catalog.create_namespace("docs_example")
```
This is the `.pyiceberg.yaml` file:
```yaml
catalog:
demo:
type: sql
uri: postgresql+psycopg2://[username]:[passwd]@localhost/demo
```
`demo` is the name of the database I'm using, it has a table named `students` which I'm trying to access.
I've attached the full stack trace for your review.
[stack_trace_iceberg.txt](https://github.com/apache/iceberg-python/files/12917749/stack_trace_iceberg.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: PostgreSql integration [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #78:
URL: https://github.com/apache/iceberg-python/issues/78#issuecomment-1771214572
I have the same feeling, I think this should be done once, and not every time you initialize the catalog (if you run a lot of jobs in parallel using Airflow, these calls can add up). I think we can add it [here](https://py.iceberg.apache.org/configuration/#sql-catalog) to the docs as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: PostgreSql integration [iceberg-python]
Posted by "gkaretka (via GitHub)" <gi...@apache.org>.
gkaretka commented on issue #78:
URL: https://github.com/apache/iceberg-python/issues/78#issuecomment-1771206442
Maybe this issue can be renamed into: _documentation_ rather than _bug_. What do you think @mobley-trent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Docs: PostgreSql integration [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #78:
URL: https://github.com/apache/iceberg-python/issues/78#issuecomment-1774101844
This should be a location to a bucket where the metadata and the data itself will be stored.
I think this finds its origin in Hive: https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdh/topics/hive-khub-location-syntax.html but it similar relevant to the SQL Catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Docs: PostgreSql integration [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #78: Docs: PostgreSql integration
URL: https://github.com/apache/iceberg-python/issues/78
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: PostgreSql integration [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #78:
URL: https://github.com/apache/iceberg-python/issues/78#issuecomment-1770729964
@mobley-trent @gkaretka Thanks for reaching out here. The tables are not created by default, but I think that might be the wrong behaviour since you both expected them to be created automatically.
To get things working on your end, you can run:
```python
catalog = load_catalog(
"demo",
**{
"type": "sql",
"uri": "postgresql+psycopg2://[username]:[passwd]@localhost/demo"
}
)
catalog.create_tables()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: PostgreSql integration [iceberg-python]
Posted by "gkaretka (via GitHub)" <gi...@apache.org>.
gkaretka commented on issue #78:
URL: https://github.com/apache/iceberg-python/issues/78#issuecomment-1770748027
Thanks 👍 was actually trying to investigate a little further and came to the same conclusion. Creating tables by calling:
`catalog.create_tables()`
solved my problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: PostgreSql integration [iceberg-python]
Posted by "gkaretka (via GitHub)" <gi...@apache.org>.
gkaretka commented on issue #78:
URL: https://github.com/apache/iceberg-python/issues/78#issuecomment-1770718118
👍 same issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Docs: PostgreSql integration [iceberg-python]
Posted by "mobley-trent (via GitHub)" <gi...@apache.org>.
mobley-trent commented on issue #78:
URL: https://github.com/apache/iceberg-python/issues/78#issuecomment-1772868143
@Fokko What should we place in the location parameter of `catalog.create_table(...)` ? Looking at the code, we are supposed to specify a warehouse location but I'm not quite sure what this means in regards to Postgresql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Docs: PostgreSql integration [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #78:
URL: https://github.com/apache/iceberg-python/issues/78#issuecomment-1771667298
Are you interested in providing a PR? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Docs: PostgreSql integration [iceberg-python]
Posted by "mobley-trent (via GitHub)" <gi...@apache.org>.
mobley-trent commented on issue #78:
URL: https://github.com/apache/iceberg-python/issues/78#issuecomment-1772102614
Yes I will look into this 👍 @Fokko
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org