You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "lk-1984 (via GitHub)" <gi...@apache.org> on 2025/03/06 16:12:16 UTC
[I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
lk-1984 opened a new issue, #1771:
URL: https://github.com/apache/iceberg-python/issues/1771
### Apache Iceberg version
None
### Please describe the bug 🐞
I have set `PYICEBERG_HOME="$(pwd)" as told in the docs.
https://py.iceberg.apache.org/api/
```
This information must be placed inside a file called .pyiceberg.yaml located either in the $HOME or %USERPROFILE% directory (depending on whether the operating system is Unix-based or Windows-based, respectively) or in the $PYICEBERG_HOME directory (if the corresponding environment variable is set).
```
```
MINIO_REGION=us-east-1
PWD=/home/pyiceberg
MINIO_PORT=9100
HOME=/home/pyiceberg
HIVE_PORT=9083
VIRTUAL_ENV=/home/pyiceberg/.venv
TERM=xterm
MINIO_USERNAME=admin
USER=pyiceberg
HIVE_HOSTNAME=localhost
SHLVL=1
PYICEBERG_HOME=/home/pyiceberg
MINIO_HOSTNAME=localhost
VIRTUAL_ENV_PROMPT=(pyiceberg)
PS1=(pyiceberg)
MINIO_PASSWORD=adminadmin
PATH=/home/pyiceberg/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/pyiceberg/.local/bin
_=/usr/bin/printenv
catalog:
hive:
uri: thrift://localhost:9083
s3.endpoint: http://localhost:9100
s3.access-key-id: admin
s3.secret-access-key: adminadmin
s3.region: us-east-1
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
```
I have Pyiceberg 0.9.0, and Python 3.12.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704465656
The command was "list".
Btw, I pulled your repo, and built the thing in venv that has Python 3.11..
```
(.venv) dev@M-GM0J7P2WM1 iceberg-python % pyiceberg --help
Usage: pyiceberg [OPTIONS] COMMAND [ARGS]...
Options:
--catalog TEXT
--verbose BOOLEAN
--output [text|json]
--ugi TEXT
--uri TEXT
--credential TEXT
--help Show this message and exit.
Commands:
create Operation to create a namespace.
describe Describe a namespace or a table.
drop Operations to drop a namespace or table.
files List all the files of the table.
list List tables or namespaces.
list-refs List all the refs in the provided table.
location Return the location of the table.
properties Properties on tables/namespaces.
rename Rename a table.
schema Get the schema of the table.
spec Return the partition spec of the table.
uuid Return the UUID of the table.
version Print pyiceberg version.
(.venv) dev@foobariceberg-python % touch .pyiceberg.yaml
(.venv) dev@foobar iceberg-python % nvim .
(.venv) dev@foobariceberg-python % pyiceberg list
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
(.venv) dev@foobar iceberg-python % pwd
/Users/dev/Projects/external/iceberg-python
(.venv) dev@foobar iceberg-python % export HOME=$(pwd)
(.venv) dev@foobar ~ % echo $HOME
/Users/dev/Projects/external/iceberg-python
(.venv) dev@foobar ~ % pyiceberg list
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
(.venv) dev@foobar~ % export PYICEBERG_HOME=$(pwd)
(.venv) dev@foobar~ % echo $PYICEBERG_HOME
/Users/dev/Projects/external/iceberg-python
(.venv) dev@foobar ~ % pyiceberg list
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
(.venv) dev@foobar~ % cat $HOME/.pyiceberg.yaml
catalog:
hive:
uri: thrift://localhost:9083
s3.endpoint: http://localhost:9100
s3.access-key-id: admin
s3.secret-access-key: adminadmin
s3.region: us-east-1
(.venv) dev@foobar~ % cat $PYICEBERG_HOME/.pyiceberg.yaml
catalog:
hive:
uri: thrift://localhost:9083
s3.endpoint: http://localhost:9100
s3.access-key-id: admin
s3.secret-access-key: adminadmin
s3.region: us-east-1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 closed issue #1771: URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME
URL: https://github.com/apache/iceberg-python/issues/1771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2705666108
https://github.com/apache/iceberg-python/blob/main/pyiceberg/utils/config.py#L152
This is is the reason why an empty config is returned.
```
def get_catalog_config(self, catalog_name: str) -> Optional[RecursiveDict]:
print(f"self.config: {self.config}")
print(f"CATALOG: {CATALOG}")
if CATALOG in self.config:
catalog_name_lower = catalog_name.lower()
catalogs = self.config[CATALOG]
print(f"catalog_name_lower: {catalog_name_lower}")
print(f"catalogs: {catalogs}")
print(f"not isinstance(catalogs, dict): {not isinstance(catalogs, dict)}")
if not isinstance(catalogs, dict):
raise ValueError(f"Catalog configurations needs to be an object: {catalog_name}")
print(f"catalog_name_lower in catalogs: {catalog_name_lower in catalogs:}")
if catalog_name_lower in catalogs:
catalog_conf = catalogs[catalog_name_lower]
assert isinstance(catalog_conf, dict), f"Configuration path catalogs.{catalog_name_lower} needs to be an object"
return catalog_conf
print("THIS SHOULD NOT HAPPEN, BUT IT DID")
return None
```
```
self.config: {'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}, 'home': '/Users/dev/Projects/external/iceberg-python'}
CATALOG: catalog
catalog_name_lower: default
catalogs: {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}
not isinstance(catalogs, dict): False
catalog_name_lower in catalogs: False
THIS SHOULD NOT HAPPEN, BUT IT DID
env: None
new_config: {}
env: {}
catalog_properties: {}
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704478405
Same:
```
(.venv) dev@foobar~ % pyiceberg list --catalog hive
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704448846
Just to clarify
1. You set `PYICEBERG_HOME` to `/home/pyiceberg` in the environment variables
I see above
```
PYICEBERG_HOME=/home/pyiceberg
```
2. `/home/pyiceberg/.pyiceberg.yaml` exists and contains the following
```
catalog:
hive:
uri: thrift://localhost:9083
s3.endpoint: http://localhost:9100
s3.access-key-id: admin
s3.secret-access-key: adminadmin
s3.region: us-east-1
```
What command did you run to see the error
```
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2705764904
So, if the user of the pyiceberg CLI would be able to give the catalog name `hive`, then it would work like when using the pyiceberg as a library - `load_catalog(name="hive")`.
Now, in reality, the pyiceberg CLI does expect that the user would give the name of the catalog through the `--catalog` flag as there is no logic whatsoever for identifying which catalog in the `.pyiceberg` file would be selected after parsing the configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704471071
could you try `pyiceberg list --catalog hive`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704503736
Ok, there is something that overrides new_config:
```
(.venv) dev@foobar~ % pyiceberg list --catalog hive
/Users/dev/Projects/external/iceberg-python
{'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}}
{'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}}
new_config: {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}
new_config: {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}
new_config: {'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}, 'home': '/Users/dev/Projects/external/iceberg-python'}
new_config: {}
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
(.venv) dev@M-GM0J7P2WM1 ~ %
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2706132436
There is actually one combination that works, and which is documented in the click help.
```
Usage: pyiceberg [OPTIONS] COMMAND [ARGS]...
```
```
pyiceberg --catalog hive list
```
This indeed works.
However, the user should be instructed correctly when it does it wrong, like `pyiceberg list --catalog hive`..
```
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
```
Is very misleading..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2715204948
Thanks @lk-1984 for the detailed debugging session :)
I agree that the CLI should provide more user friendly warnings regarding catalog selection and override. I've opened #1784 to track this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]
Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704489427
With some debug print:
```
(.venv) dev@M-GM0J7P2WM1 ~ % pyiceberg list --catalog hive
/Users/dev/Projects/external/iceberg-python
{'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}}
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org