You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "lk-1984 (via GitHub)" <gi...@apache.org> on 2025/03/06 16:12:16 UTC

[I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

lk-1984 opened a new issue, #1771:
URL: https://github.com/apache/iceberg-python/issues/1771

   ### Apache Iceberg version
   
   None
   
   ### Please describe the bug 🐞
   
   I have set `PYICEBERG_HOME="$(pwd)" as told in the docs.
   
   https://py.iceberg.apache.org/api/
   
   ```
   This information must be placed inside a file called .pyiceberg.yaml located either in the $HOME or %USERPROFILE% directory (depending on whether the operating system is Unix-based or Windows-based, respectively) or in the $PYICEBERG_HOME directory (if the corresponding environment variable is set).
   ```
   
   
   ```
   MINIO_REGION=us-east-1
   PWD=/home/pyiceberg
   MINIO_PORT=9100
   HOME=/home/pyiceberg
   HIVE_PORT=9083
   VIRTUAL_ENV=/home/pyiceberg/.venv
   TERM=xterm
   MINIO_USERNAME=admin
   USER=pyiceberg
   HIVE_HOSTNAME=localhost
   SHLVL=1
   PYICEBERG_HOME=/home/pyiceberg
   MINIO_HOSTNAME=localhost
   VIRTUAL_ENV_PROMPT=(pyiceberg)
   PS1=(pyiceberg)
   MINIO_PASSWORD=adminadmin
   PATH=/home/pyiceberg/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/pyiceberg/.local/bin
   _=/usr/bin/printenv
   catalog:
     hive:
       uri: thrift://localhost:9083
       s3.endpoint: http://localhost:9100
       s3.access-key-id: admin
       s3.secret-access-key: adminadmin
       s3.region: us-east-1
   URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
   ```
   
   I have Pyiceberg 0.9.0, and Python 3.12.
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [x] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704465656

   The command was "list".
   
   Btw, I pulled your repo, and built the thing in venv that has Python 3.11..
   
   ```
   (.venv) dev@M-GM0J7P2WM1 iceberg-python % pyiceberg --help
   Usage: pyiceberg [OPTIONS] COMMAND [ARGS]...
   
   Options:
     --catalog TEXT
     --verbose BOOLEAN
     --output [text|json]
     --ugi TEXT
     --uri TEXT
     --credential TEXT
     --help                Show this message and exit.
   
   Commands:
     create      Operation to create a namespace.
     describe    Describe a namespace or a table.
     drop        Operations to drop a namespace or table.
     files       List all the files of the table.
     list        List tables or namespaces.
     list-refs   List all the refs in the provided table.
     location    Return the location of the table.
     properties  Properties on tables/namespaces.
     rename      Rename a table.
     schema      Get the schema of the table.
     spec        Return the partition spec of the table.
     uuid        Return the UUID of the table.
     version     Print pyiceberg version.
   (.venv) dev@foobariceberg-python % touch .pyiceberg.yaml
   (.venv) dev@foobar iceberg-python % nvim .
   (.venv) dev@foobariceberg-python % pyiceberg list
   URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
   (.venv) dev@foobar iceberg-python % pwd
   /Users/dev/Projects/external/iceberg-python
   (.venv) dev@foobar iceberg-python % export HOME=$(pwd)
   (.venv) dev@foobar ~ % echo $HOME
   /Users/dev/Projects/external/iceberg-python
   (.venv) dev@foobar ~ % pyiceberg list
   URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
   (.venv) dev@foobar~ % export PYICEBERG_HOME=$(pwd)
   (.venv) dev@foobar~ % echo $PYICEBERG_HOME
   /Users/dev/Projects/external/iceberg-python
   (.venv) dev@foobar ~ % pyiceberg list
   URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
   (.venv) dev@foobar~ % cat $HOME/.pyiceberg.yaml
   catalog:
     hive:
       uri: thrift://localhost:9083
       s3.endpoint: http://localhost:9100
       s3.access-key-id: admin
       s3.secret-access-key: adminadmin
       s3.region: us-east-1
   (.venv) dev@foobar~ % cat $PYICEBERG_HOME/.pyiceberg.yaml
   catalog:
     hive:
       uri: thrift://localhost:9083
       s3.endpoint: http://localhost:9100
       s3.access-key-id: admin
       s3.secret-access-key: adminadmin
       s3.region: us-east-1
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 closed issue #1771: URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME
URL: https://github.com/apache/iceberg-python/issues/1771


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2705666108

   https://github.com/apache/iceberg-python/blob/main/pyiceberg/utils/config.py#L152
   
   This is is the reason why an empty config is returned.
   
   ```
       def get_catalog_config(self, catalog_name: str) -> Optional[RecursiveDict]:
           print(f"self.config: {self.config}")
           print(f"CATALOG: {CATALOG}")
           if CATALOG in self.config:
               catalog_name_lower = catalog_name.lower()
               catalogs = self.config[CATALOG]
               print(f"catalog_name_lower: {catalog_name_lower}")
               print(f"catalogs: {catalogs}")
               print(f"not isinstance(catalogs, dict): {not isinstance(catalogs, dict)}")
               if not isinstance(catalogs, dict):
                   raise ValueError(f"Catalog configurations needs to be an object: {catalog_name}")
               print(f"catalog_name_lower in catalogs: {catalog_name_lower in catalogs:}")
               if catalog_name_lower in catalogs:
                   catalog_conf = catalogs[catalog_name_lower]
                   assert isinstance(catalog_conf, dict), f"Configuration path catalogs.{catalog_name_lower} needs to be an object"
                   return catalog_conf
           print("THIS SHOULD NOT HAPPEN, BUT IT DID")
           return None
   ```
   
   ```
   self.config: {'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}, 'home': '/Users/dev/Projects/external/iceberg-python'}
   CATALOG: catalog
   catalog_name_lower: default
   catalogs: {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}
   not isinstance(catalogs, dict): False
   catalog_name_lower in catalogs: False
   THIS SHOULD NOT HAPPEN, BUT IT DID
   env: None
   new_config: {}
   env: {}
   catalog_properties: {}
   URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704478405

   Same:
   
   ```
   (.venv) dev@foobar~ % pyiceberg list --catalog hive
   URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704448846

   Just to clarify
   1. You set `PYICEBERG_HOME` to `/home/pyiceberg` in the environment variables
   I see above
   ```
   PYICEBERG_HOME=/home/pyiceberg
   ```
   
   2. `/home/pyiceberg/.pyiceberg.yaml` exists and contains the following
   ```
   catalog:
     hive:
       uri: thrift://localhost:9083
       s3.endpoint: http://localhost:9100
       s3.access-key-id: admin
       s3.secret-access-key: adminadmin
       s3.region: us-east-1
   ```
   
   What command did you run to see the error 
   ```
   URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2705764904

   So, if the user of the pyiceberg CLI would be able to give the catalog name `hive`, then it would work like when using the pyiceberg as a library - `load_catalog(name="hive")`.
   
   Now, in reality, the pyiceberg CLI does expect that the user would give the name of the catalog through the `--catalog` flag as there is no logic whatsoever for identifying which catalog in the `.pyiceberg` file would be selected after parsing the configuration.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704471071

   could you try `pyiceberg list --catalog hive`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704503736

   Ok, there is something that overrides new_config:
   
   ```
   (.venv) dev@foobar~ % pyiceberg list --catalog hive
   /Users/dev/Projects/external/iceberg-python
   {'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}}
   {'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}}
   new_config: {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}
   new_config: {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}
   new_config: {'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}, 'home': '/Users/dev/Projects/external/iceberg-python'}
   new_config: {}
   URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
   (.venv) dev@M-GM0J7P2WM1 ~ %
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2706132436

   There is actually one combination that works, and which is documented in the click help.
   
   ```
   Usage: pyiceberg [OPTIONS] COMMAND [ARGS]...
   ```
   
   ```
   pyiceberg --catalog hive list
   ```
   
   This indeed works.
   
   However, the user should be instructed correctly when it does it wrong, like `pyiceberg list --catalog hive`..
   
   ```
   URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
   ```
   
   Is very misleading..
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2715204948

   Thanks @lk-1984 for the detailed debugging session :) 
   
   I agree that the CLI should provide more user friendly warnings regarding catalog selection and override. I've opened #1784 to track this issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #1771:
URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2704489427

   With some debug print:
   
   ```
   (.venv) dev@M-GM0J7P2WM1 ~ % pyiceberg list --catalog hive
   /Users/dev/Projects/external/iceberg-python
   {'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}}
   URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org