You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@iceberg.apache.org by "apersilva (via GitHub)" <gi...@apache.org> on 2024/05/14 19:04:23 UTC

[I] Aws Glue error for append data [iceberg-python]

apersilva opened a new issue, #738:
URL: https://github.com/apache/iceberg-python/issues/738

   ### Apache Iceberg version
   
   0.6.0 (latest release)
   
   ### Please describe the bug 🐞
   
   A start use pyicerg with glue catalog and start error
   ![titulo](https://github.com/apache/iceberg-python/assets/169842606/a65d57d3-590e-4403-99bd-aa798882a3a4)
   The table in glue catalog have a comment column .
   It´s possible to ignore comment table for append data in table ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Aws Glue error for append data [iceberg-python]

Posted by "apersilva (via GitHub)" <gi...@apache.org>.

apersilva commented on issue #738:
URL: https://github.com/apache/iceberg-python/issues/738#issuecomment-2111055263

   └────┴───────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘Traceback (most recent call last):
     File "c:\great_teste\update_table.py", line 45, in update_table
       tabela.append(df)
     File "C:\Users\9001329\AppData\Roaming\Python\Python310\site-packages\pyiceberg\table\__init__.py", line 1057, in append
       _check_schema_compatible(self.schema(), other_schema=df.schema)
     File "C:\Users\9001329\AppData\Roaming\Python\Python310\site-packages\pyiceberg\table\__init__.py", line 175, in _check_schema_compatible
       raise ValueError(f"Mismatch in fields:\n{console.export_text()}")
   ValueError: Mismatch in fields:
   ┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓┃    ┃ Table field                                                               ┃ Dataframe field                          ┃┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩│ ❌ │ 1: nome_tabela: optional string (Nome data Tabela Processada)             │ 1: nome_tabela: option
 al string          │ 
   │ ❌ │ 2: nome_base_dados: optional string (Nome do Banco de dados que pertence  │ 2: nome_base_dados: optional string      │ 
   │    │ a tabela)                                                                 │                                          ││ ❌ │ 3: particao: optional string (Nome da particao)                           │ 3: particao: optional string             │ 
   │ ❌ │ 4: numero_registro: optional long (Quantidade de registros)               │ 4: numero_registro: optional long        │ 
   │ ❌ │ 5: process_date: optional string (parametro quando é enviado e passo para │ 5: process_date: optional string         │ 
   │    │ a funcao de escrita para particao)                                        │                                          ││ ❌ │ 6: particao_customizada: optional string (Indica que a partição é         │ 6: particao_customizada: optional string │ 
   │    │ diferente do padrão)                                                      │                                          ││ ❌ │ 7: data_criacao: optional date (Data em que foi inserido o registro)      │ 7: data_criacao: optional date           │ 
   └────┴───────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Aws Glue error for append data [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.

kevinjqliu closed issue #738: Aws Glue error for append data 
URL: https://github.com/apache/iceberg-python/issues/738


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Aws Glue error for append data [iceberg-python]

Posted by "apersilva (via GitHub)" <gi...@apache.org>.

apersilva commented on issue #738:
URL: https://github.com/apache/iceberg-python/issues/738#issuecomment-2116238350

   It´s work, thanks a lot.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Aws Glue error for append data [iceberg-python]

Posted by "ndrluis (via GitHub)" <gi...@apache.org>.

ndrluis commented on issue #738:
URL: https://github.com/apache/iceberg-python/issues/738#issuecomment-2116167599

   Sorry, I double-checked the Java implementation, and it's correct on the Python side.
   
   @apersilva, for your case, I believe you need to do something like this:
   
   ```
   from pyiceberg.io.pyarrow import schema_to_pyarrow
   
   schema = schema_to_pyarrow(tabela.schema())
   
   df = pa.Table.from_pylist(
       [
           {
               "nome_tabela": table_name,
               "nome_base_dados": database_name,
               "particao": partition_by,
               "numero_registro": size,
               "process_date": process_date,
               "particao_customizada": custom_partition,
               "data_criacao": datetime.now().date()
           }
       ],
       schema=schema
   )
   
   tabela.append(df)
   ````
   
   In a future release, there will be a function in the Schema object to return the Arrow schema, so it would look like this: `schema = tabela.schema().as_arrow()`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Aws Glue error for append data [iceberg-python]

Posted by "ndrluis (via GitHub)" <gi...@apache.org>.

ndrluis commented on issue #738:
URL: https://github.com/apache/iceberg-python/issues/738#issuecomment-2110972194

   Hello @apersilva, can you give us a minimal code example that can reproduce this error?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Aws Glue error for append data [iceberg-python]

Posted by "ndrluis (via GitHub)" <gi...@apache.org>.

ndrluis commented on issue #738:
URL: https://github.com/apache/iceberg-python/issues/738#issuecomment-2115576007

   @Fokko, can you help with clarifying the expected behavior? I believe we should compare the representations (repr) of the objects. Currently, the doc attribute is not included in the __repr__, so changing the comparison to be between repr objects might solve this problem. What do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Aws Glue error for append data [iceberg-python]

Posted by "apersilva (via GitHub)" <gi...@apache.org>.

apersilva commented on issue #738:
URL: https://github.com/apache/iceberg-python/issues/738#issuecomment-2111047010

   def update_table(database_target, table_target,database_name, table_name, partition_by,size, process_date, custom_partion):
       
       catalog =load_catalog('glue', **{
               'type': 'glue', 'verify' : False
           })
   
       tabela = catalog.load_table(f"{database_target}.{table_target}")
       
       metadata = {}
       for doc in tabela.metadata.schemas[0].columns:
           metadata.update({doc.name: f"({doc.doc})"})
       
       df = pa.Table.from_pylist(
       [
           {"nome_tabela": table_name, 
            "nome_base_dados": database_name, 
            "particao": partition_by, 
            "numero_registro": size, 
            "process_date": process_date, 
            "particao_customizada":  custom_partion,
            "data_criacao": datetime.now().date() }
       ],
       metadata=metadata      
       )
    
           
       tabela.append(df)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org