You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "stahh (via GitHub)" <gi...@apache.org> on 2023/11/16 11:56:43 UTC
[I] thread '' panicked at 'called `Result::unwrap()` on an `Err` value: TypeNotSupported { field: "SNOWFLAKE_FIEDL", tp: Decimal128(38, 0), msg: "unknown type" }', py/src/lib.rs:35:10 [arrow]
stahh opened a new issue, #38744:
URL: https://github.com/apache/arrow/issues/38744
### Describe the bug, including details regarding any error messages, version, and platform.
Error when tried to upload parquet files that imported from Snowflake into Postgres
```
Start process
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: TypeNotSupported { field: "NUMERIC_FIELD", tp: Decimal128(38, 0), msg: "unknown type" }', py/src/lib.rs:35:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "/home/.../load_parquet.py", line 24, in <module>
parquet_to_pg()
File "/home/.../load_parquet.py", line 10, in process_hem2bito
encoder = ArrowToPostgresBinaryEncoder(dataset.schema)
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: TypeNotSupported { field: "NUMERIC_FIELD", tp: Decimal128(38, 0), msg: "unknown type" }
```
Process finished with exit code 1
-----------------------------------------------------------------------------------------------------------------
Step to reproduce:
1.
```sql
create or replace TABLE DB.SCHEMA.TBL (
VARCHAR_FIELD VARCHAR(256),
NUMERIC_FIELD NUMBER(38,0));
```
2.
```sql
insert into TABLE DB.SCHEMA.TBL (VARCHAR_FIELD, NUMERIC_FIELD) VALUES ('string 1', -2395872395873952873), ('string_2', 398537985637496873496);```
3.
```sql
COPY INTO 's3://test_buckettest/'
FROM DB.SCHEMA.TBL
credentials = (
aws_key_id='**********************'
aws_secret_key='********************************'
)
header=True
OVERWRITE = TRUE
file_format = (
type = PARQUET,
COMPRESSION = NONE
);
```
4. Code to run
```python
import psycopg
import pyarrow.dataset as ds
from pgpq import ArrowToPostgresBinaryEncoder
print('Start process')
s3_uri = f's3://test_bucket/test/'
dataset = ds.dataset(s3_uri, format="parquet")
encoder = ArrowToPostgresBinaryEncoder(dataset.schema)
with psycopg.connect('postgresql://**********************') as conn:
with conn.cursor() as cursor:
with cursor.copy(
"COPY public.companies FROM STDIN WITH (FORMAT BINARY)") as copy:
copy.write(encoder.write_header())
for batch in dataset.to_batches():
copy.write(encoder.write_batch(batch))
copy.write(encoder.finish())
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] thread '' panicked at 'called `Result::unwrap()` on an `Err` value: TypeNotSupported { field: "SNOWFLAKE_FIEDL", tp: Decimal128(38, 0), msg: "unknown type" }', py/src/lib.rs:35:10 [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #38744:
URL: https://github.com/apache/arrow/issues/38744#issuecomment-1814633240
Maybe better report it to https://github.com/adriangb/pgpq then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] thread '' panicked at 'called `Result::unwrap()` on an `Err` value: TypeNotSupported { field: "SNOWFLAKE_FIEDL", tp: Decimal128(38, 0), msg: "unknown type" }', py/src/lib.rs:35:10 [arrow]
Posted by "stahh (via GitHub)" <gi...@apache.org>.
stahh commented on issue #38744:
URL: https://github.com/apache/arrow/issues/38744#issuecomment-1814641652
Oh, sorry, I thought that problem in pyarrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] thread '' panicked at 'called `Result::unwrap()` on an `Err` value: TypeNotSupported { field: "SNOWFLAKE_FIEDL", tp: Decimal128(38, 0), msg: "unknown type" }', py/src/lib.rs:35:10 [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #38744:
URL: https://github.com/apache/arrow/issues/38744#issuecomment-1814581001
The error seems to come from `pgpq` rather than `pyarrow`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] thread '' panicked at 'called `Result::unwrap()` on an `Err` value: TypeNotSupported { field: "SNOWFLAKE_FIEDL", tp: Decimal128(38, 0), msg: "unknown type" }', py/src/lib.rs:35:10 [arrow]
Posted by "stahh (via GitHub)" <gi...@apache.org>.
stahh closed issue #38744: thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: TypeNotSupported { field: "SNOWFLAKE_FIEDL", tp: Decimal128(38, 0), msg: "unknown type" }', py/src/lib.rs:35:10
URL: https://github.com/apache/arrow/issues/38744
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] thread '' panicked at 'called `Result::unwrap()` on an `Err` value: TypeNotSupported { field: "SNOWFLAKE_FIEDL", tp: Decimal128(38, 0), msg: "unknown type" }', py/src/lib.rs:35:10 [arrow]
Posted by "stahh (via GitHub)" <gi...@apache.org>.
stahh commented on issue #38744:
URL: https://github.com/apache/arrow/issues/38744#issuecomment-1814621840
> The error seems to come from `pgpq` rather than `pyarrow`?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org