You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "candiduslynx (via GitHub)" <gi...@apache.org> on 2024/05/06 18:30:17 UTC

[I] bug: Time zone info not preserved on parquet roundtrip [arrow]

candiduslynx opened a new issue, #41557:
URL: https://github.com/apache/arrow/issues/41557

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   I think that https://github.com/apache/arrow/pull/39467 may have introduced the following bug:
   1. Create a record with `arrow.FixedTypes.Timestamp_us` column & some values (note that this column has `"UTC"` zone set)
   2. Write a file
   3. Read the file & record from it
   
   The resulting column would have empty time zone.
   
   ### Component(s)
   
   Go


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Go] Time zone info not preserved on parquet roundtrip [arrow]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #41557:
URL: https://github.com/apache/arrow/issues/41557#issuecomment-2096956806

   I'll take a look at this tomorrow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Go] Time zone info not preserved on parquet roundtrip [arrow]

Posted by "candiduslynx (via GitHub)" <gi...@apache.org>.
candiduslynx commented on issue #41557:
URL: https://github.com/apache/arrow/issues/41557#issuecomment-2105020285

   @zeroshade I think I misdirected the issue, we have an issue with DuckDB reading when the data from them is returned with stripped `UTC` timezone.
   Closing as incorrect


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Go] Time zone info not preserved on parquet roundtrip [arrow]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #41557:
URL: https://github.com/apache/arrow/issues/41557#issuecomment-2100828205

   @candiduslynx Can you provide me with some explicit sample code that shows this issue?
   
   When testing this against `main` on the Arrow repo I'm unable to replicate the issue you're seeing, see below:
   
   ```go
   func TestReadWriteTimestamp() {
     mem := memory.DefaultAllocator
     fld := arrow.Field{Name: "ts", Type: array.FixedWidthTypes.Timestamp_us}
     sc := arrow.NewSchema([]arrow.Field{fld}, nil)
     bldr := array.NewRecordBuilder(mem, sc)
     defer bldr.Release()
   
     tsbuilder := bldr.Field(0).(*array.TimestampBuilder)
     tsbuilder.AppendValues([]arrow.Timestamp{123450000, 567890000}, nil)
   
     rec := bldr.NewRecord()
     defer rec.Release()
   
     tbl := array.NewTableFromRecords(sc, []arrow.Record{rec})
     defer tbl.Release()
     // tbl schema column 1 has "UTC" for timezone!
   
     var buf bytes.Buffer
     err := pqarrow.WriteTable(tbl, &buf, tbl.NumRows(),
         parquet.NewWriterProperties(parquet.WithAllocator(mem)),
               pqarrow.DefaultWriterProps()))
     if err != nil {
         panic(err)
     }
   
     rdr := bytes.NewReader(buf.Bytes())
     out, err := pqarrow.ReadTable(context.Background(), rdr,
          parquet.NewReaderProperties(mem), pqarrow.ArrowReadProperties{}, mem)
     if err != nil {
         panic(err)
     }
     defer out.Release()
   
     fmt.Println(out)
   }
   ```
   
   I get the output:
   
   ```shell
   schema:
     fields: 1
       - ts: type=timestamp[us, tz=UTC]
       metadata: ["PARQUET:field_id": "-1"]
   ts: [[123450000 567890000]]
   ```
   
   Note that it still has the UTC for the timezone in the table that I read out from the file. What version of arrow/parquet are you using? Are you writing/reading the table differently than in my example above?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] bug: Time zone info not preserved on parquet roundtrip [arrow]

Posted by "candiduslynx (via GitHub)" <gi...@apache.org>.
candiduslynx commented on issue #41557:
URL: https://github.com/apache/arrow/issues/41557#issuecomment-2096660316

   @zeroshade could you please help with addressing this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Go] Time zone info not preserved on parquet roundtrip [arrow]

Posted by "candiduslynx (via GitHub)" <gi...@apache.org>.
candiduslynx closed issue #41557: [Go] Time zone info not preserved on parquet roundtrip
URL: https://github.com/apache/arrow/issues/41557


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org