You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "HanpyBin (via GitHub)" <gi...@apache.org> on 2023/07/01 14:02:58 UTC
[GitHub] [arrow] HanpyBin opened a new issue, #36426: time32[s] changes to time32[ms] after reading table from parquet file
HanpyBin opened a new issue, #36426:
URL: https://github.com/apache/arrow/issues/36426
### Describe the bug, including details regarding any error messages, version, and platform.
I create a table with one column and its type is Time32[s]. Then I write it a parquet file. After I read data from the parquet file, I find column type changes to Time32[ms]. Why is that?
My test code is below
```cpp
int main(int argc, char** argv) {
// generate table
arrow::Time32Builder builder(arrow::time32(arrow::TimeUnit::SECOND), arrow::default_memory_pool());
builder.Append(1625150000);
builder.Append(1625150100);
builder.Append(1625150200);
std::shared_ptr<arrow::Array> array;
builder.Finish(&array);
std::shared_ptr<arrow::Field> field = arrow::field("time", arrow::time32(arrow::TimeUnit::SECOND));
std::shared_ptr<arrow::Schema> schema = arrow::schema({field});
std::vector<std::shared_ptr<arrow::Array>> columns = {array};
std::shared_ptr<arrow::Table> table = arrow::Table::Make(schema, columns);
// write to file
arrow::MemoryPool* pool = arrow::default_memory_pool();
std::shared_ptr<arrow::io::FileOutputStream> outfile;
PARQUET_ASSIGN_OR_THROW(outfile, arrow::io::FileOutputStream::Open("time321.parquet"));
PARQUET_THROW_NOT_OK(parquet::arrow::WriteTable(*table, pool, outfile, 3));
std::cout << table->schema()->ToString() << std::endl;
std::shared_ptr<arrow::io::ReadableFile> infile;
PARQUET_ASSIGN_OR_THROW(infile,
arrow::io::ReadableFile::Open("time321.parquet",
arrow::default_memory_pool()));
// read file
std::unique_ptr<parquet::arrow::FileReader> reader;
PARQUET_THROW_NOT_OK(
parquet::arrow::OpenFile(infile, arrow::default_memory_pool(), &reader));
std::shared_ptr<arrow::Table> table1;
PARQUET_THROW_NOT_OK(reader->ReadTable(&table1));
std::cout << table1->schema()->ToString() << std::endl;
std::cout << "Loaded " << table1->num_rows() << " rows in " << table1->num_columns()
<< " columns." << std::endl;
return 0;
}
```
And output is
```shell
time: time32[s]
time: time32[ms]
Loaded 3 rows in 1 columns.
```
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] HanpyBin commented on issue #36426: [C++][Parquet] time32[s] changes to time32[ms] after reading table from parquet file
Posted by "HanpyBin (via GitHub)" <gi...@apache.org>.
HanpyBin commented on issue #36426:
URL: https://github.com/apache/arrow/issues/36426#issuecomment-1616408991
> Hi @HanpyBin
>
> Parquet format itself doesn't have "second": https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L90-L120 , and arrow schema would cast Second to Micro Seconds during converting:
>
> ```c++
> // The user implicitly wants timestamp data to retain its original time units,
> // however the Arrow seconds time unit can not be represented (annotated) in
> // any version of Parquet and so must be coerced to milliseconds.
> if (type.unit() == ::arrow::TimeUnit::SECOND) {
> *logical_type =
> TimestampLogicalTypeFromArrowTimestamp(type, ::arrow::TimeUnit::MILLI);
> return Status::OK();
> }
> ```
>
> You can also see: https://arrow.apache.org/docs/cpp/parquet.html#logical-types . And maybe you can use "CAST" during read.
Thank you for your help :) I will close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] HanpyBin closed issue #36426: [C++][Parquet] time32[s] changes to time32[ms] after reading table from parquet file
Posted by "HanpyBin (via GitHub)" <gi...@apache.org>.
HanpyBin closed issue #36426: [C++][Parquet] time32[s] changes to time32[ms] after reading table from parquet file
URL: https://github.com/apache/arrow/issues/36426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #36426: [C++]time32[s] changes to time32[ms] after reading table from parquet file
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #36426:
URL: https://github.com/apache/arrow/issues/36426#issuecomment-1615941946
Hi @HanpyBin
Parquet format itself doesn't have "second": https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L90-L120 , and arrow schema would cast Second to Micro Seconds during converting:
```c++
// The user implicitly wants timestamp data to retain its original time units,
// however the Arrow seconds time unit can not be represented (annotated) in
// any version of Parquet and so must be coerced to milliseconds.
if (type.unit() == ::arrow::TimeUnit::SECOND) {
*logical_type =
TimestampLogicalTypeFromArrowTimestamp(type, ::arrow::TimeUnit::MILLI);
return Status::OK();
}
```
You can also see: https://arrow.apache.org/docs/cpp/parquet.html#logical-types . And maybe you can use "CAST" during read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org