You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/07/16 04:39:47 UTC
[GitHub] [arrow] westonpace opened a new pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
westonpace opened a new pull request #10729:
URL: https://github.com/apache/arrow/pull/10729
This fixes two issues.
* The null_count must be obtained from the indices array and not the values array
* The min/max should be based on referenced values and not all values in the values array
Note: This further adds a dependency from parquet onto arrow::compute (I use it both to compute the unique indices and to take the referenced values). This dependency already existed (column_writer.cc relies on arrow::compute::Cast) so I'm pretty sure this isn't a problem. Related: ARROW-8891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#issuecomment-895494683
forgot to to comment, these changes looks fine. thanks for tracing down the paths. I don't know why stats for datapagev2 would have been disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671594390
##########
File path: cpp/src/parquet/arrow/arrow_reader_writer_test.cc
##########
@@ -3796,6 +3796,95 @@ TEST(TestArrowWriterAdHoc, SchemaMismatch) {
ASSERT_RAISES(Invalid, writer->WriteTable(*tbl, 1));
}
+// ----------------------------------------------------------------------
+// Tests for directly writing DictionaryArray
+class TestArrowWriteDictionary : public ::testing::Test {
+ public:
+ void SetUp() override {
+ properties_ = default_arrow_writer_properties();
+ serialized_data_ = AllocateBuffer();
+ }
+
+ // Generates a range of single character strings from start to end (inclusive)
+ // Nulls will be inserted starting at index 0. If there are any null
+ // values then start will not be the true min. However, the dictionary
+ // array will always contain all the values even if they aren't all used.
+ void GenerateRange(int num_nulls, char start, char end) {
+ int num_values = end - start + 1;
+ DCHECK_LT(num_nulls, num_values);
+
+ ::arrow::StringBuilder dictionary_builder;
+ ::arrow::Int32Builder indices_builder;
+
+ for (char val = start; val <= end; val++) {
+ int32_t index = static_cast<int32_t>(val - start);
+ if (val - start >= num_nulls) {
+ ASSERT_OK(indices_builder.Append(index));
+ } else {
+ ASSERT_OK(indices_builder.AppendNull());
+ }
+ ASSERT_OK(dictionary_builder.Append(&val, 1));
+ }
+
+ std::shared_ptr<::arrow::Array> dictionary;
+ std::shared_ptr<::arrow::Array> indices;
+ ASSERT_OK(dictionary_builder.Finish(&dictionary));
+ ASSERT_OK(indices_builder.Finish(&indices));
+
+ ASSERT_OK_AND_ASSIGN(auto test_column,
+ ::arrow::DictionaryArray::FromArrays(indices, dictionary));
+
+ auto schema = ::arrow::schema({::arrow::field("values", test_column->type())});
+ auto batch = ::arrow::RecordBatch::Make(schema, num_values, {test_column});
+ ASSERT_OK_AND_ASSIGN(test_data_, ::arrow::Table::FromRecordBatches({batch}));
+ }
+
+ void WriteToBuffer() {
+ auto out_stream = std::make_shared<::arrow::io::BufferOutputStream>(serialized_data_);
+ auto writer_properties = default_writer_properties();
+ std::unique_ptr<FileWriter> writer;
+ ASSERT_OK(FileWriter::Open(*test_data_->schema(), ::arrow::default_memory_pool(),
+ out_stream, writer_properties,
+ default_arrow_writer_properties(), &writer));
+ ASSERT_OK(writer->WriteTable(*test_data_, std::numeric_limits<int64_t>::max()));
+ ASSERT_OK(writer->Close());
+ }
+
+ std::shared_ptr<FileMetaData> LoadWrittenMetadata() {
+ auto buffer_reader = std::make_shared<::arrow::io::BufferReader>(serialized_data_);
+ auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+ return parquet_reader->metadata();
+ }
+
+ void CheckNullCount() {
+ GenerateRange(5, 'a', 'z');
+ WriteToBuffer();
+ auto metadata = LoadWrittenMetadata();
+ auto stats = metadata->RowGroup(0)->ColumnChunk(0)->statistics();
+ ASSERT_TRUE(stats->HasNullCount());
+ ASSERT_EQ(stats->null_count(), 5);
+ ASSERT_EQ(stats->num_values(), 21);
+ }
+
+ void CheckMinMax() {
+ GenerateRange(5, 'a', 'z');
+ WriteToBuffer();
+ auto metadata = LoadWrittenMetadata();
+ auto stats = metadata->RowGroup(0)->ColumnChunk(0)->statistics();
+ ASSERT_TRUE(stats->HasMinMax());
+ ASSERT_EQ(stats->EncodeMin(), "f");
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r677950980
##########
File path: cpp/src/parquet/column_writer.cc
##########
@@ -1490,7 +1490,8 @@ Status TypedColumnWriterImpl<DType>::WriteArrowDictionary(
// TODO(wesm): If some dictionary values are unobserved, then the
Review comment:
Sorry for taking so long to get back to this but I dug into this a bit today and I can't find any page statistics. Either in the code or the thrift definition. Can you give me a pointer to the statistics objects you think might be incorrect.
There is a `PageEncodingStats` which has a `count` but that appears to be counting the number of pages and not counting the rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671401408
##########
File path: cpp/src/parquet/arrow/arrow_reader_writer_test.cc
##########
@@ -3796,6 +3796,95 @@ TEST(TestArrowWriterAdHoc, SchemaMismatch) {
ASSERT_RAISES(Invalid, writer->WriteTable(*tbl, 1));
}
+// ----------------------------------------------------------------------
+// Tests for directly writing DictionaryArray
+class TestArrowWriteDictionary : public ::testing::Test {
+ public:
+ void SetUp() override {
+ properties_ = default_arrow_writer_properties();
+ serialized_data_ = AllocateBuffer();
+ }
+
+ // Generates a range of single character strings from start to end (inclusive)
+ // Nulls will be inserted starting at index 0. If there are any null
+ // values then start will not be the true min. However, the dictionary
+ // array will always contain all the values even if they aren't all used.
+ void GenerateRange(int num_nulls, char start, char end) {
+ int num_values = end - start + 1;
+ DCHECK_LT(num_nulls, num_values);
+
+ ::arrow::StringBuilder dictionary_builder;
+ ::arrow::Int32Builder indices_builder;
+
+ for (char val = start; val <= end; val++) {
+ int32_t index = static_cast<int32_t>(val - start);
+ if (val - start >= num_nulls) {
+ ASSERT_OK(indices_builder.Append(index));
+ } else {
+ ASSERT_OK(indices_builder.AppendNull());
+ }
+ ASSERT_OK(dictionary_builder.Append(&val, 1));
+ }
+
+ std::shared_ptr<::arrow::Array> dictionary;
+ std::shared_ptr<::arrow::Array> indices;
+ ASSERT_OK(dictionary_builder.Finish(&dictionary));
+ ASSERT_OK(indices_builder.Finish(&indices));
+
+ ASSERT_OK_AND_ASSIGN(auto test_column,
+ ::arrow::DictionaryArray::FromArrays(indices, dictionary));
+
+ auto schema = ::arrow::schema({::arrow::field("values", test_column->type())});
+ auto batch = ::arrow::RecordBatch::Make(schema, num_values, {test_column});
+ ASSERT_OK_AND_ASSIGN(test_data_, ::arrow::Table::FromRecordBatches({batch}));
+ }
+
+ void WriteToBuffer() {
+ auto out_stream = std::make_shared<::arrow::io::BufferOutputStream>(serialized_data_);
+ auto writer_properties = default_writer_properties();
+ std::unique_ptr<FileWriter> writer;
+ ASSERT_OK(FileWriter::Open(*test_data_->schema(), ::arrow::default_memory_pool(),
+ out_stream, writer_properties,
+ default_arrow_writer_properties(), &writer));
+ ASSERT_OK(writer->WriteTable(*test_data_, std::numeric_limits<int64_t>::max()));
+ ASSERT_OK(writer->Close());
+ }
+
+ std::shared_ptr<FileMetaData> LoadWrittenMetadata() {
+ auto buffer_reader = std::make_shared<::arrow::io::BufferReader>(serialized_data_);
+ auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+ return parquet_reader->metadata();
+ }
+
+ void CheckNullCount() {
+ GenerateRange(5, 'a', 'z');
+ WriteToBuffer();
+ auto metadata = LoadWrittenMetadata();
+ auto stats = metadata->RowGroup(0)->ColumnChunk(0)->statistics();
+ ASSERT_TRUE(stats->HasNullCount());
+ ASSERT_EQ(stats->null_count(), 5);
+ ASSERT_EQ(stats->num_values(), 21);
+ }
+
+ void CheckMinMax() {
+ GenerateRange(5, 'a', 'z');
+ WriteToBuffer();
+ auto metadata = LoadWrittenMetadata();
+ auto stats = metadata->RowGroup(0)->ColumnChunk(0)->statistics();
+ ASSERT_TRUE(stats->HasMinMax());
+ ASSERT_EQ(stats->EncodeMin(), "f");
Review comment:
nit: min and max should probably use EXPECT_EQ instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671594773
##########
File path: cpp/src/parquet/arrow/arrow_reader_writer_test.cc
##########
@@ -3796,6 +3796,95 @@ TEST(TestArrowWriterAdHoc, SchemaMismatch) {
ASSERT_RAISES(Invalid, writer->WriteTable(*tbl, 1));
}
+// ----------------------------------------------------------------------
+// Tests for directly writing DictionaryArray
+class TestArrowWriteDictionary : public ::testing::Test {
+ public:
+ void SetUp() override {
+ properties_ = default_arrow_writer_properties();
+ serialized_data_ = AllocateBuffer();
+ }
+
+ // Generates a range of single character strings from start to end (inclusive)
+ // Nulls will be inserted starting at index 0. If there are any null
+ // values then start will not be the true min. However, the dictionary
+ // array will always contain all the values even if they aren't all used.
+ void GenerateRange(int num_nulls, char start, char end) {
+ int num_values = end - start + 1;
+ DCHECK_LT(num_nulls, num_values);
+
+ ::arrow::StringBuilder dictionary_builder;
+ ::arrow::Int32Builder indices_builder;
+
+ for (char val = start; val <= end; val++) {
+ int32_t index = static_cast<int32_t>(val - start);
+ if (val - start >= num_nulls) {
+ ASSERT_OK(indices_builder.Append(index));
+ } else {
+ ASSERT_OK(indices_builder.AppendNull());
+ }
+ ASSERT_OK(dictionary_builder.Append(&val, 1));
+ }
+
+ std::shared_ptr<::arrow::Array> dictionary;
+ std::shared_ptr<::arrow::Array> indices;
+ ASSERT_OK(dictionary_builder.Finish(&dictionary));
+ ASSERT_OK(indices_builder.Finish(&indices));
+
+ ASSERT_OK_AND_ASSIGN(auto test_column,
+ ::arrow::DictionaryArray::FromArrays(indices, dictionary));
+
+ auto schema = ::arrow::schema({::arrow::field("values", test_column->type())});
+ auto batch = ::arrow::RecordBatch::Make(schema, num_values, {test_column});
+ ASSERT_OK_AND_ASSIGN(test_data_, ::arrow::Table::FromRecordBatches({batch}));
+ }
+
+ void WriteToBuffer() {
+ auto out_stream = std::make_shared<::arrow::io::BufferOutputStream>(serialized_data_);
+ auto writer_properties = default_writer_properties();
+ std::unique_ptr<FileWriter> writer;
+ ASSERT_OK(FileWriter::Open(*test_data_->schema(), ::arrow::default_memory_pool(),
+ out_stream, writer_properties,
+ default_arrow_writer_properties(), &writer));
+ ASSERT_OK(writer->WriteTable(*test_data_, std::numeric_limits<int64_t>::max()));
+ ASSERT_OK(writer->Close());
+ }
+
+ std::shared_ptr<FileMetaData> LoadWrittenMetadata() {
+ auto buffer_reader = std::make_shared<::arrow::io::BufferReader>(serialized_data_);
+ auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+ return parquet_reader->metadata();
+ }
+
+ void CheckNullCount() {
+ GenerateRange(5, 'a', 'z');
+ WriteToBuffer();
+ auto metadata = LoadWrittenMetadata();
+ auto stats = metadata->RowGroup(0)->ColumnChunk(0)->statistics();
+ ASSERT_TRUE(stats->HasNullCount());
+ ASSERT_EQ(stats->null_count(), 5);
+ ASSERT_EQ(stats->num_values(), 21);
+ }
+
+ void CheckMinMax() {
+ GenerateRange(5, 'a', 'z');
+ WriteToBuffer();
+ auto metadata = LoadWrittenMetadata();
Review comment:
For this PR I've backed away from auto. I'll have to check myself going forward as my inclination had been "whenever possible".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#issuecomment-881177476
And apparently: PARQUET-1783 a triple wammy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671602048
##########
File path: cpp/src/parquet/column_writer.cc
##########
@@ -1490,7 +1490,8 @@ Status TypedColumnWriterImpl<DType>::WriteArrowDictionary(
// TODO(wesm): If some dictionary values are unobserved, then the
Review comment:
functional concern (unfortunately I think the performance would be worse since it will require extra allocations for each batch). Statistics are stored at two levels rowgroup (column chunk) and page level. The batching in done here and in other locations in the code to be able to get some level of vectorization but not make any individual page too large. I might have traced the code incorrectly but the current location for updating statistics will only cover correct statistics at the row group level. It seems like this is an orthogonal bug so you can maybe do it as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671403080
##########
File path: cpp/src/parquet/arrow/arrow_reader_writer_test.cc
##########
@@ -3796,6 +3796,95 @@ TEST(TestArrowWriterAdHoc, SchemaMismatch) {
ASSERT_RAISES(Invalid, writer->WriteTable(*tbl, 1));
}
+// ----------------------------------------------------------------------
+// Tests for directly writing DictionaryArray
+class TestArrowWriteDictionary : public ::testing::Test {
+ public:
+ void SetUp() override {
+ properties_ = default_arrow_writer_properties();
+ serialized_data_ = AllocateBuffer();
+ }
+
+ // Generates a range of single character strings from start to end (inclusive)
+ // Nulls will be inserted starting at index 0. If there are any null
+ // values then start will not be the true min. However, the dictionary
+ // array will always contain all the values even if they aren't all used.
+ void GenerateRange(int num_nulls, char start, char end) {
+ int num_values = end - start + 1;
+ DCHECK_LT(num_nulls, num_values);
+
+ ::arrow::StringBuilder dictionary_builder;
+ ::arrow::Int32Builder indices_builder;
+
+ for (char val = start; val <= end; val++) {
+ int32_t index = static_cast<int32_t>(val - start);
+ if (val - start >= num_nulls) {
+ ASSERT_OK(indices_builder.Append(index));
+ } else {
+ ASSERT_OK(indices_builder.AppendNull());
+ }
+ ASSERT_OK(dictionary_builder.Append(&val, 1));
+ }
+
+ std::shared_ptr<::arrow::Array> dictionary;
+ std::shared_ptr<::arrow::Array> indices;
+ ASSERT_OK(dictionary_builder.Finish(&dictionary));
+ ASSERT_OK(indices_builder.Finish(&indices));
+
+ ASSERT_OK_AND_ASSIGN(auto test_column,
+ ::arrow::DictionaryArray::FromArrays(indices, dictionary));
+
+ auto schema = ::arrow::schema({::arrow::field("values", test_column->type())});
+ auto batch = ::arrow::RecordBatch::Make(schema, num_values, {test_column});
+ ASSERT_OK_AND_ASSIGN(test_data_, ::arrow::Table::FromRecordBatches({batch}));
+ }
+
+ void WriteToBuffer() {
+ auto out_stream = std::make_shared<::arrow::io::BufferOutputStream>(serialized_data_);
+ auto writer_properties = default_writer_properties();
+ std::unique_ptr<FileWriter> writer;
+ ASSERT_OK(FileWriter::Open(*test_data_->schema(), ::arrow::default_memory_pool(),
+ out_stream, writer_properties,
+ default_arrow_writer_properties(), &writer));
+ ASSERT_OK(writer->WriteTable(*test_data_, std::numeric_limits<int64_t>::max()));
+ ASSERT_OK(writer->Close());
+ }
+
+ std::shared_ptr<FileMetaData> LoadWrittenMetadata() {
+ auto buffer_reader = std::make_shared<::arrow::io::BufferReader>(serialized_data_);
+ auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+ return parquet_reader->metadata();
+ }
+
+ void CheckNullCount() {
+ GenerateRange(5, 'a', 'z');
+ WriteToBuffer();
+ auto metadata = LoadWrittenMetadata();
+ auto stats = metadata->RowGroup(0)->ColumnChunk(0)->statistics();
+ ASSERT_TRUE(stats->HasNullCount());
+ ASSERT_EQ(stats->null_count(), 5);
+ ASSERT_EQ(stats->num_values(), 21);
+ }
+
+ void CheckMinMax() {
+ GenerateRange(5, 'a', 'z');
+ WriteToBuffer();
+ auto metadata = LoadWrittenMetadata();
Review comment:
Please generally careful with the use of auto. The [style guide recommends it just of saving space](https://google.github.io/styleguide/cppguide.html#Type_deduction)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671375657
##########
File path: cpp/src/parquet/statistics.h
##########
@@ -291,6 +291,9 @@ class TypedStatistics : public Statistics {
/// arrow::BinaryArray
virtual void Update(const ::arrow::Array& values) = 0;
+ virtual void UpdateArrowDictionary(const ::arrow::Array& indices,
Review comment:
I think making update take an optional null_count parameter actually makes the most sense. Not sure if you are subscribed to dev@parquet but the null_count statistics are incorrect for repeated fields https://issues.apache.org/jira/browse/PARQUET-2067 and I think the fix will involve passing along the null count as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r670979161
##########
File path: cpp/src/parquet/statistics.h
##########
@@ -291,6 +291,9 @@ class TypedStatistics : public Statistics {
/// arrow::BinaryArray
virtual void Update(const ::arrow::Array& values) = 0;
+ virtual void UpdateArrowDictionary(const ::arrow::Array& indices,
Review comment:
I can move the "compute referenced values" logic to the caller so that statistics.cc doesn't depend on ::arrow::compute but I think I still need a second Update method here because the null count comes from the indices array and the min/max from the values array.
Alternatively, I could change the existing Update method to take an optional null_count parameter (where -1 flags to use the old behavior and grab from the array). I think I'll do this latter approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace closed pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace closed pull request #10729:
URL: https://github.com/apache/arrow/pull/10729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#issuecomment-881177183
This also fixes: ARROW-11634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r670963422
##########
File path: cpp/src/parquet/statistics.cc
##########
@@ -567,6 +568,27 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
SetMinMaxPair(comparator_->GetMinMax(values));
}
+ void UpdateArrowDictionary(const ::arrow::Array& indices,
+ const ::arrow::Array& dictionary) {
+ IncrementNullCount(indices.null_count());
+ IncrementNumValues(indices.length() - indices.null_count());
+
+ if (indices.null_count() == indices.length()) {
+ return;
+ }
+
+ ::arrow::compute::ExecContext ctx(pool_);
+ PARQUET_ASSIGN_OR_THROW(auto referenced_indices,
+ ::arrow::compute::Unique(indices, &ctx));
+ PARQUET_ASSIGN_OR_THROW(
Review comment:
this allocate a whole new array? maybe we should file a follow-up JIRA to make this more efficient.
In particular it seems like it could be more efficient to get map of dictionary index to sort order (I think we already have a kernel for this) if necessary each time before calculating the statistics if there are any new entries then iterate through the indices doing a comparison in index space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#issuecomment-887987107
I modified the test to test the case where there are multiple row groups and I added a small optimization (it will only call `compute::Take` if the # of referenced indices is smaller than the size of the dictionary). There is one outstanding comment on column chunk level statistics vs page statistics but if that is resolved (and CI remains green) I will go ahead and merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671593544
##########
File path: cpp/src/parquet/statistics.cc
##########
@@ -567,6 +568,27 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
SetMinMaxPair(comparator_->GetMinMax(values));
}
+ void UpdateArrowDictionary(const ::arrow::Array& indices,
+ const ::arrow::Array& dictionary) {
+ IncrementNullCount(indices.null_count());
+ IncrementNumValues(indices.length() - indices.null_count());
+
+ if (indices.null_count() == indices.length()) {
+ return;
+ }
+
+ ::arrow::compute::ExecContext ctx(pool_);
+ PARQUET_ASSIGN_OR_THROW(auto referenced_indices,
+ ::arrow::compute::Unique(indices, &ctx));
+ PARQUET_ASSIGN_OR_THROW(
Review comment:
First, I'm not sure if it was clear, but there are actually two array allocations here. First it creates a unique array which is a (hopefully small) subset of the indices. Second, it creates a referenced values array which is a (probably not much smaller) subset of the values. Sadly the MinMax functions don't yet support strings. There are kernels (I think) to get a sort order but I can't see how that will help because they don't take selection vectors yet. We could take the output from `Unique` improve Comparator to take a selection vector. This would save the second allocation. However, I'm reluctant to undertake that work when I think the eventual solution will be for the MinMax kernels to support all types. I've created PARQUET-2068 for follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r677952982
##########
File path: cpp/src/parquet/column_writer.cc
##########
@@ -1490,7 +1490,8 @@ Status TypedColumnWriterImpl<DType>::WriteArrowDictionary(
// TODO(wesm): If some dictionary values are unobserved, then the
Review comment:
https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L530 and https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L584 I thought I traced the code an we currently populate them but I could be mistaken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#issuecomment-881802397
@emkornfield Thanks for your quick review. I believe I've addressed your points (except possibly for the WriteIndicesChunk note). I don't think this will make the cutoff for RC0 but hopefully we can get it in if there is a follow-up RC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r670962655
##########
File path: cpp/src/parquet/statistics.h
##########
@@ -291,6 +291,9 @@ class TypedStatistics : public Statistics {
/// arrow::BinaryArray
virtual void Update(const ::arrow::Array& values) = 0;
+ virtual void UpdateArrowDictionary(const ::arrow::Array& indices,
Review comment:
can this be moved to the caller instead. It would be nice to try to avoid more coupling of Arrow with the default parquet library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#issuecomment-881171588
https://issues.apache.org/jira/browse/ARROW-12513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671593084
##########
File path: cpp/src/parquet/statistics.h
##########
@@ -291,6 +291,9 @@ class TypedStatistics : public Statistics {
/// arrow::BinaryArray
virtual void Update(const ::arrow::Array& values) = 0;
+ virtual void UpdateArrowDictionary(const ::arrow::Array& indices,
Review comment:
I ended up just exposing `IncrementNullCount` and `IncrementNumValues` and then added `bool update_counts` to `Update`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r670961610
##########
File path: cpp/src/parquet/statistics.cc
##########
@@ -567,6 +568,27 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
SetMinMaxPair(comparator_->GetMinMax(values));
}
+ void UpdateArrowDictionary(const ::arrow::Array& indices,
+ const ::arrow::Array& dictionary) {
+ IncrementNullCount(indices.null_count());
+ IncrementNumValues(indices.length() - indices.null_count());
+
+ if (indices.null_count() == indices.length()) {
+ return;
+ }
+
+ ::arrow::compute::ExecContext ctx(pool_);
+ PARQUET_ASSIGN_OR_THROW(auto referenced_indices,
Review comment:
please spell out the auto types here and below, its not really clear what the output type is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#issuecomment-881177867
@westonpace this looks ok to me, I think the only real blocker is not putting this in the parquet statistics class but moving to the call-site.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671593696
##########
File path: cpp/src/parquet/arrow/arrow_reader_writer_test.cc
##########
@@ -3796,6 +3796,95 @@ TEST(TestArrowWriterAdHoc, SchemaMismatch) {
ASSERT_RAISES(Invalid, writer->WriteTable(*tbl, 1));
}
+// ----------------------------------------------------------------------
+// Tests for directly writing DictionaryArray
+class TestArrowWriteDictionary : public ::testing::Test {
+ public:
+ void SetUp() override {
+ properties_ = default_arrow_writer_properties();
+ serialized_data_ = AllocateBuffer();
+ }
+
+ // Generates a range of single character strings from start to end (inclusive)
+ // Nulls will be inserted starting at index 0. If there are any null
+ // values then start will not be the true min. However, the dictionary
+ // array will always contain all the values even if they aren't all used.
+ void GenerateRange(int num_nulls, char start, char end) {
Review comment:
Yes, I had grander things in mind and then didn't need them. I simplified the test considerably and added cases for some of the other binary types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#issuecomment-895487924
Forgot about this. Rebasing and then merging on green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r670964526
##########
File path: cpp/src/parquet/arrow/arrow_reader_writer_test.cc
##########
@@ -3796,6 +3796,95 @@ TEST(TestArrowWriterAdHoc, SchemaMismatch) {
ASSERT_RAISES(Invalid, writer->WriteTable(*tbl, 1));
}
+// ----------------------------------------------------------------------
+// Tests for directly writing DictionaryArray
+class TestArrowWriteDictionary : public ::testing::Test {
+ public:
+ void SetUp() override {
+ properties_ = default_arrow_writer_properties();
+ serialized_data_ = AllocateBuffer();
+ }
+
+ // Generates a range of single character strings from start to end (inclusive)
+ // Nulls will be inserted starting at index 0. If there are any null
+ // values then start will not be the true min. However, the dictionary
+ // array will always contain all the values even if they aren't all used.
+ void GenerateRange(int num_nulls, char start, char end) {
Review comment:
this seems overly complex, could you construct the arrays in question with JSON and then combine into a single test case that verifies all of the statistics?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r677953978
##########
File path: cpp/src/parquet/column_writer.cc
##########
@@ -1490,7 +1490,8 @@ Status TypedColumnWriterImpl<DType>::WriteArrowDictionary(
// TODO(wesm): If some dictionary values are unobserved, then the
Review comment:
It will also be important if we ever implement [columindex writing](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L938)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#issuecomment-887990180
LGTM module placement of stats update (inside per page processing or not)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671385913
##########
File path: cpp/src/parquet/column_writer.cc
##########
@@ -1490,7 +1490,8 @@ Status TypedColumnWriterImpl<DType>::WriteArrowDictionary(
// TODO(wesm): If some dictionary values are unobserved, then the
Review comment:
this comment should be deleted. I also think you need to move this into WriteIndicesChunk above (I think you need to adjust your test to have batch size and page size small enough to generate multiple pages to catch this (I actually guess this is a different bug though).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671601430
##########
File path: cpp/src/parquet/arrow/arrow_reader_writer_test.cc
##########
@@ -3796,6 +3796,95 @@ TEST(TestArrowWriterAdHoc, SchemaMismatch) {
ASSERT_RAISES(Invalid, writer->WriteTable(*tbl, 1));
}
+// ----------------------------------------------------------------------
+// Tests for directly writing DictionaryArray
+class TestArrowWriteDictionary : public ::testing::Test {
+ public:
+ void SetUp() override {
+ properties_ = default_arrow_writer_properties();
+ serialized_data_ = AllocateBuffer();
+ }
+
+ // Generates a range of single character strings from start to end (inclusive)
+ // Nulls will be inserted starting at index 0. If there are any null
+ // values then start will not be the true min. However, the dictionary
+ // array will always contain all the values even if they aren't all used.
+ void GenerateRange(int num_nulls, char start, char end) {
+ int num_values = end - start + 1;
+ DCHECK_LT(num_nulls, num_values);
+
+ ::arrow::StringBuilder dictionary_builder;
+ ::arrow::Int32Builder indices_builder;
+
+ for (char val = start; val <= end; val++) {
+ int32_t index = static_cast<int32_t>(val - start);
+ if (val - start >= num_nulls) {
+ ASSERT_OK(indices_builder.Append(index));
+ } else {
+ ASSERT_OK(indices_builder.AppendNull());
+ }
+ ASSERT_OK(dictionary_builder.Append(&val, 1));
+ }
+
+ std::shared_ptr<::arrow::Array> dictionary;
+ std::shared_ptr<::arrow::Array> indices;
+ ASSERT_OK(dictionary_builder.Finish(&dictionary));
+ ASSERT_OK(indices_builder.Finish(&indices));
+
+ ASSERT_OK_AND_ASSIGN(auto test_column,
+ ::arrow::DictionaryArray::FromArrays(indices, dictionary));
+
+ auto schema = ::arrow::schema({::arrow::field("values", test_column->type())});
+ auto batch = ::arrow::RecordBatch::Make(schema, num_values, {test_column});
+ ASSERT_OK_AND_ASSIGN(test_data_, ::arrow::Table::FromRecordBatches({batch}));
+ }
+
+ void WriteToBuffer() {
+ auto out_stream = std::make_shared<::arrow::io::BufferOutputStream>(serialized_data_);
+ auto writer_properties = default_writer_properties();
+ std::unique_ptr<FileWriter> writer;
+ ASSERT_OK(FileWriter::Open(*test_data_->schema(), ::arrow::default_memory_pool(),
+ out_stream, writer_properties,
+ default_arrow_writer_properties(), &writer));
+ ASSERT_OK(writer->WriteTable(*test_data_, std::numeric_limits<int64_t>::max()));
+ ASSERT_OK(writer->Close());
+ }
+
+ std::shared_ptr<FileMetaData> LoadWrittenMetadata() {
+ auto buffer_reader = std::make_shared<::arrow::io::BufferReader>(serialized_data_);
+ auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+ return parquet_reader->metadata();
+ }
+
+ void CheckNullCount() {
+ GenerateRange(5, 'a', 'z');
+ WriteToBuffer();
+ auto metadata = LoadWrittenMetadata();
+ auto stats = metadata->RowGroup(0)->ColumnChunk(0)->statistics();
+ ASSERT_TRUE(stats->HasNullCount());
+ ASSERT_EQ(stats->null_count(), 5);
+ ASSERT_EQ(stats->num_values(), 21);
+ }
+
+ void CheckMinMax() {
+ GenerateRange(5, 'a', 'z');
+ WriteToBuffer();
+ auto metadata = LoadWrittenMetadata();
Review comment:
I think I might be the only person that calls people on it, I think most active developers will use auto alot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r670961210
##########
File path: cpp/src/parquet/statistics.cc
##########
@@ -567,6 +568,27 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
SetMinMaxPair(comparator_->GetMinMax(values));
}
+ void UpdateArrowDictionary(const ::arrow::Array& indices,
+ const ::arrow::Array& dictionary) {
+ IncrementNullCount(indices.null_count());
+ IncrementNumValues(indices.length() - indices.null_count());
+
+ if (indices.null_count() == indices.length()) {
+ return;
+ }
+
+ ::arrow::compute::ExecContext ctx(pool_);
Review comment:
we probably want to turn off threading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r670978233
##########
File path: cpp/src/parquet/statistics.cc
##########
@@ -567,6 +568,27 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
SetMinMaxPair(comparator_->GetMinMax(values));
}
+ void UpdateArrowDictionary(const ::arrow::Array& indices,
+ const ::arrow::Array& dictionary) {
+ IncrementNullCount(indices.null_count());
+ IncrementNumValues(indices.length() - indices.null_count());
+
+ if (indices.null_count() == indices.length()) {
+ return;
+ }
+
+ ::arrow::compute::ExecContext ctx(pool_);
+ PARQUET_ASSIGN_OR_THROW(auto referenced_indices,
+ ::arrow::compute::Unique(indices, &ctx));
+ PARQUET_ASSIGN_OR_THROW(
Review comment:
I'll create a follow-up. Another approach I considered was that we could compute the unique indices and then pass those down as a selection filter to the GetMinMax function. My goal was minimizing change to the existing code.
Or, since we have all these compute kernels now, just use ::arrow::compute::MinMax and pass the min & max into the statistics directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#issuecomment-892173390
CI failures appear unrelated. I'll merge this tomorrow assuming no concerns about https://github.com/apache/arrow/pull/10729#discussion_r681470482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671594372
##########
File path: cpp/src/parquet/column_writer.cc
##########
@@ -1490,7 +1490,8 @@ Status TypedColumnWriterImpl<DType>::WriteArrowDictionary(
// TODO(wesm): If some dictionary values are unobserved, then the
Review comment:
I've deleted the comment but I don't yet understand the point about WriteIndicesChunk. I will need to look at this again later. Is it a performance concern (don't make these extra allocations for the entire array all at once but make the extra allocations many times at the batch level to avoid total memory use) or is it a functional concern (maybe statistics are stored on a per-batch basis?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671592867
##########
File path: cpp/src/parquet/statistics.cc
##########
@@ -567,6 +568,27 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
SetMinMaxPair(comparator_->GetMinMax(values));
}
+ void UpdateArrowDictionary(const ::arrow::Array& indices,
+ const ::arrow::Array& dictionary) {
+ IncrementNullCount(indices.null_count());
+ IncrementNumValues(indices.length() - indices.null_count());
+
+ if (indices.null_count() == indices.length()) {
+ return;
+ }
+
+ ::arrow::compute::ExecContext ctx(pool_);
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671368919
##########
File path: cpp/src/parquet/statistics.cc
##########
@@ -567,6 +568,27 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
SetMinMaxPair(comparator_->GetMinMax(values));
}
+ void UpdateArrowDictionary(const ::arrow::Array& indices,
+ const ::arrow::Array& dictionary) {
+ IncrementNullCount(indices.null_count());
+ IncrementNumValues(indices.length() - indices.null_count());
+
+ if (indices.null_count() == indices.length()) {
+ return;
+ }
+
+ ::arrow::compute::ExecContext ctx(pool_);
+ PARQUET_ASSIGN_OR_THROW(auto referenced_indices,
+ ::arrow::compute::Unique(indices, &ctx));
+ PARQUET_ASSIGN_OR_THROW(
Review comment:
If we have MinMax and it works for dictionary arrays that sounds like a good idea to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r681470482
##########
File path: cpp/src/parquet/column_writer.cc
##########
@@ -1490,7 +1490,8 @@ Status TypedColumnWriterImpl<DType>::WriteArrowDictionary(
// TODO(wesm): If some dictionary values are unobserved, then the
Review comment:
Sorry for the delay. I've expanded the unit test to verify page level statistics. It appears that we recompute the null count in `WriteIndicesChunk` and do so based on `def_levels` which appears to be calculated on a completely different code path (`MultipathLevelBuilderResult`). So the concern about indices vs. dictionary does not apply and the null counts are correct.
I did notice we don't encode page level min/max stats at all. I'm not sure if that is a bug or not (although, if so, I'd tackle that as a separate PR/JIRA). So, if the unit test seems ok and the logic above seems valid then I think this is good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#issuecomment-892031859
Also, it seems we were not writing page statistics at all for data page V2. I added it back in but wasn't sure if that was intentionally disabled for any reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10729: ARROW-12513: [C++][Parquet] Parquet Writer always puts null_count=0 in Parquet statistics for dictionary-encoded array with nulls
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10729:
URL: https://github.com/apache/arrow/pull/10729#discussion_r671592983
##########
File path: cpp/src/parquet/statistics.cc
##########
@@ -567,6 +568,27 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
SetMinMaxPair(comparator_->GetMinMax(values));
}
+ void UpdateArrowDictionary(const ::arrow::Array& indices,
+ const ::arrow::Array& dictionary) {
+ IncrementNullCount(indices.null_count());
+ IncrementNumValues(indices.length() - indices.null_count());
+
+ if (indices.null_count() == indices.length()) {
+ return;
+ }
+
+ ::arrow::compute::ExecContext ctx(pool_);
+ PARQUET_ASSIGN_OR_THROW(auto referenced_indices,
Review comment:
I removed all instances of `auto` unless it was blatantly obvious (e.g. returning from make_shared). Although in this spot it is Datum which isn't much clearer :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org