You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/04/15 19:46:43 UTC
[GitHub] [arrow] wjones127 opened a new pull request, #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
wjones127 opened a new pull request, #12901:
URL: https://github.com/apache/arrow/pull/12901
Recently was using this class in a third-party library and had a hard time finding information about these classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #12901:
URL: https://github.com/apache/arrow/pull/12901#discussion_r854486926
##########
python/pyarrow/_parquet.pyx:
##########
@@ -96,72 +98,87 @@ cdef class Statistics(_Weakrefable):
@property
def has_min_max(self):
+ """bool: whether min and max are present"""
Review Comment:
> does numpydoc/napoleon parse it properly?
Either version isn't really parsed into anything special. Hence why I have to have the `` :class:`XXX` `` to get cross references.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12901:
URL: https://github.com/apache/arrow/pull/12901#issuecomment-1100326062
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on code in PR #12901:
URL: https://github.com/apache/arrow/pull/12901#discussion_r853760054
##########
python/pyarrow/_parquet.pyx:
##########
@@ -96,72 +98,87 @@ cdef class Statistics(_Weakrefable):
@property
def has_min_max(self):
+ """bool: whether min and max are present"""
Review Comment:
Maybe doing something like `Whether min and max are present (bool).` is also an option?
(I wanted to add the same comment about this style not being consistent with numpydoc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
wjones127 commented on PR #12901:
URL: https://github.com/apache/arrow/pull/12901#issuecomment-1103277401
I've documented the remaining methods and fixed capitalization and missing periods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #12901:
URL: https://github.com/apache/arrow/pull/12901#discussion_r854492797
##########
python/pyarrow/_parquet.pyx:
##########
@@ -174,14 +218,27 @@ cdef class ParquetLogicalType(_Weakrefable):
cdef init(self, const shared_ptr[const CParquetLogicalType]& type):
self.type = type
+ def __repr__(self):
+ return "{}\n {}".format(object.__repr__(self), str(self))
+
def __str__(self):
return frombytes(self.type.get().ToString(), safe=True)
def to_json(self):
+ """Get a JSON string containing type and type parameters
Review Comment:
```suggestion
"""Get a JSON string containing type and type parameters.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #12901:
URL: https://github.com/apache/arrow/pull/12901#discussion_r853360716
##########
python/pyarrow/_parquet.pyx:
##########
@@ -49,6 +49,8 @@ cimport cpython as cp
cdef class Statistics(_Weakrefable):
+ """Statistics for a single column in a single row group"""
Review Comment:
```suggestion
"""Statistics for a single column in a single row group."""
```
##########
python/pyarrow/_parquet.pyx:
##########
@@ -96,72 +98,87 @@ cdef class Statistics(_Weakrefable):
@property
def has_min_max(self):
+ """bool: whether min and max are present"""
return self.statistics.get().HasMinMax()
@property
def has_null_count(self):
+ """bool: whether null count is present"""
return self.statistics.get().HasNullCount()
@property
def has_distinct_count(self):
+ """bool: whether distinct count is preset"""
return self.statistics.get().HasDistinctCount()
@property
def min_raw(self):
+ """bool, int, float, or bytes: min value as physical type"""
if self.has_min_max:
return _cast_statistic_raw_min(self.statistics.get())
else:
return None
@property
def max_raw(self):
+ """bool, int, float, or bytes: max value as physical type"""
if self.has_min_max:
return _cast_statistic_raw_max(self.statistics.get())
else:
return None
@property
def min(self):
+ """min value as logical type"""
if self.has_min_max:
return _cast_statistic_min(self.statistics.get())
else:
return None
@property
def max(self):
+ """max value as logical type"""
if self.has_min_max:
return _cast_statistic_max(self.statistics.get())
else:
return None
@property
def null_count(self):
+ """int: number of null values in chunk"""
return self.statistics.get().null_count()
@property
def distinct_count(self):
+ """int: distinct number of values in chunk"""
+ # TODO: Why is this always zero? ARROW-11793
return self.statistics.get().distinct_count()
@property
def num_values(self):
+ """int: number of non-null values"""
return self.statistics.get().num_values()
@property
def physical_type(self):
+ """str: physical type of column"""
raw_physical_type = self.statistics.get().physical_type()
return physical_type_name_from_enum(raw_physical_type)
@property
def logical_type(self):
+ """:class:`ParquetLogicalType`: logical type of column"""
return wrap_logical_type(self.statistics.get().descr().logical_type())
@property
def converted_type(self):
+ """str or None: legacy converted type"""
raw_converted_type = self.statistics.get().descr().converted_type()
return converted_type_name_from_enum(raw_converted_type)
cdef class ParquetLogicalType(_Weakrefable):
+ """Logical type of parquet type"""
Review Comment:
```suggestion
"""Logical type of Parquet type."""
```
##########
python/pyarrow/_parquet.pyx:
##########
@@ -392,45 +418,62 @@ cdef class ColumnChunkMetaData(_Weakrefable):
@property
def compression(self):
+ """str: type of compression used for column.
+
+ One of 'UNCOMPRESSED', 'SNAPPY', 'GZIP', 'LZO', 'BROTLI', 'LZ4', 'ZSTD',
+ or 'UNKNOWN'."""
return compression_name_from_enum(self.metadata.compression())
@property
def encodings(self):
+ """tuple of str: encodings used for column
+
+ One of 'PLAIN', 'BIT_PACKED', 'RLE', 'BYTE_STREAM_SPLIT', 'DELTA_BINARY_PACKED',
+ 'DELTA_BYTE_ARRAY'."""
return tuple(map(encoding_name_from_enum, self.metadata.encodings()))
@property
def has_dictionary_page(self):
+ """bool: whether there is dictionary data present in the column chunk"""
return bool(self.metadata.has_dictionary_page())
@property
def dictionary_page_offset(self):
+ """int: offset of dictionary page reglative to column chunk offset"""
if self.has_dictionary_page:
return self.metadata.dictionary_page_offset()
else:
return None
@property
def data_page_offset(self):
+ """int: offset of data page reglative to column chunk offset"""
return self.metadata.data_page_offset()
@property
def has_index_page(self):
+ """Not yet supported"""
raise NotImplementedError('not supported in parquet-cpp')
@property
def index_page_offset(self):
+ """Not yet supported"""
raise NotImplementedError("parquet-cpp doesn't return valid values")
@property
def total_compressed_size(self):
+ """int: compresssed size in bytes"""
return self.metadata.total_compressed_size()
@property
def total_uncompressed_size(self):
+ """int: uncompressed size in bytes"""
return self.metadata.total_uncompressed_size()
cdef class RowGroupMetaData(_Weakrefable):
+ """Metadata for a single row group"""
Review Comment:
```suggestion
"""Metadata for a single row group."""
```
##########
python/pyarrow/_parquet.pyx:
##########
@@ -732,13 +809,25 @@ cdef class ParquetSchema(_Weakrefable):
return self.schema.Equals(deref(other.schema))
def column(self, i):
+ """Return the schema for a single column
+
+ Parameters
+ ----------
+ i : int
+ index of column in schema
+
+ Returns
+ -------
+ column_schema : ColumnSchema
+ """
if i < 0 or i >= len(self):
raise IndexError('{0} out of bounds'.format(i))
return ColumnSchema(self, i)
cdef class ColumnSchema(_Weakrefable):
+ """Schema for a single column"""
Review Comment:
```suggestion
"""Schema for a single column."""
```
##########
python/pyarrow/_parquet.pyx:
##########
@@ -96,72 +98,87 @@ cdef class Statistics(_Weakrefable):
@property
def has_min_max(self):
+ """bool: whether min and max are present"""
Review Comment:
Huh, did not realize numpydoc recommended this style for property docstrings. It looks like the sentences should still end in a period, though?
##########
python/pyarrow/_parquet.pyx:
##########
@@ -609,6 +672,17 @@ cdef class FileMetaData(_Weakrefable):
return None
def row_group(self, int i):
+ """Get metadata for row group at index i
Review Comment:
```suggestion
"""Get metadata for row group at index i.
```
##########
python/pyarrow/_parquet.pyx:
##########
@@ -670,6 +744,8 @@ cdef class FileMetaData(_Weakrefable):
cdef class ParquetSchema(_Weakrefable):
+ """A Parquet schema"""
Review Comment:
```suggestion
"""A Parquet schema."""
```
##########
python/pyarrow/_parquet.pyx:
##########
@@ -481,14 +524,17 @@ cdef class RowGroupMetaData(_Weakrefable):
Review Comment:
There are some other methods above that aren't documented, should we document those?
##########
python/pyarrow/_parquet.pyx:
##########
@@ -504,6 +550,8 @@ def _reconstruct_filemetadata(Buffer serialized):
cdef class FileMetaData(_Weakrefable):
+ """Parquet metadata for a single file"""
Review Comment:
```suggestion
"""Parquet metadata for a single file."""
```
##########
python/pyarrow/_parquet.pyx:
##########
@@ -292,6 +309,8 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
cdef class ColumnChunkMetaData(_Weakrefable):
+ """Column metadata for a single row group"""
Review Comment:
```suggestion
"""Column metadata for a single row group."""
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12901:
URL: https://github.com/apache/arrow/pull/12901#issuecomment-1100326003
https://issues.apache.org/jira/browse/ARROW-16114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #12901:
URL: https://github.com/apache/arrow/pull/12901#discussion_r854488812
##########
python/pyarrow/_parquet.pyx:
##########
@@ -96,72 +98,87 @@ cdef class Statistics(_Weakrefable):
@property
def has_min_max(self):
+ """bool: whether min and max are present"""
Review Comment:
In that case I slightly prefer what Joris has but how it is is also fine (so long as it's punctuated)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #12901:
URL: https://github.com/apache/arrow/pull/12901#discussion_r854386330
##########
python/pyarrow/_parquet.pyx:
##########
@@ -96,72 +98,87 @@ cdef class Statistics(_Weakrefable):
@property
def has_min_max(self):
+ """bool: whether min and max are present"""
Review Comment:
Oh huh.
I think what Joris suggests looks a bit better, but does numpydoc/napoleon parse it properly? If not I'd rather stick with how it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #12901:
URL: https://github.com/apache/arrow/pull/12901#issuecomment-1110309115
['Python', 'R'] benchmarks have high level of regressions.
[ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/459f4204294c41b9a2c874f61bf68161...e9b14301584c4d9d9ec5bd7df77c46a0/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on code in PR #12901:
URL: https://github.com/apache/arrow/pull/12901#discussion_r853762049
##########
python/pyarrow/_parquet.pyx:
##########
@@ -75,6 +77,13 @@ cdef class Statistics(_Weakrefable):
self.converted_type)
def to_dict(self):
+ """Get dictionary represenation of statistics.
Review Comment:
```suggestion
"""
Get dictionary represenation of statistics.
```
Another style nit .., but let's maybe try to do this consistently. I think we mostly start on the next line for multi-line docstring, and below in this diff it's a mix of both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #12901:
URL: https://github.com/apache/arrow/pull/12901#issuecomment-1107654694
Benchmark runs are scheduled for baseline = 3b7c0e29582c8d7cb5180633569eb274fe9e42a0 and contender = 2e7acabf7ba9c4df8621918e82fe795a227a3b31. 2e7acabf7ba9c4df8621918e82fe795a227a3b31 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/8ef3eee637aa4beeb2f0a903c48d572f...16fe70ac5c264e499cce56c24b8e5ccd/)
[Failed] [test-mac-arm](https://conbench.ursa.dev/compare/runs/4fe22fcee72b4b80a49d110d47310e96...5ff4b178d30142488a895bb82ba01e39/)
[Failed :arrow_down:0.75% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/459f4204294c41b9a2c874f61bf68161...e9b14301584c4d9d9ec5bd7df77c46a0/)
[Finished :arrow_down:0.29% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/3fb720450e40435e93d3c9901bc30bc3...6c0d2641c4894670857226310bd264c3/)
Buildkite builds:
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/570| `2e7acabf` ec2-t3-xlarge-us-east-2>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/558| `2e7acabf` test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/556| `2e7acabf` ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/568| `2e7acabf` ursa-thinkcentre-m75q>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/569| `3b7c0e29` ec2-t3-xlarge-us-east-2>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/557| `3b7c0e29` test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/555| `3b7c0e29` ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/567| `3b7c0e29` ursa-thinkcentre-m75q>
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #12901:
URL: https://github.com/apache/arrow/pull/12901#discussion_r854385525
##########
python/pyarrow/_parquet.pyx:
##########
@@ -96,72 +98,87 @@ cdef class Statistics(_Weakrefable):
@property
def has_min_max(self):
+ """bool: whether min and max are present"""
Review Comment:
What Joris suggested is also used by some projects: https://developer.lsst.io/python/numpydoc.html#documenting-class-properties
I'm good with either way. Do we prefer putting the type at the end?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #12901:
URL: https://github.com/apache/arrow/pull/12901#discussion_r853368102
##########
python/pyarrow/_parquet.pyx:
##########
@@ -96,72 +98,87 @@ cdef class Statistics(_Weakrefable):
@property
def has_min_max(self):
+ """bool: whether min and max are present"""
Review Comment:
Yeah I don't think it's adopted anywhere else in Arrow, but it seemed like the best way to do it to me. Using a Return section felt verbose and IIRC doesn't show the type in the attribute summary table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #12901:
URL: https://github.com/apache/arrow/pull/12901#discussion_r854063455
##########
python/pyarrow/_parquet.pyx:
##########
@@ -96,72 +98,87 @@ cdef class Statistics(_Weakrefable):
@property
def has_min_max(self):
+ """bool: whether min and max are present"""
Review Comment:
I actually mean that as-is, this seems to be the recommended numpydoc style, at least as parsed by Napoleon: https://github.com/sphinx-contrib/napoleon/blob/dce30797b7a229ccebda4030f65482d501427794/docs/source/example_numpy.py#L280-L282
It just looks quite different from the rest of numpydoc style.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #12901: ARROW-16114: [Docs][Python] Document Parquet FileMetaData
URL: https://github.com/apache/arrow/pull/12901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org