You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "gszadovszky (via GitHub)" <gi...@apache.org> on 2024/02/13 12:47:32 UTC
[PR] PARQUET-2431: Handle ByteBufferAllocator gracefully [parquet-mr]
gszadovszky opened a new pull request, #1274:
URL: https://github.com/apache/parquet-mr/pull/1274
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [PARQUET-2431](https://issues.apache.org/jira/browse/PARQUET-2431) issues and references
them in the PR title. For example, "PARQUET-1234: My Parquet PR"
- https://issues.apache.org/jira/browse/PARQUET-XXX
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines
from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Style
- [x] My contribution adheres to the code style guidelines and Spotless passes.
- To apply the necessary changes, run `mvn spotless:apply -Pvector-plugins`
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain Javadoc that explain what it does
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] PARQUET-2431: Handle ByteBufferAllocator gracefully [parquet-mr]
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky merged PR #1274:
URL: https://github.com/apache/parquet-mr/pull/1274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] PARQUET-2431: Handle ByteBufferAllocator gracefully [parquet-mr]
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #1274:
URL: https://github.com/apache/parquet-mr/pull/1274#discussion_r1494126621
##########
parquet-column/src/main/java/org/apache/parquet/column/impl/ColumnReaderBase.java:
##########
@@ -729,7 +729,7 @@ private void initDataReader(Encoding dataEncoding, ByteBufferInputStream in, int
if (CorruptDeltaByteArrays.requiresSequentialReads(writerVersion, dataEncoding)
&& previousReader != null
- && previousReader instanceof RequiresPreviousReader) {
+ && dataColumn instanceof RequiresPreviousReader) {
Review Comment:
Just reading the code :)
This is about handling a very old bug at writing parquet files. I don't think we have too many files with this error out there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] PARQUET-2431: Handle ByteBufferAllocator gracefully [parquet-mr]
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #1274:
URL: https://github.com/apache/parquet-mr/pull/1274#discussion_r1494135917
##########
parquet-common/src/main/java/org/apache/parquet/bytes/MultiBufferInputStream.java:
##########
@@ -43,15 +42,15 @@ class MultiBufferInputStream extends ByteBufferInputStream {
private List<ByteBuffer> markBuffers = new ArrayList<>();
MultiBufferInputStream(List<ByteBuffer> buffers) {
- this.buffers = buffers;
+ List<ByteBuffer> buffersCopy = new ArrayList<>(buffers);
Review Comment:
TBH, this is not needed. It was kind of automatic to copy the list instead of using the one got from outside. But the whole point of these classes is performance. So I'll revert this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] PARQUET-2431: Handle ByteBufferAllocator gracefully [parquet-mr]
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on PR #1274:
URL: https://github.com/apache/parquet-mr/pull/1274#issuecomment-1951996474
Thank you, @wgtmac!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] PARQUET-2431: Handle ByteBufferAllocator gracefully [parquet-mr]
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on PR #1274:
URL: https://github.com/apache/parquet-mr/pull/1274#issuecomment-1943280473
@shangxinli, if you may have some time, could you check it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] PARQUET-2431: Handle ByteBufferAllocator gracefully [parquet-mr]
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #1274:
URL: https://github.com/apache/parquet-mr/pull/1274#discussion_r1494133508
##########
parquet-common/src/main/java/org/apache/parquet/bytes/TrackingByteBufferAllocator.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.bytes;
+
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * A wrapper {@link ByteBufferAllocator} implementation that tracks whether all allocated buffers are released. It
+ * throws the related exception at {@link #close()} if any buffer remains un-released. It also clears the buffers at
+ * release so if they continued being used it'll generate errors.
+ * <p>To be used for testing purposes.
Review Comment:
How would we differentiate test and prod envs? I would not like to put warns in the test env either.
One would need to explicitly define a `ByteBufferAllocator` instance to use it in prod. I would expect reading the specs before using a class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org