You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2025/08/15 11:33:21 UTC
[PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
ArnavBalyan opened a new pull request, #3268:
URL: https://github.com/apache/parquet-java/pull/3268
- Resolves the TODO comment in TestMemPageStore.java by adding proper assertions for page storage/retrieval
- Added some more assertions to verify
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268#issuecomment-3204317322
cc @shangxinli could you please take a look thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268#issuecomment-3243552717
Thanks @ArnavBalyan for polishing it! (TBH I think we should delete this file since it actually tests nothing meaningful)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268#discussion_r2306323480
##########
parquet-column/src/test/java/org/apache/parquet/column/mem/TestMemPageStore.java:
##########
@@ -46,19 +49,42 @@ public void test() throws IOException {
ColumnDescriptor col = new ColumnDescriptor(path, PrimitiveTypeName.INT64, 2, 2);
LongStatistics stats = new LongStatistics();
PageWriter pageWriter = memPageStore.getPageWriter(col);
+
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
+
PageReader pageReader = memPageStore.getPageReader(col);
long totalValueCount = pageReader.getTotalValueCount();
- LOG.info(String.valueOf(totalValueCount));
+ LOG.info("Total value count: " + totalValueCount);
+
+ assertEquals("Expected total value count to be 836 (4 pages * 209 values)", 836, totalValueCount);
+
int total = 0;
+ int pageCount = 0;
do {
DataPage readPage = pageReader.readPage();
+
+ // Assert page was successfully read
+ assertNotNull("Page should not be null", readPage);
+ // Assert page has expected value count
+ assertEquals("Each page should have 209 values", 209, readPage.getValueCount());
+ // Assert encodings when the implementation is DataPageV1
+ if (readPage instanceof DataPageV1) {
Review Comment:
Added the assertion thanks
##########
parquet-column/src/test/java/org/apache/parquet/column/mem/TestMemPageStore.java:
##########
@@ -46,19 +49,42 @@ public void test() throws IOException {
ColumnDescriptor col = new ColumnDescriptor(path, PrimitiveTypeName.INT64, 2, 2);
LongStatistics stats = new LongStatistics();
PageWriter pageWriter = memPageStore.getPageWriter(col);
+
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
+
PageReader pageReader = memPageStore.getPageReader(col);
long totalValueCount = pageReader.getTotalValueCount();
- LOG.info(String.valueOf(totalValueCount));
+ LOG.info("Total value count: " + totalValueCount);
+
+ assertEquals("Expected total value count to be 836 (4 pages * 209 values)", 836, totalValueCount);
+
int total = 0;
+ int pageCount = 0;
do {
DataPage readPage = pageReader.readPage();
+
+ // Assert page was successfully read
+ assertNotNull("Page should not be null", readPage);
+ // Assert page has expected value count
+ assertEquals("Each page should have 209 values", 209, readPage.getValueCount());
+ // Assert encodings when the implementation is DataPageV1
+ if (readPage instanceof DataPageV1) {
+ DataPageV1 v1 = (DataPageV1) readPage;
+ assertEquals("Page repetition level encoding should be BIT_PACKED", BIT_PACKED, v1.getRlEncoding());
+ assertEquals("Page definition level encoding should be BIT_PACKED", BIT_PACKED, v1.getDlEncoding());
+ assertEquals("Page value encoding should be PLAIN", PLAIN, v1.getValueEncoding());
+ }
+
total += readPage.getValueCount();
- LOG.info(readPage.toString());
- // TODO: assert
+ pageCount++;
+ LOG.info("Page " + pageCount + ": " + readPage.toString());
Review Comment:
Removed the redundant log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268#issuecomment-3232107155
cc @wgtmac can you please take a look, have addressed comments thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268#issuecomment-3238938942
cc @wgtmac gentle reminder thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268#issuecomment-3243057386
@wgtmac sorry for the ping, just wanted to remind had addressed all comments could you PTAL thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268#discussion_r2296562982
##########
parquet-column/src/test/java/org/apache/parquet/column/mem/TestMemPageStore.java:
##########
@@ -46,19 +49,42 @@ public void test() throws IOException {
ColumnDescriptor col = new ColumnDescriptor(path, PrimitiveTypeName.INT64, 2, 2);
LongStatistics stats = new LongStatistics();
PageWriter pageWriter = memPageStore.getPageWriter(col);
+
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
+
PageReader pageReader = memPageStore.getPageReader(col);
long totalValueCount = pageReader.getTotalValueCount();
- LOG.info(String.valueOf(totalValueCount));
+ LOG.info("Total value count: " + totalValueCount);
+
+ assertEquals("Expected total value count to be 836 (4 pages * 209 values)", 836, totalValueCount);
+
int total = 0;
+ int pageCount = 0;
do {
DataPage readPage = pageReader.readPage();
+
+ // Assert page was successfully read
+ assertNotNull("Page should not be null", readPage);
+ // Assert page has expected value count
+ assertEquals("Each page should have 209 values", 209, readPage.getValueCount());
+ // Assert encodings when the implementation is DataPageV1
+ if (readPage instanceof DataPageV1) {
+ DataPageV1 v1 = (DataPageV1) readPage;
+ assertEquals("Page repetition level encoding should be BIT_PACKED", BIT_PACKED, v1.getRlEncoding());
+ assertEquals("Page definition level encoding should be BIT_PACKED", BIT_PACKED, v1.getDlEncoding());
+ assertEquals("Page value encoding should be PLAIN", PLAIN, v1.getValueEncoding());
+ }
+
total += readPage.getValueCount();
- LOG.info(readPage.toString());
- // TODO: assert
+ pageCount++;
+ LOG.info("Page " + pageCount + ": " + readPage.toString());
Review Comment:
Sure will remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268#issuecomment-3238939248
cc @wgtmac @shangxinli gentle reminder thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268#discussion_r2296555944
##########
parquet-column/src/test/java/org/apache/parquet/column/mem/TestMemPageStore.java:
##########
@@ -46,19 +49,42 @@ public void test() throws IOException {
ColumnDescriptor col = new ColumnDescriptor(path, PrimitiveTypeName.INT64, 2, 2);
LongStatistics stats = new LongStatistics();
PageWriter pageWriter = memPageStore.getPageWriter(col);
+
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
+
PageReader pageReader = memPageStore.getPageReader(col);
long totalValueCount = pageReader.getTotalValueCount();
- LOG.info(String.valueOf(totalValueCount));
+ LOG.info("Total value count: " + totalValueCount);
+
+ assertEquals("Expected total value count to be 836 (4 pages * 209 values)", 836, totalValueCount);
+
int total = 0;
+ int pageCount = 0;
do {
DataPage readPage = pageReader.readPage();
+
+ // Assert page was successfully read
+ assertNotNull("Page should not be null", readPage);
+ // Assert page has expected value count
+ assertEquals("Each page should have 209 values", 209, readPage.getValueCount());
+ // Assert encodings when the implementation is DataPageV1
+ if (readPage instanceof DataPageV1) {
Review Comment:
I'd prefer asserting that this is a DataPageV1
##########
parquet-column/src/test/java/org/apache/parquet/column/mem/TestMemPageStore.java:
##########
@@ -46,19 +49,42 @@ public void test() throws IOException {
ColumnDescriptor col = new ColumnDescriptor(path, PrimitiveTypeName.INT64, 2, 2);
LongStatistics stats = new LongStatistics();
PageWriter pageWriter = memPageStore.getPageWriter(col);
+
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
+
PageReader pageReader = memPageStore.getPageReader(col);
long totalValueCount = pageReader.getTotalValueCount();
- LOG.info(String.valueOf(totalValueCount));
+ LOG.info("Total value count: " + totalValueCount);
+
+ assertEquals("Expected total value count to be 836 (4 pages * 209 values)", 836, totalValueCount);
+
int total = 0;
+ int pageCount = 0;
do {
DataPage readPage = pageReader.readPage();
+
+ // Assert page was successfully read
+ assertNotNull("Page should not be null", readPage);
+ // Assert page has expected value count
+ assertEquals("Each page should have 209 values", 209, readPage.getValueCount());
+ // Assert encodings when the implementation is DataPageV1
+ if (readPage instanceof DataPageV1) {
+ DataPageV1 v1 = (DataPageV1) readPage;
+ assertEquals("Page repetition level encoding should be BIT_PACKED", BIT_PACKED, v1.getRlEncoding());
+ assertEquals("Page definition level encoding should be BIT_PACKED", BIT_PACKED, v1.getDlEncoding());
+ assertEquals("Page value encoding should be PLAIN", PLAIN, v1.getValueEncoding());
+ }
+
total += readPage.getValueCount();
- LOG.info(readPage.toString());
- // TODO: assert
+ pageCount++;
+ LOG.info("Page " + pageCount + ": " + readPage.toString());
Review Comment:
Should we totally remove the LOG? It is useless at this moment.
##########
parquet-column/src/test/java/org/apache/parquet/column/mem/TestMemPageStore.java:
##########
@@ -46,19 +49,42 @@ public void test() throws IOException {
ColumnDescriptor col = new ColumnDescriptor(path, PrimitiveTypeName.INT64, 2, 2);
LongStatistics stats = new LongStatistics();
PageWriter pageWriter = memPageStore.getPageWriter(col);
+
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
+
PageReader pageReader = memPageStore.getPageReader(col);
long totalValueCount = pageReader.getTotalValueCount();
- LOG.info(String.valueOf(totalValueCount));
+ LOG.info("Total value count: " + totalValueCount);
+
+ assertEquals("Expected total value count to be 836 (4 pages * 209 values)", 836, totalValueCount);
+
int total = 0;
+ int pageCount = 0;
do {
DataPage readPage = pageReader.readPage();
+
+ // Assert page was successfully read
Review Comment:
It seems that TestMemPageStore previously tests nothing but merely to prove that methods of MemPageStore are working.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268#issuecomment-3206753964
cc @wgtmac could you pls also take a look thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3267: Add comprehensive assertions to TestMemPageStore [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #3268:
URL: https://github.com/apache/parquet-java/pull/3268#discussion_r2306337358
##########
parquet-column/src/test/java/org/apache/parquet/column/mem/TestMemPageStore.java:
##########
@@ -46,19 +49,42 @@ public void test() throws IOException {
ColumnDescriptor col = new ColumnDescriptor(path, PrimitiveTypeName.INT64, 2, 2);
LongStatistics stats = new LongStatistics();
PageWriter pageWriter = memPageStore.getPageWriter(col);
+
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[743]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
pageWriter.writePage(BytesInput.from(new byte[735]), 209, stats, BIT_PACKED, BIT_PACKED, PLAIN);
+
PageReader pageReader = memPageStore.getPageReader(col);
long totalValueCount = pageReader.getTotalValueCount();
- LOG.info(String.valueOf(totalValueCount));
+ LOG.info("Total value count: " + totalValueCount);
+
+ assertEquals("Expected total value count to be 836 (4 pages * 209 values)", 836, totalValueCount);
+
int total = 0;
+ int pageCount = 0;
do {
DataPage readPage = pageReader.readPage();
+
+ // Assert page was successfully read
Review Comment:
Yes seems like previously`MemPageStore` was just supposed to test the MemPageStore upper level functionality. Seems that `TestMemColumn` does the actual data and lower level checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org