You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "usberkeley (via GitHub)" <gi...@apache.org> on 2024/11/11 06:25:26 UTC
[PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
usberkeley opened a new pull request, #12231:
URL: https://github.com/apache/hudi/pull/12231
### Change Logs
1. Improve SizeAwareDataInputStream to implement idempotent
2. Improve SizeAwareDataInputStream#skipBytes to avoid potential skip byte count errors
### Impact
Enhance robustness
### Risk level (write none, low medium or high below)
none
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "usberkeley (via GitHub)" <gi...@apache.org>.
usberkeley commented on PR #12231:
URL: https://github.com/apache/hudi/pull/12231#issuecomment-2516148896
@nsivabalan Dear Sivabalan, Could you please help review this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12231:
URL: https://github.com/apache/hudi/pull/12231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12231:
URL: https://github.com/apache/hudi/pull/12231#issuecomment-2467340325
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89494236022ef4f629adf3c60b46a2b35d9bedd3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "usberkeley (via GitHub)" <gi...@apache.org>.
usberkeley commented on PR #12231:
URL: https://github.com/apache/hudi/pull/12231#issuecomment-2487234243
> Can a corrupt `SizeAwareDataInputStream` be reused in other codes?
A corrupt SizeAwareDataInputStream cannot be used in other code because it lacks robustness. Under boundary conditions, skipBytes may go out of bounds, and APIs like readInt are non-idempotent, posing potential risks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12231:
URL: https://github.com/apache/hudi/pull/12231#issuecomment-2467465919
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1566",
"triggerID" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89494236022ef4f629adf3c60b46a2b35d9bedd3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1566)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "usberkeley (via GitHub)" <gi...@apache.org>.
usberkeley commented on code in PR #12231:
URL: https://github.com/apache/hudi/pull/12231#discussion_r1964991386
##########
hudi-common/src/test/java/org/apache/hudi/common/fs/TestSizeAwareDataInputStream.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+/**
+ * Unit tests for the SizeAwareDataInputStream class.
+ */
+public class TestSizeAwareDataInputStream {
+
+ private SizeAwareDataInputStream sizeAwareDataInputStream;
+ private ByteArrayInputStream byteArrayInputStream;
+
+ /**
+ * Initializes the input stream with a predefined byte array.
+ */
+ @BeforeEach
+ void setUp() {
+ byte[] data = {0, 0, 0, 1, 2, 3, 4, 5};
+ byteArrayInputStream = new ByteArrayInputStream(data);
+ sizeAwareDataInputStream = new SizeAwareDataInputStream(new DataInputStream(byteArrayInputStream));
+ }
+
+ /**
+ * Tests the readInt method to ensure it reads an integer correctly and updates the byte count.
+ */
+ @Test
+ void testReadInt() throws IOException {
+ int value = sizeAwareDataInputStream.readInt();
+ assertEquals(1, value);
+ assertEquals(4, sizeAwareDataInputStream.getNumberOfBytesRead());
+ }
+
+ /**
+ * Tests the readFully method to ensure it reads bytes into a buffer and updates the byte count.
+ */
+ @Test
+ void testReadFully() throws IOException {
+ byte[] buffer = new byte[4];
+ sizeAwareDataInputStream.readFully(buffer);
+ assertArrayEquals(new byte[]{0, 0, 0, 1}, buffer);
+ assertEquals(4, sizeAwareDataInputStream.getNumberOfBytesRead());
+ }
+
+ /**
+ * Tests the readFully method with offset to ensure it reads bytes into a specific part of the buffer and updates the byte count.
+ */
+ @Test
+ void testReadFullyWithOffset() throws IOException {
+ byte[] buffer = new byte[6];
+ sizeAwareDataInputStream.readFully(buffer, 2, 4);
+ assertArrayEquals(new byte[]{0, 0, 0, 0, 0, 1}, buffer);
+ assertEquals(4, sizeAwareDataInputStream.getNumberOfBytesRead());
+ }
+
+ /**
+ * Tests the skipBytes method to ensure it skips the correct number of bytes and updates the byte count.
+ */
+ @Test
+ void testSkipBytes() throws IOException {
+ int skipped = sizeAwareDataInputStream.skipBytes(2);
+ assertEquals(2, skipped);
+ assertEquals(2, sizeAwareDataInputStream.getNumberOfBytesRead());
Review Comment:
> Would be good to call `sizeAwareDataInputStream.readInt()` or `sizeAwareDataInputStream.readFully(byte[])` after skipping bytes and validate that the skipping is correct by checking the results of `readInt()` or the byte buffer afterwards.
Got it, thank you very much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12231:
URL: https://github.com/apache/hudi/pull/12231#issuecomment-2469546008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1566",
"triggerID" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"triggerType" : "PUSH"
}, {
"hash" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1581",
"triggerID" : "2469473259",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 89494236022ef4f629adf3c60b46a2b35d9bedd3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1566) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1581)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12231:
URL: https://github.com/apache/hudi/pull/12231#issuecomment-2467342556
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1566",
"triggerID" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89494236022ef4f629adf3c60b46a2b35d9bedd3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1566)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "usberkeley (via GitHub)" <gi...@apache.org>.
usberkeley commented on PR #12231:
URL: https://github.com/apache/hudi/pull/12231#issuecomment-2469473259
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "usberkeley (via GitHub)" <gi...@apache.org>.
usberkeley commented on PR #12231:
URL: https://github.com/apache/hudi/pull/12231#issuecomment-2488568979
> > A corrupt SizeAwareDataInputStream cannot be used in other code
>
> I mean if there is an opportunity for that to happen.
Oh, I see now. The current call chain doesn't trigger it; this is just a code optimization PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12231:
URL: https://github.com/apache/hudi/pull/12231#issuecomment-2469474297
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1566",
"triggerID" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"triggerType" : "PUSH"
}, {
"hash" : "89494236022ef4f629adf3c60b46a2b35d9bedd3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1581",
"triggerID" : "2469473259",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 89494236022ef4f629adf3c60b46a2b35d9bedd3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1566) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1581)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12231:
URL: https://github.com/apache/hudi/pull/12231#issuecomment-2487179617
Can a corrupt `SizeAwareDataInputStream` be reused in other codes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12231:
URL: https://github.com/apache/hudi/pull/12231#issuecomment-2488007874
> A corrupt SizeAwareDataInputStream cannot be used in other code
I mean if there is an opportunity for that to happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8501] Improve SizeAwareDataInputStream to implement idempotent [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12231:
URL: https://github.com/apache/hudi/pull/12231#discussion_r1964985621
##########
hudi-common/src/test/java/org/apache/hudi/common/fs/TestSizeAwareDataInputStream.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+/**
+ * Unit tests for the SizeAwareDataInputStream class.
+ */
+public class TestSizeAwareDataInputStream {
+
+ private SizeAwareDataInputStream sizeAwareDataInputStream;
+ private ByteArrayInputStream byteArrayInputStream;
+
+ /**
+ * Initializes the input stream with a predefined byte array.
+ */
+ @BeforeEach
+ void setUp() {
+ byte[] data = {0, 0, 0, 1, 2, 3, 4, 5};
+ byteArrayInputStream = new ByteArrayInputStream(data);
+ sizeAwareDataInputStream = new SizeAwareDataInputStream(new DataInputStream(byteArrayInputStream));
+ }
+
+ /**
+ * Tests the readInt method to ensure it reads an integer correctly and updates the byte count.
+ */
+ @Test
+ void testReadInt() throws IOException {
+ int value = sizeAwareDataInputStream.readInt();
+ assertEquals(1, value);
+ assertEquals(4, sizeAwareDataInputStream.getNumberOfBytesRead());
+ }
+
+ /**
+ * Tests the readFully method to ensure it reads bytes into a buffer and updates the byte count.
+ */
+ @Test
+ void testReadFully() throws IOException {
+ byte[] buffer = new byte[4];
+ sizeAwareDataInputStream.readFully(buffer);
+ assertArrayEquals(new byte[]{0, 0, 0, 1}, buffer);
+ assertEquals(4, sizeAwareDataInputStream.getNumberOfBytesRead());
+ }
+
+ /**
+ * Tests the readFully method with offset to ensure it reads bytes into a specific part of the buffer and updates the byte count.
+ */
+ @Test
+ void testReadFullyWithOffset() throws IOException {
+ byte[] buffer = new byte[6];
+ sizeAwareDataInputStream.readFully(buffer, 2, 4);
+ assertArrayEquals(new byte[]{0, 0, 0, 0, 0, 1}, buffer);
+ assertEquals(4, sizeAwareDataInputStream.getNumberOfBytesRead());
+ }
+
+ /**
+ * Tests the skipBytes method to ensure it skips the correct number of bytes and updates the byte count.
+ */
+ @Test
+ void testSkipBytes() throws IOException {
+ int skipped = sizeAwareDataInputStream.skipBytes(2);
+ assertEquals(2, skipped);
+ assertEquals(2, sizeAwareDataInputStream.getNumberOfBytesRead());
Review Comment:
Would be good to call `sizeAwareDataInputStream.readInt()` or `sizeAwareDataInputStream.readFully(byte[])` after skipping bytes and validate that the skipping is correct by checking the results of `readInt()` or the byte buffer afterwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org