You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by jo...@apache.org on 2021/12/16 02:11:05 UTC
[zeppelin] branch master updated: [ZEPPELIN-5594] HDFS fileId, blockSize should be read as "long", not "int".
This is an automated email from the ASF dual-hosted git repository.
jongyoul pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new d85e3fc [ZEPPELIN-5594] HDFS fileId, blockSize should be read as "long", not "int".
d85e3fc is described below
commit d85e3fc50232b36c8a47f9694896013d127d35a2
Author: 이유비 <e....@navercorp.com>
AuthorDate: Mon Nov 29 16:37:43 2021 +0900
[ZEPPELIN-5594] HDFS fileId, blockSize should be read as "long", not "int".
### What is this PR for?
`fileId`, `blockSize` should be declared as `long`, not `int`.
### What type of PR is it?
Bug Fix
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-5594
### How should this be tested?
* Strongly recommended: add automated unit tests for any new or changed behavior
- I modified some value in the `HDFSFileInterpreterTest.java` to simulate integer overflow.
* Outline any manual steps to test the PR here.
- In the `file` interpreter, if you try to access any file with high (> 2147483647) `fileId` or `blockSize`, you will see the exception(`java.lang.NumberFormatException`).
### Screenshots (if appropriate)
```
ERROR [2021-11-29 14:46:51,519] ({FIFOScheduler-org.apache.zeppelin.file.FileInterpreter1871691756-Worker-1} HDFSFileInterpreter.java[listAll]:249) - listall: listDir /user/*******
com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: Expected an int but was 4947954640 at line 1 column 585 path $.FileStatuses.FileStatus[2].fileId
```
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
- I don't think so.
Author: 이유비 <e....@navercorp.com>
Closes #4269 from eubnara/ZEPPELIN-5594 and squashes the following commits:
b6289c8a6 [이유비] HDFS fileId, blockSize should be read as "long", not "int".
---
file/src/main/java/org/apache/zeppelin/file/HDFSFileInterpreter.java | 4 ++--
.../test/java/org/apache/zeppelin/file/HDFSFileInterpreterTest.java | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/file/src/main/java/org/apache/zeppelin/file/HDFSFileInterpreter.java b/file/src/main/java/org/apache/zeppelin/file/HDFSFileInterpreter.java
index 2a13b3c..4df1f64 100644
--- a/file/src/main/java/org/apache/zeppelin/file/HDFSFileInterpreter.java
+++ b/file/src/main/java/org/apache/zeppelin/file/HDFSFileInterpreter.java
@@ -66,9 +66,9 @@ public class HDFSFileInterpreter extends FileInterpreter {
*/
public class OneFileStatus {
public long accessTime;
- public int blockSize;
+ public long blockSize;
public int childrenNum;
- public int fileId;
+ public long fileId;
public String group;
public long length;
public long modificationTime;
diff --git a/file/src/test/java/org/apache/zeppelin/file/HDFSFileInterpreterTest.java b/file/src/test/java/org/apache/zeppelin/file/HDFSFileInterpreterTest.java
index aa69886..10d4453 100644
--- a/file/src/test/java/org/apache/zeppelin/file/HDFSFileInterpreterTest.java
+++ b/file/src/test/java/org/apache/zeppelin/file/HDFSFileInterpreterTest.java
@@ -141,7 +141,7 @@ public class HDFSFileInterpreterTest extends TestCase {
class MockFileSystem {
HashMap<String, String> mfs = new HashMap<>();
static final String FILE_STATUSES =
- "{\"accessTime\":0,\"blockSize\":0,\"childrenNum\":1,\"fileId\":16389," +
+ "{\"accessTime\":0,\"blockSize\":0,\"childrenNum\":1,\"fileId\":4947954640," +
"\"group\":\"hadoop\",\"length\":0,\"modificationTime\":1438548219672," +
"\"owner\":\"yarn\",\"pathSuffix\":\"app-logs\",\"permission\":\"777\"," +
"\"replication\":0,\"storagePolicy\":0,\"type\":\"DIRECTORY\"},\n" +
@@ -184,7 +184,7 @@ class MockFileSystem {
);
mfs.put("/tmp?op=LISTSTATUS",
"{\"FileStatuses\":{\"FileStatus\":[\n" +
- " {\"accessTime\":1441253097489,\"blockSize\":134217728,\"childrenNum\":0," +
+ " {\"accessTime\":1441253097489,\"blockSize\":2147483648,\"childrenNum\":0," +
"\"fileId\":16400,\"group\":\"hdfs\",\"length\":1645," +
"\"modificationTime\":1441253097517,\"owner\":\"hdfs\"," +
"\"pathSuffix\":\"ida8c06540_date040315\",\"permission\":\"755\"," +