You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/02/08 07:22:32 UTC
[GitHub] [ozone] ArafatCloudera opened a new pull request #3054: HDDS-4126. Freon key generator should support >2GB files.
ArafatCloudera opened a new pull request #3054:
URL: https://github.com/apache/ozone/pull/3054
## What changes were proposed in this pull request?
fileSize parameter in HadoopFsGenerator is an int which limits it to max 2GB. But works well with long.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4126
## How was this patch tested?
Na
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatCloudera commented on pull request #3054: HDDS-4126. Freon key generator should support >2GB files.
Posted by GitBox <gi...@apache.org>.
ArafatCloudera commented on pull request #3054:
URL: https://github.com/apache/ozone/pull/3054#issuecomment-1033366386
> @ArafatCloudera I have just noticed that there are a few other cases where the same limit applies:
>
> ```
> hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/StreamingGenerator.java
> 66: private int fileSize;
>
> hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3KeyGenerator.java
> 73: private int fileSize;
>
> hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopDirTreeGenerator.java
> 73: private int fileSizeInBytes;
>
> hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/containergenerator/BaseGenerator.java
> 44: private int keySize;
> ```
>
> Would you mind changing these as well to `long`?
Sure will get it done !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3054: HDDS-4126. Freon key generator should support >2GB files.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #3054:
URL: https://github.com/apache/ozone/pull/3054#issuecomment-1033694147
> I could only make changes for StreamingGenerator.java & HadoopDirTreeGenerator.java
That's fine. Thanks for checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #3054: HDDS-4126. Freon key generator should support >2GB files.
Posted by GitBox <gi...@apache.org>.
adoroszlai merged pull request #3054:
URL: https://github.com/apache/ozone/pull/3054
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatCloudera commented on pull request #3054: HDDS-4126. Freon key generator should support >2GB files.
Posted by GitBox <gi...@apache.org>.
ArafatCloudera commented on pull request #3054:
URL: https://github.com/apache/ozone/pull/3054#issuecomment-1033653289
@adoroszlai I could only make changes for StreamingGenerator.java & HadoopDirTreeGenerator.java
I could not make changes to S3KeyGenerator.java because the **RandomStringUtils.randomAscii(fileSize)** function on **line:122** takes only Integer as an argument.
```
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3KeyGenerator.java
73: private int fileSize;
```
I could not make changes to BaseGenerator.java because a lot of the code would have to be rewritten in other files.
```
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/containergenerator/BaseGenerator.java
44: private int keySize;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatCloudera commented on a change in pull request #3054: HDDS-4126. Freon key generator should support >2GB files.
Posted by GitBox <gi...@apache.org>.
ArafatCloudera commented on a change in pull request #3054:
URL: https://github.com/apache/ozone/pull/3054#discussion_r803629084
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/containergenerator/GeneratorDatanode.java
##########
@@ -256,7 +256,7 @@ public void generateData(long index) throws Exception {
//loop to create multiple chunks per blocks
while (writtenBytes < getKeySize()) {
int currentChunkSize =
- Math.min(getKeySize() - writtenBytes, chunkSize);
+ Math.min(getKeySize() - writtenBytes, chunkSize);
Review comment:
Yea sorry about that, I have removed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3054: HDDS-4126. Freon key generator should support >2GB files.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #3054:
URL: https://github.com/apache/ozone/pull/3054#issuecomment-1036301766
Thanks @ArafatCloudera for the patch, @jojochuang for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #3054: HDDS-4126. Freon key generator should support >2GB files.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #3054:
URL: https://github.com/apache/ozone/pull/3054#discussion_r802594638
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/containergenerator/GeneratorDatanode.java
##########
@@ -256,7 +256,7 @@ public void generateData(long index) throws Exception {
//loop to create multiple chunks per blocks
while (writtenBytes < getKeySize()) {
int currentChunkSize =
- Math.min(getKeySize() - writtenBytes, chunkSize);
+ Math.min(getKeySize() - writtenBytes, chunkSize);
Review comment:
`GeneratorDatanode` only has whitespace change. Can you please omit it from the patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org