You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/08/15 08:00:40 UTC
[GitHub] [carbondata] yutaoChina opened a new pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
yutaoChina opened a new pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892
### write carbon file less than 1M , configured minimum value (dfs.namenode.fs-limits.min-block-size);can't write hdfs file
### the default set block size is 1024 byte and compare with file size;when file size big than 1kb but less than 1M ,can not write file to hdfs ;so set default block size to 2M
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] yutaoChina commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
yutaoChina commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-675874179
>
>
> @yutaoChina : Thanks for working on this.
> a) please handle the compilation error
> b) please create a jira issue and add it in the issue header
this is jira issues
https://issues.apache.org/jira/browse/CARBONDATA-3925
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3892: [CARBONDATA-3925] flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677539801
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2068/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3892: [CARBONDATA-3925] flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-678023200
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2086/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3892: [CARBONDATA-3925] flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677734623
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2080/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677439466
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3805/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3892: [CARBONDATA-3925] flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677668597
@yutaoChina : you can run
`mvn clean install -Pspark-2.3 -Pbuild-with-format -DskipTests` in your IDE to find all findbugs, checkstyle, scalastyle issues.
Now also it has some check style issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3892: [CARBONDATA-3925] flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-678025176
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3827/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-674700072
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2001/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3892: [CARBONDATA-3925] flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677550766
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3809/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677445838
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2064/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677162461
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #3892: [CARBONDATA-3925] flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] yutaoChina commented on a change in pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
yutaoChina commented on a change in pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#discussion_r472681680
##########
File path: integration/flink/src/main/java/org/apache/carbon/core/metadata/StageManager.java
##########
@@ -81,7 +81,7 @@ public static void writeStageInput(final String stageInputPath, final StageInput
private static void writeSuccessFile(final String successFilePath) throws IOException {
final DataOutputStream segmentStatusSuccessOutputStream =
FileFactory.getDataOutputStream(successFilePath,
- CarbonCommonConstants.BYTEBUFFER_SIZE, 1024);
+ CarbonCommonConstants.BYTEBUFFER_SIZE, 1024 * 1024 * 2);
Review comment:
i set it 2M beacuase hdfs (dfs.namenode.fs-limits.min-block-size) configured minimum value size is 1M and in CarbonUtil.java class
`getMaxOfBlockAndFileSize(long blockSize, long fileSize) `method use `long maxSize = blockSize;
if (fileSize > blockSize) {
maxSize = fileSize;
}`
if default size or filesize less than 1M program will get error ;
why 2M ? default is 1M so default * 2 bigger than it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-674365676
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#discussion_r471286842
##########
File path: integration/flink/src/main/java/org/apache/carbon/core/metadata/StageManager.java
##########
@@ -81,7 +81,7 @@ public static void writeStageInput(final String stageInputPath, final StageInput
private static void writeSuccessFile(final String successFilePath) throws IOException {
final DataOutputStream segmentStatusSuccessOutputStream =
FileFactory.getDataOutputStream(successFilePath,
- CarbonCommonConstants.BYTEBUFFER_SIZE, 1024);
+ CarbonCommonConstants.BYTEBUFFER_SIZE, 1024 * 1024 * 2);
Review comment:
what if the file size is greater than 2 MB ? why 2MB selected for this ?
may be need to pass the actual file size ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-674649559
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] yutaoChina commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
yutaoChina commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677487173
>
>
> @yutaoChina : Thanks for working on this.
> a) please handle the compilation error
> b) please create a jira issue and add it in the issue header
my jira id is yutaochina
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-674699373
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3741/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-674706635
@yutaoChina : Thanks for working on this.
a) please handle the compilation error
b) please create a jira issue and add it in the issue header
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3892: [CARBONDATA-3925] flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677736438
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3821/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3892: [CARBONDATA-3925] flink write carbon file to hdfs when file size is less than 1M,can't write
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-678049710
LGTM. Thanks for the contribution!
Spark 2.3.4 build failure is a known random issue, we are working on it. So, merging this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org