You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/02/07 11:34:05 UTC
[GitHub] [carbondata] Indhumathi27 opened a new pull request #4090: [WIP] Use CarbonFile API instead of java File API while writing Flink data
Indhumathi27 opened a new pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090
### Why is this PR needed?
### What changes were proposed in this PR?
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
### Is any new testcase added?
- No
- Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4090: [WIP] Use CarbonFile API instead of java File API while writing Flink data
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774700250
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5439/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4090: [CARBONDATA-4122] Support HDFS Carbon writer for Flink Carbon Streaming
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774971494
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5440/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4090: [WIP] Use CarbonFile API instead of java File API while writing Flink data
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774671972
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3677/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4090: [WIP] Use CarbonFile API instead of java File API while writing Flink data
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774701297
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3678/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4090: [CARBONDATA-4122] Use CarbonFile API instead of java File API for Flink CarbonLocalWriter
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#discussion_r573488319
##########
File path: docs/flink-integration-guide.md
##########
@@ -78,7 +78,7 @@ limitations under the License.
val carbonProperties = new Properties
// Set the carbon properties here, such as date format, store location, etc.
- // Create carbon bulk writer factory. Two writer types are supported: 'Local' and 'S3'.
+ // Create carbon bulk writer factory. Three writer types are supported: 'Local', Hdfs' and 'S3'.
Review comment:
Chaned code to use CarbonLocalWriter itself to handle Local and Hdfs FileSystems. Please review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4090: [CARBONDATA-4122] Use CarbonFile API instead of java File API for Flink CarbonLocalWriter
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-776532783
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3689/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #4090: [CARBONDATA-4122] Support HDFS Carbon writer for Flink Carbon Streaming
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#discussion_r572953288
##########
File path: docs/flink-integration-guide.md
##########
@@ -78,7 +78,7 @@ limitations under the License.
val carbonProperties = new Properties
// Set the carbon properties here, such as date format, store location, etc.
- // Create carbon bulk writer factory. Two writer types are supported: 'Local' and 'S3'.
+ // Create carbon bulk writer factory. Three writer types are supported: 'Local', Hdfs' and 'S3'.
Review comment:
Everywhere if we use file factory API and support HDFS conf input, only one writer is enough right ? do we need 3 writers ?
Because in carbon table or SDK we don't create multiple type of writers to handle this kind of scenario.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4090: [CARBONDATA-4122] Support HDFS Carbon writer for Flink Carbon Streaming
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774975863
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3679/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4090: [CARBONDATA-4122] Support HDFS Carbon writer for Flink Carbon Streaming
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#discussion_r572968625
##########
File path: docs/flink-integration-guide.md
##########
@@ -78,7 +78,7 @@ limitations under the License.
val carbonProperties = new Properties
// Set the carbon properties here, such as date format, store location, etc.
- // Create carbon bulk writer factory. Two writer types are supported: 'Local' and 'S3'.
+ // Create carbon bulk writer factory. Three writer types are supported: 'Local', Hdfs' and 'S3'.
Review comment:
yes. i also thought about the same. But since, already they have implemented writers for LOCAL and S3 type, i have implemented for HDFS. But i can see, there are some differences only for S3 writer, some extra configurations are needed and they are not creating directory while writing stage directories in S3. you can check CarbonS3Writer.commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4090: [WIP] Use CarbonFile API instead of java File API while writing Flink data
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774671125
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5438/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #4090: [CARBONDATA-4122] Use CarbonFile API instead of java File API for Flink CarbonLocalWriter
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-776719317
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4090: [CARBONDATA-4122] Use CarbonFile API instead of java File API for Flink CarbonLocalWriter
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4090: [CARBONDATA-4122] Use CarbonFile API instead of java File API for Flink CarbonLocalWriter
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-776532399
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5450/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org