You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/03/12 13:02:44 UTC
[GitHub] [carbondata] marchpure opened a new pull request #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
marchpure opened a new pull request #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667
### Why is this PR needed?
Input type of Spark's FilePartition will change from Seq to Array.
We shall create filepartition by entering ARRAY type, which is compatible with both Array and Seq.
### What changes were proposed in this PR?
create filepartition by entering ARRAY type, instead of entering Seq type.
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598329978
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/738/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598178267
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2441/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598325099
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2446/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request
#3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#discussion_r394295019
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/load/CsvRDDHelper.scala
##########
@@ -20,7 +20,7 @@ package org.apache.carbondata.spark.load
import java.text.SimpleDateFormat
import java.util.{Date, Locale}
-import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.{ArrayBuffer, WrappedArray}
Review comment:
Revert this change
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request
#3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#discussion_r394294843
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/load/CsvRDDHelper.scala
##########
@@ -94,7 +93,7 @@ object CsvRDDHelper {
def closePartition(): Unit = {
if (currentFiles.nonEmpty) {
val newPartition =
- CarbonToSparkAdapter.createFilePartition(
+ FilePartition(
Review comment:
Should be FilePartition( partitions.size, currentFiles.toArray)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3667:
[HOTFIX] Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#discussion_r392010381
##########
File path: integration/spark/src/main/spark2.3/org/apache/carbondata/spark/adapter/CarbonToSparkAdapter.scala
##########
@@ -17,12 +17,12 @@
package org.apache.carbondata.spark.adapter
-import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.{ArrayBuffer, WrappedArray}
import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
object CarbonToSparkAdapter {
def createFilePartition(index: Int, files: ArrayBuffer[PartitionedFile]) = {
- FilePartition(index, files.toArray.toSeq)
Review comment:
better to modify CsvRDDHelper directly and remove adapter/CarbonToSparkAdapter.scala （both 2.3 and 2.4）
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-599043991
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/762/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-599051312
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2470/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598230507
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2444/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-599007584
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2465/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-599028318
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2469/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-599028184
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/761/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598245231
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/736/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-605810388
@marchpure : please reply to the comments and get it merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3667: [HOTFIX]
Enhanced compatibility with Spark's FilePartition interface
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598997172
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/757/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services