You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by chandrasaripaka <gi...@git.apache.org> on 2018/12/24 07:26:48 UTC
[GitHub] carbondata pull request #3021: Cdh support
GitHub user chandrasaripaka opened a pull request:
https://github.com/apache/carbondata/pull/3021
Cdh support
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ No] Any interfaces changed?
- [ No] Any backward compatibility impacted?
- [ Yes] Document update required?
- [ Yes] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ No Large Changes] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
This is just only the way we fix for CDH5.14.2, and the Spark2.2.0, the way how parquet treats the data.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/DataHeaps/carbondata cdh-support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/3021.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3021
----
commit 746657e7e47fc69a62dbf14b3a7449108b92342c
Author: Chandrasekhar Saripaka <ch...@...>
Date: 2018-07-25T17:10:20Z
Merge pull request #1 from apache/master
merge back
commit 0c4c5f5d43c779c62864ed7ab99cf9ef91b082f2
Author: Chandrasekhar Saripaka <ch...@...>
Date: 2018-11-23T12:24:19Z
Merge pull request #3 from apache/master
Merge Carbondata master
commit 91d4ee54bfac7bad56dede27059896c766b5f42f
Author: Chandrasekhar Saripaka <ch...@...>
Date: 2018-12-13T07:42:30Z
Merge pull request #4 from apache/master
Updating the fork master
commit 834e702d0e2e8211987c465e36596d78028ea8b2
Author: Chandrasekhar Saripaka <ch...@...>
Date: 2018-12-14T13:36:26Z
Merge pull request #5 from apache/master
Merge Master to DataHeaps Master
commit 2ebfdf3142ee3f22e7938d735ca5a88c117f0848
Author: Chandrasekhar Saripaka <ch...@...>
Date: 2018-12-21T08:51:13Z
Merge pull request #6 from apache/master
Merge Upstream Master
commit 0c777259d54d3b99acd9d62b75cddde7e726afc1
Author: ChandraSaripaka <ch...@...>
Date: 2018-12-24T07:19:30Z
CARBONDATA-3193: Build support for CDH 5.14.2 Spark 2.2.0
----
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by chandrasaripaka <gi...@git.apache.org>.
Github user chandrasaripaka commented on the issue:
https://github.com/apache/carbondata/pull/3021
https://github.com/apache/carbondata/commit/b0733ecbf380d7956dee57a9048dd7537620744e
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by chandrasaripaka <gi...@git.apache.org>.
Github user chandrasaripaka commented on the issue:
https://github.com/apache/carbondata/pull/3021
Please try this:
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-sql_2.11/2.2.0.cloudera2/
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by qiuchenjian <gi...@git.apache.org>.
Github user qiuchenjian commented on the issue:
https://github.com/apache/carbondata/pull/3021
Does carbon not support Cdh5.14.2 spark2.2.0 using -Pspark-2.2 ?
Did CDH change the spark interface ,so that carbon can't run succesfully ?
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3021
I will check on it
On Wed, 2 Jan 2019 at 10:37 PM, Chandrasekhar Saripaka <
notifications@github.com> wrote:
> b0733ec
> <https://github.com/apache/carbondata/commit/b0733ecbf380d7956dee57a9048dd7537620744e>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/carbondata/pull/3021#issuecomment-450922573>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AHwdxlnPiNf3k7TSQcnRFjD_jqqCTHEiks5u_OdkgaJpZM4ZgM5V>
> .
>
--
Thanks & Regards,
Ravi
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by chandrasaripaka <gi...@git.apache.org>.
Github user chandrasaripaka commented on the issue:
https://github.com/apache/carbondata/pull/3021
@chenliang613 , I am ok with it..as in the corporate we still need to livw with it.. we can update the jira and close it.
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3021
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1923/
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3021
@chandrasaripaka I can't find the spark maven dependency for CDH5.14.2, But I am able to build with CDH spark versions `2.2.0-cdh6.0.1` and `2.2.0.cloudera3`. Only the problem here I found is it does not have `spark-hive-thriftserver` jar in cloudera repo, so classes related it like `CarbonThriftServer` and `CarbonSQLCLIDriver` cannot compile. Apart from I am able to compile carbon with `2.2.0-cdh6.0.1` and `2.2.0.cloudera3` versions. I am not sure why CDH does not include `spark-hive-thriftserver` jar in there repo.
Please send the repository for CDH5.14.2, so that I can verify this version also.
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by chandrasaripaka <gi...@git.apache.org>.
Github user chandrasaripaka commented on the issue:
https://github.com/apache/carbondata/pull/3021
Please check the repo here.
https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo_514x.html#maven_5142
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by chandrasaripaka <gi...@git.apache.org>.
Github user chandrasaripaka commented on the issue:
https://github.com/apache/carbondata/pull/3021
Thank you..let's use your pull request and close this...so that we can have a single.place to comment.
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3021
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2228/
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3021
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10206/
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by chandrasaripaka <gi...@git.apache.org>.
Github user chandrasaripaka commented on the issue:
https://github.com/apache/carbondata/pull/3021
We get error from a commit related to altertableschema.. which is not there in cloudera spark
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3021
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2132/
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on the issue:
https://github.com/apache/carbondata/pull/3021
@chandrasaripaka please let us know 3026 if solved your issues?
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by chandrasaripaka <gi...@git.apache.org>.
Github user chandrasaripaka commented on the issue:
https://github.com/apache/carbondata/pull/3021
@jackylk , @ravipesala Can somebody validate this against the CDH5.14.2 and could you please create a profile for build validation for CDH Libraries.. Please let me know where to add..if I need to trigger a different profile build.
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3021
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10176/
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by chandrasaripaka <gi...@git.apache.org>.
Github user chandrasaripaka commented on the issue:
https://github.com/apache/carbondata/pull/3021
@ravipesala , Yes you can let me know how you want to inherit..I think defining a interface for the wrapper would be a good fit.. In our local fork we did it using spark-2.2.0 normal thrift server, it is ok if we dont run thrift server from cloudera distribution, that works.
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3021
I went through this link earlier, but I cannot find spark 2.2 version in this distribution. I can find only `1.6.0-cdh5.14.4` of spark here.
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by chandrasaripaka <gi...@git.apache.org>.
Github user chandrasaripaka commented on the issue:
https://github.com/apache/carbondata/pull/3021
This commit breaks the cloudera 2.2.0cloudera2
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on the issue:
https://github.com/apache/carbondata/pull/3021
@chandrasaripaka
As I know, spark 2.2.0 is not a stable version, it is better to consider other more stable versions.
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3021
@chandrasaripaka Please check the PR https://github.com/apache/carbondata/pull/3026
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3021
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1953/
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3021
@chandrasaripaka , I got the issue, but creating many duplicate files may not be a good idea as it will be difficult to maintain, I will try to do with reflection.
And one more question is I don't find the package `spark-hive-thriftserver_2.11` from cloudera, without this package we cannot run the carbon thrift server. Where to find this package? Or Is it ok if we don't run carbon thrift server from cloudera distribution?
---