You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@carbondata.apache.org by ajantha-bhat <gi...@git.apache.org> on 2018/05/09 12:44:29 UTC

[GitHub] carbondata pull request #2289: [CARBONDATA-2435] Remove SDK dependency on sp...

GitHub user ajantha-bhat opened a pull request:

    https://github.com/apache/carbondata/pull/2289

    [CARBONDATA-2435] Remove SDK dependency on spark jars.

    [CARBONDATA-2435] Remove SDK dependency on spark jars.
    Problem and cause : when sdk writer is used in standalone cluster
    without spark jars,
    exception is thrown during reverse dictionary cache initialize time.
    
    Solution: carbon SDK doesn't support dictionary encoding, This spark
    dependency is only for dictionary encoding.
    Move the spark dependency code inside dictionary encoding if block.
    So that SDK flow will not have to access spark class.
    
    Be sure to do all of the following checklist to help us incorporate 
    your contribution quickly and easily:
    
     - [ ] Any interfaces changed? NA
     
     - [ ] Any backward compatibility impacted?NA
     
     - [ ] Document update required?NA
    
     - [ ] Testing done. NA
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ajantha-bhat/carbondata master_new

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2289
    
----
commit 14d95921795aaac4dff9cfe46ffd118e3fdf7388
Author: ajantha-bhat <aj...@...>
Date:   2018-05-09T12:37:56Z

    [CARBONDATA-2435] Remove SDK dependency on spark jars.
    
    Problem and cause : when sdk writer is used in standalone cluster
    without spark jars,
    exception is thrown during reverse dictionary cache initialize time.
    
    Solution: carbon SDK doesn't support dictionary encoding, This spark
    dependency is only for dictionary encoding.
    so, move the spark dependency code inside dictionary encoding if block.
    So, that SDK flow will not have to access spark class.

----


---

[GitHub] carbondata issue #2289: [CARBONDATA-2435] Remove SDK dependency on spark jar...

Posted by ajantha-bhat <gi...@git.apache.org>.

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2289
  
    @ravipesala : please review the PR


---

[GitHub] carbondata issue #2289: [CARBONDATA-2435] Remove SDK dependency on spark jar...

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2289
  
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4615/



---

[GitHub] carbondata issue #2289: [CARBONDATA-2435] Remove SDK dependency on spark jar...

Posted by ravipesala <gi...@git.apache.org>.

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2289
  
    @jackylk In this PR he try to access dictionary cache only in case if any dictionary fileds are there, otherwise it will not initialize any classes of dictionary.


---

[GitHub] carbondata issue #2289: [CARBONDATA-2435] Remove SDK dependency on spark jar...

Posted by jackylk <gi...@git.apache.org>.

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2289
  
    But in this PR there is no modification regarding spark
    Are you looking for `private static String className = "org.apache.spark.util.SizeEstimator";` in ObjectSizeCalculator?


---

[GitHub] carbondata issue #2289: [CARBONDATA-2435] Remove SDK dependency on spark jar...

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2289
  
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5775/



---

[GitHub] carbondata issue #2289: [CARBONDATA-2435] Remove SDK dependency on spark jar...

Posted by ajantha-bhat <gi...@git.apache.org>.

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2289
  
    @jackylk : ObjectSizeCalculator is called during ReverseDictonaryCache init time [please refer the issue callstack in jira] but reverse dictionary cache no need to initialize for sdk flows . Because it is needed only for dictionary encoding. But sdk don't support dictionary encoding. 
    
    So, I have moved the initialize of Reverse dictionary cache inside dictionary encoding check. so sdk flow this will not be called. so, that spark class no need to find. hence we dont get this issue exception


---

[GitHub] carbondata issue #2289: [CARBONDATA-2435] Remove SDK dependency on spark jar...

Posted by ravipesala <gi...@git.apache.org>.

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2289
  
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4828/



---