You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@carbondata.apache.org by xuchuanyin <gi...@git.apache.org> on 2018/06/19 08:49:56 UTC

[GitHub] carbondata pull request #2382: [CARBONDATA-2513][32K] Support write long str...

GitHub user xuchuanyin opened a pull request:

    https://github.com/apache/carbondata/pull/2382

    [CARBONDATA-2513][32K] Support write long string from dataframe

    Support write long string from dataframe
    
    Sample for usage:
    ```
    longStringDF.write
      .format("carbondata")
      .option("tableName", longStringTable)
      .option("single_pass", "false")
      .option("sort_columns", "name")
      .option("long_string_columns", "description, note")
      .mode(SaveMode.Overwrite)
      .save()
    ```
    
    Be sure to do all of the following checklist to help us incorporate 
    your contribution quickly and easily:
    
     - [x] Any interfaces changed?
     `NO`
     - [ ] Any backward compatibility impacted?
      `NO`
     - [ ] Document update required?
     `NO`
     - [ ] Testing done
            Please provide details on 
            - Whether new unit test cases have been added or why no new tests are required?
    `Tests added`
            - How it is tested? Please attach test report.
    `Tested in local machine`
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. 
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xuchuanyin/carbondata 0620_long_string_dataframe

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2382.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2382
    
----
commit b689d66493521452ff9938415e0d0aa66b56c2c5
Author: xuchuanyin <xu...@...>
Date:   2018-06-02T07:17:04Z

    Support string longer than 32000 characters
    
    Add a table property 'long_string_columns' in create table DDL that
    indicate those columns will contain more than 32000 characters.
    
    Internally in Carbondata,
    1. add a new datatype called `text` to represent the long string column
    2. add a new encoding called `DIRECT_COMPRESS_TEXT` to the text column
    page meta
    3. Use an integer (previously short) to store the length of bytes
    content.

commit f145c6c60238c400b5db6a6bf2696246b698154a
Author: xuchuanyin <xu...@...>
Date:   2018-06-05T12:46:26Z

    rename datatype name from text to varchar

commit 4180f8118d1ff90205b0f1567bef2cdfee3a1b62
Author: xuchuanyin <xu...@...>
Date:   2018-06-12T12:35:58Z

    Add 2GB constraint for one column page

commit 710845b155ed5b7a611a900c70b0d766d80ae48d
Author: xuchuanyin <xu...@...>
Date:   2018-06-14T12:11:40Z

    update tests

commit 74106d2793ed97615a439576b1c16d34bfaa3ab7
Author: xuchuanyin <xu...@...>
Date:   2018-06-19T07:49:57Z

    support write long string from dataframe

----


---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by xuchuanyin <gi...@git.apache.org>.

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    retest this please


---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6403/



---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by kumarvishal09 <gi...@git.apache.org>.

Github user kumarvishal09 commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    LGTM


---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6430/



---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by kumarvishal09 <gi...@git.apache.org>.

Github user kumarvishal09 commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    @xuchuanyin Please rebase


---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5262/



---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5237/



---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by xuchuanyin <gi...@git.apache.org>.

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    rebased


---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6381/



---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6408/



---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by xuchuanyin <gi...@git.apache.org>.

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    retest this please


---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5216/



---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6397/



---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by xuchuanyin <gi...@git.apache.org>.

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    retest it please


---

[GitHub] carbondata issue #2382: [CARBONDATA-2513][32K] Support write long string fro...

Posted by ravipesala <gi...@git.apache.org>.

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2382
  
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5345/



---