You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/04/13 22:52:57 UTC

[GitHub] [incubator-hudi] lamber-ken opened a new pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

lamber-ken opened a new pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file
URL: https://github.com/apache/incubator-hudi/pull/1512
 
 
   ## What is the purpose of the pull request
   
   In order to support multiple storage formats, store `hoodie.table.base.file.format` to hoodie.properties.
   
   - `hoodie.table.base.file.format`: default `PARQUET`
   
   ## Brief change log
   
     - Store `hoodie.table.base.file.format` to hoodie.properties.
   
   ## Verify this pull request
   
   This pull request is a minal rework.
   
   ## Committer checklist
   
    - [X] Has a corresponding JIRA in PR title & commit
    
    - [X] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-615885170
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=h1) Report
   > Merging [#1512](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/75523657a4b5c67209e48c1c989a4dd017b4fdd1&el=desc) will **increase** coverage by `0.06%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1512/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1512      +/-   ##
   ============================================
   + Coverage     72.28%   72.35%   +0.06%     
   - Complexity      294      296       +2     
   ============================================
     Files           374      374              
     Lines         16365    16378      +13     
     Branches       1648     1649       +1     
   ============================================
   + Hits          11830    11850      +20     
   + Misses         3804     3798       -6     
   + Partials        731      730       -1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.44% <ø> (ø)` | `37.00 <0.00> (ø)` | |
   | [...org/apache/hudi/common/model/HoodieFileFormat.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUZpbGVGb3JtYXQuamF2YQ==) | `100.00% <100.00%> (ø)` | `2.00 <0.00> (+2.00)` | |
   | [...rg/apache/hudi/common/table/HoodieTableConfig.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlQ29uZmlnLmphdmE=) | `50.00% <100.00%> (+1.56%)` | `0.00 <0.00> (ø)` | |
   | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `84.17% <100.00%> (+0.41%)` | `0.00 <0.00> (ø)` | |
   | [...main/scala/org/apache/hudi/DataSourceOptions.scala](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvRGF0YVNvdXJjZU9wdGlvbnMuc2NhbGE=) | `93.54% <100.00%> (+0.28%)` | `0.00 <0.00> (ø)` | |
   | [...n/scala/org/apache/hudi/HoodieSparkSqlWriter.scala](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvSG9vZGllU3BhcmtTcWxXcml0ZXIuc2NhbGE=) | `53.37% <100.00%> (+0.57%)` | `0.00 <0.00> (ø)` | |
   | [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `78.24% <100.00%> (+0.10%)` | `11.00 <0.00> (ø)` | |
   | [...src/main/java/org/apache/hudi/metrics/Metrics.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9NZXRyaWNzLmphdmE=) | `72.22% <0.00%> (+13.88%)` | `0.00% <0.00%> (ø%)` | |
   | [...g/apache/hudi/metrics/InMemoryMetricsReporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9Jbk1lbW9yeU1ldHJpY3NSZXBvcnRlci5qYXZh) | `80.00% <0.00%> (+40.00%)` | `0.00% <0.00%> (ø%)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=footer). Last update [7552365...6a2fdaf](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hudi] vinothchandar commented on pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1512:
URL: https://github.com/apache/hudi/pull/1512#issuecomment-650699549


   @prashantwason @bvaradar IIUC some of this PR overlaps with the changes you are making as well. can you both clarify so we can close or revive this as needed..  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar closed pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
vinothchandar closed pull request #1512:
URL: https://github.com/apache/hudi/pull/1512


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] bvaradar commented on pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #1512:
URL: https://github.com/apache/hudi/pull/1512#issuecomment-650710398


   yes @vinothchandar , this overlaps with PR-1687 which got merged. PR-1687 exposes setting base-file format through deltastreamer and spark data-source writer. 
   
    @lamber-ken : Can you check current master branch and see if you still need any other functionality?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-615885170
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=h1) Report
   > Merging [#1512](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/75523657a4b5c67209e48c1c989a4dd017b4fdd1&el=desc) will **increase** coverage by `0.06%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1512/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1512      +/-   ##
   ============================================
   + Coverage     72.28%   72.35%   +0.06%     
   - Complexity      294      296       +2     
   ============================================
     Files           374      374              
     Lines         16365    16378      +13     
     Branches       1648     1649       +1     
   ============================================
   + Hits          11830    11850      +20     
   + Misses         3804     3798       -6     
   + Partials        731      730       -1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.44% <ø> (ø)` | `37.00 <0.00> (ø)` | |
   | [...org/apache/hudi/common/model/HoodieFileFormat.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUZpbGVGb3JtYXQuamF2YQ==) | `100.00% <100.00%> (ø)` | `2.00 <0.00> (+2.00)` | |
   | [...rg/apache/hudi/common/table/HoodieTableConfig.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlQ29uZmlnLmphdmE=) | `50.00% <100.00%> (+1.56%)` | `0.00 <0.00> (ø)` | |
   | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `84.17% <100.00%> (+0.41%)` | `0.00 <0.00> (ø)` | |
   | [...main/scala/org/apache/hudi/DataSourceOptions.scala](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvRGF0YVNvdXJjZU9wdGlvbnMuc2NhbGE=) | `93.54% <100.00%> (+0.28%)` | `0.00 <0.00> (ø)` | |
   | [...n/scala/org/apache/hudi/HoodieSparkSqlWriter.scala](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvSG9vZGllU3BhcmtTcWxXcml0ZXIuc2NhbGE=) | `53.37% <100.00%> (+0.57%)` | `0.00 <0.00> (ø)` | |
   | [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `78.24% <100.00%> (+0.10%)` | `11.00 <0.00> (ø)` | |
   | [...src/main/java/org/apache/hudi/metrics/Metrics.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9NZXRyaWNzLmphdmE=) | `72.22% <0.00%> (+13.88%)` | `0.00% <0.00%> (ø%)` | |
   | [...g/apache/hudi/metrics/InMemoryMetricsReporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1512/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9Jbk1lbW9yeU1ldHJpY3NSZXBvcnRlci5qYXZh) | `80.00% <0.00%> (+40.00%)` | `0.00% <0.00%> (ø%)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=footer). Last update [7552365...6a2fdaf](https://codecov.io/gh/apache/incubator-hudi/pull/1512?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-614308722
 
 
   Fair enough.. Hypothetically, we could support a world where base files across partitions are in different formats.. So given its not a real problem today.. we can deal with it down the line? 
   
   btw there are also things like index.type which should go there today and are not.. IIRC there is a ticket for that as well.. main challenge there is how would we apply this change to the hoodie.properties file.. (dont think overwriting will be atomic...) also does this change also fix older tables automatically.. If we can flesh those out, that may be good. wdyt 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-613225596
 
 
   @lamber-ken trying to understand why this should be needed.. can't we distinguish based on the input format for ORC, where we can create HoodieOrcInputFormat .. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken edited a comment on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
lamber-ken edited a comment on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-614255323
 
 
   hi @vinothchandar @pratyakshsharma, thanks very much for reviewing this. 
   
   The base file format for each hudi dataset can't be changed across writes, so we need to store this option to hoodie.properties. This is the first step to support multiple data formats (read / write options)
   
   ![image](https://user-images.githubusercontent.com/20113411/79383212-63d64f00-7f97-11ea-8fc1-9fdfe34a911d.png)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-615908872
 
 
   Hi @vinothchandar, apologies for the delay.
   
   > different partitions use different formats
   
   I know hive support that feature, it's easy for hive. If we want to support different partitions use different format, we alos need to store these partition file format infomation to `hoodie.properties` file as shown bellow, these are required when read / write.
   
   At present, I can't think of a better way, correct me if I'm wrong or do you have any good ideas?
   ```
   // hoodie.properties
   hoodie.table.name=hudi_mor_table
   hoodie.archivelog.folder=archived
   hoodie.table.type=COPY_ON_WRITE
   hoodie.timeline.layout.version=0
   
   hoodie.table.base.file.format.partition.2020.01.08=ORC
   hoodie.table.base.file.format.partition.2020.01.09=PARQUET
   hoodie.table.base.file.format.partition.2020.01.10=PARQUET
   hoodie.table.base.file.format.partition.2020.01.11=ORC
   hoodie.table.base.file.format.partition.2020.01.12=ORC
   
   
   // sync to hive, need to set fileformat
   ALTER TABLE day_table PARTITION (dt='20200805') SET FILEFORMAT rcfile;
   ```
   
   <br>
   
   > store things like index.type ..
   
   I found a jira about it, "Persist the HoodieIndex type in the hoodie.properties file" [1],
   It seems to be another topic : )
   
   [1] https://issues.apache.org/jira/browse/HUDI-37
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-614255323
 
 
   hi @vinothchandar @pratyakshsharma, thanks very much for reviewing this. 
   
   The base file format for each hudi dataset can't be changed across writes, so we need to store this option to hoodie.properties.
   
   ![image](https://user-images.githubusercontent.com/20113411/79383212-63d64f00-7f97-11ea-8fc1-9fdfe34a911d.png)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] pratyakshsharma commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-613673881
 
 
   > @lamber-ken trying to understand why this should be needed.. can't we distinguish based on the input format for ORC, where we can create HoodieOrcInputFormat ..
   
   +1 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1512:
URL: https://github.com/apache/incubator-hudi/pull/1512#discussion_r416452010



##########
File path: hudi-spark/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -142,6 +143,16 @@ object DataSourceWriteOptions {
   val MOR_TABLE_TYPE_OPT_VAL = HoodieTableType.MERGE_ON_READ.name
   val DEFAULT_TABLE_TYPE_OPT_VAL = COW_TABLE_TYPE_OPT_VAL
 
+  /**
+    * The table base file format for the underlying data, for this write.
+    * Note that this can't change across writes.
+    *
+    * Default: PARQUET
+    */
+  val TABLE_FILE_FORMAT_OPT_KEY = "hoodie.table.base.file.format"

Review comment:
       Done, thanks




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1512:
URL: https://github.com/apache/incubator-hudi/pull/1512#discussion_r413255248



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileFormat.java
##########
@@ -22,7 +22,7 @@
  * Hoodie file format.
  */
 public enum HoodieFileFormat {
-  PARQUET(".parquet"), HOODIE_LOG(".log");
+  PARQUET(".parquet"), HOODIE_LOG(".log"), ORC(".orc");

Review comment:
       Reasonable, done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] lamber-ken commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1512:
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-617055212


   All review comments are addressed and fixed, thanks you all.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1512:
URL: https://github.com/apache/incubator-hudi/pull/1512#discussion_r411589627



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -131,6 +131,9 @@ public static void createHoodieProperties(FileSystem fs, Path metadataFolder, Pr
         // Use latest Version as default unless forced by client
         properties.setProperty(HOODIE_TIMELINE_LAYOUT_VERSION, TimelineLayoutVersion.CURR_VERSION.toString());
       }
+      if (!properties.containsKey(HOODIE_BASE_FILE_FORMAT_PROP_NAME)) {
+        properties.setProperty(HOODIE_BASE_FILE_FORMAT_PROP_NAME, DEFAULT_BASE_FILE_FORMAT.name());
+      }

Review comment:
       hi @bvaradar, it already exists in master branch.
   ```
   // HoodieTableConfig#getBaseFileFormat
   
   /**
    * Get the base file storage format.
    *
    * @return HoodieFileFormat for the base file Storage format
    */
   public HoodieFileFormat getBaseFileFormat() {
     if (props.containsKey(HOODIE_BASE_FILE_FORMAT_PROP_NAME)) {
       return HoodieFileFormat.valueOf(props.getProperty(HOODIE_BASE_FILE_FORMAT_PROP_NAME));
     }
     if (props.containsKey(HOODIE_RO_FILE_FORMAT_PROP_NAME)) {
       return HoodieFileFormat.valueOf(props.getProperty(HOODIE_RO_FILE_FORMAT_PROP_NAME));
     }
     return DEFAULT_BASE_FILE_FORMAT;
   }
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1512:
URL: https://github.com/apache/incubator-hudi/pull/1512#discussion_r413254988



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -190,6 +190,9 @@ public Operation convert(String value) throws ParameterException {
     @Parameter(names = {"--table-type"}, description = "Type of table. COPY_ON_WRITE (or) MERGE_ON_READ", required = true)
     public String tableType;
 
+    @Parameter(names = {"--table-file-format"}, description = "BaseFileFormat of table. PARQUET (or) ORC")

Review comment:
       Reasonable, done

##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HDFSParquetImporter.java
##########
@@ -251,6 +252,8 @@ public void validate(String name, String value) {
     public String tableName = null;
     @Parameter(names = {"--table-type", "-tt"}, description = "Table type", required = true)
     public String tableType = null;
+    @Parameter(names = {"--table-file-format", "-tff"}, description = "The base file storage format")

Review comment:
       Reasonable, done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] bvaradar commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1512:
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-616631319


   @lamber-ken : Storing per-partition specific metadata in hoodie.properties wont work as we are not versioning hoodie.properties. There is no atomicity guarantees across different cloud storages   for writers. 
   
   I think supporting different file formats within a table is not a priority but if we have to do, then we can instead store in .hoodie_partition_metadata first time we create it per partition. Hudi Hive Sync needs to then read this for each new partition getting added to hive to register the correct input format for that partition. I am not sure how Spark, Presto and Impala would work in this case. We need to evaluate before venturing out to supporting this.
   
   Coming back to the original objective of this PR, since hoodie.properties is effectively write-once, we can use it to store default file format of the hoodie table. As part of HoodieTableMetaClient.initTableType alone, we should set the file format of table once. TO support existing tables,  we can keep the default storage layout as "PARQUET" and if the setting is not present in hoodie.properties (for already created tables), we should use "PARQUET" as default.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1512:
URL: https://github.com/apache/incubator-hudi/pull/1512#discussion_r416109136



##########
File path: hudi-spark/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -142,6 +143,16 @@ object DataSourceWriteOptions {
   val MOR_TABLE_TYPE_OPT_VAL = HoodieTableType.MERGE_ON_READ.name
   val DEFAULT_TABLE_TYPE_OPT_VAL = COW_TABLE_TYPE_OPT_VAL
 
+  /**
+    * The table base file format for the underlying data, for this write.
+    * Note that this can't change across writes.
+    *
+    * Default: PARQUET
+    */
+  val TABLE_FILE_FORMAT_OPT_KEY = "hoodie.table.base.file.format"

Review comment:
       sorry did not catch this before.. so can we introduce this at the `WriteConfig` level, that way it's common for both datasource and deltastreamer.. 
   Any write config can be passed through the datasource anyway




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1512:
URL: https://github.com/apache/incubator-hudi/pull/1512#discussion_r411480163



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -131,6 +131,9 @@ public static void createHoodieProperties(FileSystem fs, Path metadataFolder, Pr
         // Use latest Version as default unless forced by client
         properties.setProperty(HOODIE_TIMELINE_LAYOUT_VERSION, TimelineLayoutVersion.CURR_VERSION.toString());
       }
+      if (!properties.containsKey(HOODIE_BASE_FILE_FORMAT_PROP_NAME)) {
+        properties.setProperty(HOODIE_BASE_FILE_FORMAT_PROP_NAME, DEFAULT_BASE_FILE_FORMAT.name());
+      }

Review comment:
       Can you also add a getBaseFileFormat() API which returns PARQUET if not set in hoodie.properties.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1512:
URL: https://github.com/apache/incubator-hudi/pull/1512#discussion_r411783820



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileFormat.java
##########
@@ -22,7 +22,7 @@
  * Hoodie file format.
  */
 public enum HoodieFileFormat {
-  PARQUET(".parquet"), HOODIE_LOG(".log");
+  PARQUET(".parquet"), HOODIE_LOG(".log"), ORC(".orc");

Review comment:
       again, this can happen in a PR, that actually adds ORC support.. not now ?>

##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -190,6 +190,9 @@ public Operation convert(String value) throws ParameterException {
     @Parameter(names = {"--table-type"}, description = "Type of table. COPY_ON_WRITE (or) MERGE_ON_READ", required = true)
     public String tableType;
 
+    @Parameter(names = {"--table-file-format"}, description = "BaseFileFormat of table. PARQUET (or) ORC")

Review comment:
       adding something like this without actual ORC support feel bit premature and misleading to me.. for e,g if we release the code in few weeks, deltastreams help will be very misleading.. let's wait till we have datasource or some level of progress with ORC atleast ? 

##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HDFSParquetImporter.java
##########
@@ -251,6 +252,8 @@ public void validate(String name, String value) {
     public String tableName = null;
     @Parameter(names = {"--table-type", "-tt"}, description = "Table type", required = true)
     public String tableType = null;
+    @Parameter(names = {"--table-file-format", "-tff"}, description = "The base file storage format")

Review comment:
       same here.. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] vinothchandar commented on pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1512:
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-620202899


   Just 1 suggestion. LGTM otherwise . thanks @lamber-ken 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] lamber-ken edited a comment on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

Posted by GitBox <gi...@apache.org>.
lamber-ken edited a comment on issue #1512:
URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-617055212


   @vinothchandar, all review comments are addressed and fixed, thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org