You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/01/14 02:38:49 UTC
[GitHub] [hudi] liujinhui1994 opened a new pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
liujinhui1994 opened a new pull request #2443:
URL: https://github.com/apache/hudi/pull/2443
Make whether the failure of connect hive affects hudi ingest process configurable
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
Currently, In an ETL pipeline(eg, kafka -> hudi -> hive), If the process of hudi to hive failed, the job is still running.
I think we can add a switch to control the job behavior(fail or keep running) when kafka to hudi is ok, while hudi to hive failed, leave the choice to user. since ingesting data to hudi and sync to hive is a complete task in some scenes.
## Brief change log
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r567795367
##########
File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
##########
@@ -57,13 +57,23 @@
public static final String SUFFIX_READ_OPTIMIZED_TABLE = "_ro";
private final HiveSyncConfig cfg;
- private final HoodieHiveClient hoodieHiveClient;
+ private HoodieHiveClient hoodieHiveClient = null;
private final String snapshotTableName;
private final Option<String> roTableTableName;
public HiveSyncTool(HiveSyncConfig cfg, HiveConf configuration, FileSystem fs) {
super(configuration.getAllProperties(), fs);
- this.hoodieHiveClient = new HoodieHiveClient(cfg, configuration, fs);
+
+ try {
+ this.hoodieHiveClient = new HoodieHiveClient(cfg, configuration, fs);
+ } catch (RuntimeException e) {
+ if (cfg.ignoreConnectException) {
+ LOG.error("Got runtime exception when hive syncing", e);
Review comment:
minor. can we also add something like "but still proceeding as configured to ignoring hive connect exceptions"
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -293,6 +293,8 @@ public static HiveSyncConfig buildHiveSyncConfig(TypedProperties props, String b
DataSourceWriteOptions.DEFAULT_HIVE_USE_JDBC_OPT_VAL()));
hiveSyncConfig.autoCreateDatabase = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_AUTO_CREATE_DATABASE_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_AUTO_CREATE_DATABASE_OPT_KEY()));
+ hiveSyncConfig.ignoreConnectException = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_IGNORE_CONNECT_EXCEPTION_OPT_KEY(),
Review comment:
@yanghua : are you suggesting, we can make this config cover exceptions in syncHoodieTable() as well ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r579644520
##########
File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
##########
@@ -58,8 +58,8 @@
private final HiveSyncConfig cfg;
private HoodieHiveClient hoodieHiveClient = null;
- private final String snapshotTableName;
- private final Option<String> roTableTableName;
+ private String snapshotTableName = null;
+ private Option<String> roTableTableName = null;
Review comment:
what about `roTableName`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-782091102
cool, sounds good π
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-780286206
@liujinhui1994 : A gentle ping on this PR. Can you please address the feedback and update the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan removed a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan removed a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-782802631
@vinothchandar @yanghua : A general doubt on supporting this config knob. As quoted above if connection fails and if we set this config of interest to true, then IMetaStoreClient will not be initialized within HoodieHiveClient. So, for all methods in this class we have to handle null.
For eg createTable, getTableSchema, etc . I am not sure how to go about this.
One approach we can take.
for some operations we can do our best to return empty or false, or something of those sort.
For eg, getPartitionEvents, doesTableExist we can return empty list / return false.
addPartitionsToTable we can just let it as no-op.
But for other operations like getTableSchema() we can throw exceptions.
What are you thoughts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760147630
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=h1) Report
> Merging [#2443](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=desc) (56f3f68) into [master](https://codecov.io/gh/apache/hudi/commit/b0010bf3b449a9d2e01955b0746b795e22e577db?el=desc) (b0010bf) will **increase** coverage by `18.36%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2443 +/- ##
=============================================
+ Coverage 51.15% 69.51% +18.36%
+ Complexity 3212 357 -2855
=============================================
Files 436 53 -383
Lines 19987 1929 -18058
Branches 2057 230 -1827
=============================================
- Hits 10224 1341 -8883
+ Misses 8922 456 -8466
+ Partials 841 132 -709
```
| Flag | Coverage Ξ | Complexity Ξ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `69.51% <ΓΈ> (+0.12%)` | `0.00 <ΓΈ> (ΓΈ)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=tree) | Coverage Ξ | Complexity Ξ | |
|---|---|---|---|
| [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `68.72% <0.00%> (-0.26%)` | `18.00% <0.00%> (ΓΈ%)` | |
| [.../main/java/org/apache/hudi/util/AvroConvertor.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL0F2cm9Db252ZXJ0b3IuamF2YQ==) | | | |
| [...pache/hudi/common/fs/FailSafeConsistencyGuard.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0ZhaWxTYWZlQ29uc2lzdGVuY3lHdWFyZC5qYXZh) | | | |
| [...va/org/apache/hudi/common/model/CleanFileInfo.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0NsZWFuRmlsZUluZm8uamF2YQ==) | | | |
| [...java/org/apache/hudi/common/util/NetworkUtils.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvTmV0d29ya1V0aWxzLmphdmE=) | | | |
| [...udi/common/util/queue/BoundedInMemoryExecutor.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvcXVldWUvQm91bmRlZEluTWVtb3J5RXhlY3V0b3IuamF2YQ==) | | | |
| [...oop/realtime/HoodieParquetRealtimeInputFormat.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL0hvb2RpZVBhcnF1ZXRSZWFsdGltZUlucHV0Rm9ybWF0LmphdmE=) | | | |
| [...oning/compaction/CompactionV2MigrationHandler.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL3ZlcnNpb25pbmcvY29tcGFjdGlvbi9Db21wYWN0aW9uVjJNaWdyYXRpb25IYW5kbGVyLmphdmE=) | | | |
| [.../java/org/apache/hudi/hadoop/InputPathHandler.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0lucHV0UGF0aEhhbmRsZXIuamF2YQ==) | | | |
| [.../apache/hudi/common/fs/ConsistencyGuardConfig.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0NvbnNpc3RlbmN5R3VhcmRDb25maWcuamF2YQ==) | | | |
| ... and [373 more](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #2443:
URL: https://github.com/apache/hudi/pull/2443
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760147630
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r567796439
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -293,6 +293,8 @@ public static HiveSyncConfig buildHiveSyncConfig(TypedProperties props, String b
DataSourceWriteOptions.DEFAULT_HIVE_USE_JDBC_OPT_VAL()));
hiveSyncConfig.autoCreateDatabase = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_AUTO_CREATE_DATABASE_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_AUTO_CREATE_DATABASE_OPT_KEY()));
+ hiveSyncConfig.ignoreConnectException = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_IGNORE_CONNECT_EXCEPTION_OPT_KEY(),
Review comment:
> @yanghua : are you suggesting, we can make this config cover exceptions in syncHoodieTable() as well ?
@nsivabalan Yes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r570230938
##########
File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
##########
@@ -57,13 +57,23 @@
public static final String SUFFIX_READ_OPTIMIZED_TABLE = "_ro";
private final HiveSyncConfig cfg;
- private final HoodieHiveClient hoodieHiveClient;
+ private HoodieHiveClient hoodieHiveClient = null;
private final String snapshotTableName;
private final Option<String> roTableTableName;
public HiveSyncTool(HiveSyncConfig cfg, HiveConf configuration, FileSystem fs) {
super(configuration.getAllProperties(), fs);
- this.hoodieHiveClient = new HoodieHiveClient(cfg, configuration, fs);
+
+ try {
+ this.hoodieHiveClient = new HoodieHiveClient(cfg, configuration, fs);
Review comment:
I guess there could be gap/bug in here. Did you happen to test it out. if hoodieHiveClient is null (due to connect execution) and if caller subsequently calls syncHoodieTable(), won't we end up with NullPointerException? Can you please check.
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -293,6 +293,8 @@ public static HiveSyncConfig buildHiveSyncConfig(TypedProperties props, String b
DataSourceWriteOptions.DEFAULT_HIVE_USE_JDBC_OPT_VAL()));
hiveSyncConfig.autoCreateDatabase = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_AUTO_CREATE_DATABASE_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_AUTO_CREATE_DATABASE_OPT_KEY()));
+ hiveSyncConfig.ignoreConnectException = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_IGNORE_CONNECT_EXCEPTION_OPT_KEY(),
Review comment:
nope. was just clarifying :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-784233741
@yanghua : I synced up w/ Vinoth on this. here is our idea. we will make the config (naming) generic for now. but lets not make it wider. bcoz, if sync explicitly fails probably user need to get notified. Since this diff was raised mainly for connect execption, we will leave it as is. But in future if some customer needs to ignore sync exceptions as well, we can consider silently ignoring those as well. wdyt?
If you agree, only fix we need for now is to fix the config naming. I can work on it and push in a small commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760147630
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=h1) Report
> Merging [#2443](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=desc) (7baf5de) into [master](https://codecov.io/gh/apache/hudi/commit/b0010bf3b449a9d2e01955b0746b795e22e577db?el=desc) (b0010bf) will **increase** coverage by `18.31%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2443 +/- ##
=============================================
+ Coverage 51.15% 69.46% +18.31%
+ Complexity 3212 356 -2856
=============================================
Files 436 53 -383
Lines 19987 1929 -18058
Branches 2057 230 -1827
=============================================
- Hits 10224 1340 -8884
+ Misses 8922 456 -8466
+ Partials 841 133 -708
```
| Flag | Coverage Ξ | Complexity Ξ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `69.46% <ΓΈ> (+0.07%)` | `0.00 <ΓΈ> (ΓΈ)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=tree) | Coverage Ξ | Complexity Ξ | |
|---|---|---|---|
| [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `68.72% <0.00%> (-0.26%)` | `18.00% <0.00%> (ΓΈ%)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.00% <0.00%> (ΓΈ)` | `50.00% <0.00%> (ΓΈ%)` | |
| [.../main/java/org/apache/hudi/cli/utils/HiveUtil.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL3V0aWxzL0hpdmVVdGlsLmphdmE=) | | | |
| [.../org/apache/hudi/common/engine/EngineProperty.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2VuZ2luZS9FbmdpbmVQcm9wZXJ0eS5qYXZh) | | | |
| [...che/hudi/common/table/log/HoodieLogFileReader.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGaWxlUmVhZGVyLmphdmE=) | | | |
| [...ava/org/apache/hudi/cli/commands/TableCommand.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1RhYmxlQ29tbWFuZC5qYXZh) | | | |
| [...on/table/view/RemoteHoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUmVtb3RlSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | | | |
| [...java/org/apache/hudi/common/fs/StorageSchemes.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL1N0b3JhZ2VTY2hlbWVzLmphdmE=) | | | |
| [...util/jvm/OpenJ9MemoryLayoutSpecification64bit.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvanZtL09wZW5KOU1lbW9yeUxheW91dFNwZWNpZmljYXRpb242NGJpdC5qYXZh) | | | |
| [...common/table/view/PriorityBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUHJpb3JpdHlCYXNlZEZpbGVTeXN0ZW1WaWV3LmphdmE=) | | | |
| ... and [373 more](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760147630
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-785506762
> @yanghua : I synced up w/ Vinoth on this. here is our idea. we will make the config (naming) generic for now. but lets not make it wider to ignore exceptions for sync. bcoz, if sync explicitly fails probably user need to get notified. Since this diff was raised mainly for connect execption, we will leave it as is. But in future if some customer needs to ignore sync exceptions as well, we can consider silently ignoring those as well(so that we don't introduce a new config and can use the same config). wdyt?
> If you agree, only fix we need for now is to fix the config naming. I can work on it and push in a small commit.
OK, agree.
The original intention of my idea is actually to not want the failure of synchronization to affect hudi's tasks. We have experienced situations where Hive is unavailable and all jobs fail. Then we completely ignored the failure, but when it failed, we implemented log warnings.
But as a realization of the community, I agree with you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760147630
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-785655618
got it, thanks. I have addressed all pending feedbacks. Will land once CI turns green.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r568523091
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -293,6 +293,8 @@ public static HiveSyncConfig buildHiveSyncConfig(TypedProperties props, String b
DataSourceWriteOptions.DEFAULT_HIVE_USE_JDBC_OPT_VAL()));
hiveSyncConfig.autoCreateDatabase = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_AUTO_CREATE_DATABASE_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_AUTO_CREATE_DATABASE_OPT_KEY()));
+ hiveSyncConfig.ignoreConnectException = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_IGNORE_CONNECT_EXCEPTION_OPT_KEY(),
Review comment:
> @yanghua : are you suggesting, we can make this config cover exceptions in syncHoodieTable() as well ?
Yes, do you have any concerns? In our scenario, we met all hudi spark jobs failed when the hive server is not available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r570230938
##########
File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
##########
@@ -57,13 +57,23 @@
public static final String SUFFIX_READ_OPTIMIZED_TABLE = "_ro";
private final HiveSyncConfig cfg;
- private final HoodieHiveClient hoodieHiveClient;
+ private HoodieHiveClient hoodieHiveClient = null;
private final String snapshotTableName;
private final Option<String> roTableTableName;
public HiveSyncTool(HiveSyncConfig cfg, HiveConf configuration, FileSystem fs) {
super(configuration.getAllProperties(), fs);
- this.hoodieHiveClient = new HoodieHiveClient(cfg, configuration, fs);
+
+ try {
+ this.hoodieHiveClient = new HoodieHiveClient(cfg, configuration, fs);
Review comment:
I guess there could be gap/bug in here. Did you happen to test it out. if hoodieHiveClient is null (due to connect execution) and if caller subsequently calls syncHoodieTable(), won't we end up with NullPointerException? Can you please check.
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -293,6 +293,8 @@ public static HiveSyncConfig buildHiveSyncConfig(TypedProperties props, String b
DataSourceWriteOptions.DEFAULT_HIVE_USE_JDBC_OPT_VAL()));
hiveSyncConfig.autoCreateDatabase = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_AUTO_CREATE_DATABASE_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_AUTO_CREATE_DATABASE_OPT_KEY()));
+ hiveSyncConfig.ignoreConnectException = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_IGNORE_CONNECT_EXCEPTION_OPT_KEY(),
Review comment:
nope. was just clarifying :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r579751395
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -293,6 +293,8 @@ public static HiveSyncConfig buildHiveSyncConfig(TypedProperties props, String b
DataSourceWriteOptions.DEFAULT_HIVE_USE_JDBC_OPT_VAL()));
hiveSyncConfig.autoCreateDatabase = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_AUTO_CREATE_DATABASE_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_AUTO_CREATE_DATABASE_OPT_KEY()));
+ hiveSyncConfig.ignoreConnectException = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_IGNORE_CONNECT_EXCEPTION_OPT_KEY(),
Review comment:
@liujinhui1994 : do you have any concerns to make this cover all exception w/ hive sync? I don't see you have addressed this feedback.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r567184581
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -293,6 +293,8 @@ public static HiveSyncConfig buildHiveSyncConfig(TypedProperties props, String b
DataSourceWriteOptions.DEFAULT_HIVE_USE_JDBC_OPT_VAL()));
hiveSyncConfig.autoCreateDatabase = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_AUTO_CREATE_DATABASE_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_AUTO_CREATE_DATABASE_OPT_KEY()));
+ hiveSyncConfig.ignoreConnectException = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_IGNORE_CONNECT_EXCEPTION_OPT_KEY(),
Review comment:
Can we extend a larger range from `connect exception` to `sync exception`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r568523091
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -293,6 +293,8 @@ public static HiveSyncConfig buildHiveSyncConfig(TypedProperties props, String b
DataSourceWriteOptions.DEFAULT_HIVE_USE_JDBC_OPT_VAL()));
hiveSyncConfig.autoCreateDatabase = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_AUTO_CREATE_DATABASE_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_AUTO_CREATE_DATABASE_OPT_KEY()));
+ hiveSyncConfig.ignoreConnectException = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_IGNORE_CONNECT_EXCEPTION_OPT_KEY(),
Review comment:
> @yanghua : are you suggesting, we can make this config cover exceptions in syncHoodieTable() as well ?
Yes, do you have any concerns? In our scenario, we met all hudi spark jobs failed when the hive server is not available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-782985138
> Just to confirm, we don't need to fix anything wrt HoodieHiveClient wrt this patch is it?
yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760147630
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760147630
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=h1) Report
> Merging [#2443](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=desc) (0d98db7) into [master](https://codecov.io/gh/apache/hudi/commit/e3d3677b7e7899705b624925666317f0c074f7c7?el=desc) (e3d3677) will **decrease** coverage by `41.04%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2443 +/- ##
============================================
- Coverage 50.73% 9.68% -41.05%
+ Complexity 3064 48 -3016
============================================
Files 419 53 -366
Lines 18797 1930 -16867
Branches 1922 230 -1692
============================================
- Hits 9536 187 -9349
+ Misses 8487 1730 -6757
+ Partials 774 13 -761
```
| Flag | Coverage Ξ | Complexity Ξ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `100.00% <ΓΈ> (ΓΈ)` | `0.00 <ΓΈ> (ΓΈ)` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.68% <ΓΈ> (-59.80%)` | `0.00 <ΓΈ> (ΓΈ)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=tree) | Coverage Ξ | Complexity Ξ | |
|---|---|---|---|
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| ... and [393 more](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-784233741
@yanghua : I synced up w/ Vinoth on this. here is our idea. we will make the config (naming) generic for now. but lets not make it wider to ignore exceptions for sync. bcoz, if sync explicitly fails probably user need to get notified. Since this diff was raised mainly for connect execption, we will leave it as is. But in future if some customer needs to ignore sync exceptions as well, we can consider silently ignoring those as well(so that we don't introduce a new config and can use the same config). wdyt?
If you agree, only fix we need for now is to fix the config naming. I can work on it and push in a small commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r567796439
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -293,6 +293,8 @@ public static HiveSyncConfig buildHiveSyncConfig(TypedProperties props, String b
DataSourceWriteOptions.DEFAULT_HIVE_USE_JDBC_OPT_VAL()));
hiveSyncConfig.autoCreateDatabase = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_AUTO_CREATE_DATABASE_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_AUTO_CREATE_DATABASE_OPT_KEY()));
+ hiveSyncConfig.ignoreConnectException = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_IGNORE_CONNECT_EXCEPTION_OPT_KEY(),
Review comment:
@yanghua : are you suggesting, we can make this config cover exceptions in syncHoodieTable() as well ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-782672436
@nsivabalan @yanghua please help review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-780366281
Thank you for your attention. I want to make changes after verification in the production environment. I will update in the near future
------------------ Original ------------------
From: Sivabalan Narayanan <notifications@github.com>
Date: Wed,Feb 17,2021 0:18 PM
To: apache/hudi <hudi@noreply.github.com>
Cc: liujinhui <965147871@qq.com>, Mention <mention@noreply.github.com>
Subject: Re: [apache/hudi] [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable (#2443)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-784233741
@yanghua : I synced up w/ Vinoth on this. here is our idea. we will make the config (naming) generic for now. but lets not make it wider to ignore exceptions for sync. bcoz, if sync explicitly fails probably user need to get notified. Since this diff was raised mainly for connect execption, we will leave it as is. But in future if some customer needs to ignore sync exceptions as well, we can consider silently ignoring those as well. wdyt?
If you agree, only fix we need for now is to fix the config naming. I can work on it and push in a small commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-782802631
@vinothchandar @yanghua : A general doubt on supporting this config knob. As quoted above if connection fails and if we set this config of interest to true, then IMetaStoreClient will not be initialized within HoodieHiveClient. So, for all methods in this class we have to handle null.
For eg createTable, getTableSchema, etc . I am not sure how to go about this.
One approach we can take.
for some operations we can do our best to return empty or false, or something of those sort.
For eg, getPartitionEvents, doesTableExist we can return empty list / return false.
addPartitionsToTable we can just let it as no-op.
But for other operations like getTableSchema() we can throw exceptions.
What are you thoughts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-783558211
I have pushed a commit to add tests and address the roTableRename. But not sure if we can make the Config span any exception in general or just connect exception.
@vinothchandar @n3nash : Vino yang suggested we can ignore any exception w/ hive sync in general and proceed if the newly introduced config is set. As of now(in this patch), we only consider connect exceptions. Any thoughts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760147630
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=h1) Report
> Merging [#2443](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=desc) (86faca0) into [master](https://codecov.io/gh/apache/hudi/commit/c4afd179c1983a382b8a5197d800b0f5dba254de?el=desc) (c4afd17) will **increase** coverage by `19.33%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2443 +/- ##
=============================================
+ Coverage 50.18% 69.51% +19.33%
+ Complexity 3050 357 -2693
=============================================
Files 419 53 -366
Lines 18931 1929 -17002
Branches 1948 230 -1718
=============================================
- Hits 9500 1341 -8159
+ Misses 8656 456 -8200
+ Partials 775 132 -643
```
| Flag | Coverage Ξ | Complexity Ξ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `69.51% <ΓΈ> (+0.08%)` | `0.00 <ΓΈ> (ΓΈ)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=tree) | Coverage Ξ | Complexity Ξ | |
|---|---|---|---|
| [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `68.72% <0.00%> (-0.26%)` | `18.00% <0.00%> (ΓΈ%)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.35% <0.00%> (-0.15%)` | `51.00% <0.00%> (+1.00%)` | :arrow_down: |
| [...org/apache/hudi/common/table/log/AppendResult.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9BcHBlbmRSZXN1bHQuamF2YQ==) | | | |
| [...i/hive/SlashEncodedDayPartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvU2xhc2hFbmNvZGVkRGF5UGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | | | |
| [.../apache/hudi/common/bootstrap/FileStatusUtils.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2Jvb3RzdHJhcC9GaWxlU3RhdHVzVXRpbHMuamF2YQ==) | | | |
| [...apache/hudi/common/model/HoodieCleaningPolicy.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUNsZWFuaW5nUG9saWN5LmphdmE=) | | | |
| [...org/apache/hudi/common/bloom/BloomFilterUtils.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2Jsb29tL0Jsb29tRmlsdGVyVXRpbHMuamF2YQ==) | | | |
| [...common/table/view/PriorityBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUHJpb3JpdHlCYXNlZEZpbGVTeXN0ZW1WaWV3LmphdmE=) | | | |
| [...java/org/apache/hudi/common/util/ParquetUtils.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvUGFycXVldFV0aWxzLmphdmE=) | | | |
| [...he/hudi/common/table/timeline/dto/BaseFileDTO.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL2R0by9CYXNlRmlsZURUTy5qYXZh) | | | |
| ... and [356 more](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r579750318
##########
File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
##########
@@ -58,8 +58,8 @@
private final HiveSyncConfig cfg;
private HoodieHiveClient hoodieHiveClient = null;
- private final String snapshotTableName;
- private final Option<String> roTableTableName;
+ private String snapshotTableName = null;
+ private Option<String> roTableTableName = null;
Review comment:
guess it's a typo. Probably we can fix it while you are at it.
##########
File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
##########
@@ -57,13 +57,23 @@
public static final String SUFFIX_READ_OPTIMIZED_TABLE = "_ro";
private final HiveSyncConfig cfg;
- private final HoodieHiveClient hoodieHiveClient;
+ private HoodieHiveClient hoodieHiveClient = null;
private final String snapshotTableName;
private final Option<String> roTableTableName;
public HiveSyncTool(HiveSyncConfig cfg, HiveConf configuration, FileSystem fs) {
super(configuration.getAllProperties(), fs);
- this.hoodieHiveClient = new HoodieHiveClient(cfg, configuration, fs);
+
+ try {
+ this.hoodieHiveClient = new HoodieHiveClient(cfg, configuration, fs);
+ } catch (RuntimeException e) {
+ if (cfg.ignoreConnectException) {
+ LOG.error("Got runtime exception when hive syncing", e);
Review comment:
Did you address this feedback ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-782990952
@nsivabalan Thanks for the suggestion, I will make changes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760270387
@liujinhui1994 Travis is red. @wangxianghu help to review firstly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760147630
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=h1) Report
> Merging [#2443](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=desc) (7baf5de) into [master](https://codecov.io/gh/apache/hudi/commit/b0010bf3b449a9d2e01955b0746b795e22e577db?el=desc) (b0010bf) will **increase** coverage by `0.11%`.
> The diff coverage is `76.66%`.
[](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2443 +/- ##
============================================
+ Coverage 51.15% 51.26% +0.11%
- Complexity 3212 3234 +22
============================================
Files 436 438 +2
Lines 19987 20107 +120
Branches 2057 2073 +16
============================================
+ Hits 10224 10308 +84
- Misses 8922 8947 +25
- Partials 841 852 +11
```
| Flag | Coverage Ξ | Complexity Ξ | |
|---|---|---|---|
| hudicli | `36.87% <ΓΈ> (-0.04%)` | `0.00 <ΓΈ> (ΓΈ)` | |
| hudiclient | `100.00% <ΓΈ> (ΓΈ)` | `0.00 <ΓΈ> (ΓΈ)` | |
| hudicommon | `51.35% <ΓΈ> (-0.04%)` | `0.00 <ΓΈ> (ΓΈ)` | |
| hudiflink | `46.85% <ΓΈ> (+1.41%)` | `0.00 <ΓΈ> (ΓΈ)` | |
| hudihadoopmr | `33.16% <ΓΈ> (ΓΈ)` | `0.00 <ΓΈ> (ΓΈ)` | |
| hudisparkdatasource | `69.71% <0.00%> (-0.03%)` | `0.00 <0.00> (ΓΈ)` | |
| hudisync | `49.62% <79.31%> (+1.00%)` | `0.00 <8.00> (ΓΈ)` | |
| huditimelineservice | `66.49% <ΓΈ> (ΓΈ)` | `0.00 <ΓΈ> (ΓΈ)` | |
| hudiutilities | `69.46% <ΓΈ> (+0.07%)` | `0.00 <ΓΈ> (ΓΈ)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=tree) | Coverage Ξ | Complexity Ξ | |
|---|---|---|---|
| [...n/scala/org/apache/hudi/HoodieSparkSqlWriter.scala](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVNwYXJrU3FsV3JpdGVyLnNjYWxh) | `49.82% <0.00%> (+<0.01%)` | `0.00 <0.00> (ΓΈ)` | |
| [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `70.37% <77.77%> (+1.29%)` | `17.00 <8.00> (+3.00)` | |
| [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `97.72% <100.00%> (+0.10%)` | `2.00 <0.00> (ΓΈ)` | |
| [...java/org/apache/hudi/common/util/CleanerUtils.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQ2xlYW5lclV0aWxzLmphdmE=) | `47.72% <0.00%> (-15.91%)` | `6.00% <0.00%> (ΓΈ%)` | |
| [...udi/operator/partitioner/BucketAssignFunction.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9vcGVyYXRvci9wYXJ0aXRpb25lci9CdWNrZXRBc3NpZ25GdW5jdGlvbi5qYXZh) | `84.78% <0.00%> (-7.53%)` | `21.00% <0.00%> (+13.00%)` | :arrow_down: |
| [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRXcml0ZXIuamF2YQ==) | `78.12% <0.00%> (-1.57%)` | `26.00% <0.00%> (ΓΈ%)` | |
| [...c/main/java/org/apache/hudi/common/fs/FSUtils.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0ZTVXRpbHMuamF2YQ==) | `49.31% <0.00%> (-0.46%)` | `61.00% <0.00%> (ΓΈ%)` | |
| [...rg/apache/hudi/cli/commands/SavepointsCommand.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1NhdmVwb2ludHNDb21tYW5kLmphdmE=) | `13.84% <0.00%> (-0.44%)` | `3.00% <0.00%> (ΓΈ%)` | |
| [...ain/java/org/apache/hudi/avro/HoodieAvroUtils.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXZyby9Ib29kaWVBdnJvVXRpbHMuamF2YQ==) | `55.66% <0.00%> (-0.44%)` | `38.00% <0.00%> (ΓΈ%)` | |
| [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `68.72% <0.00%> (-0.26%)` | `18.00% <0.00%> (ΓΈ%)` | |
| ... and [16 more](https://codecov.io/gh/apache/hudi/pull/2443/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on a change in pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#discussion_r579660766
##########
File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
##########
@@ -58,8 +58,8 @@
private final HiveSyncConfig cfg;
private HoodieHiveClient hoodieHiveClient = null;
- private final String snapshotTableName;
- private final Option<String> roTableTableName;
+ private String snapshotTableName = null;
+ private Option<String> roTableTableName = null;
Review comment:
I think itβs better to keep the original name
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2443: [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760147630
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org