You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/03/28 06:36:30 UTC
[GitHub] [incubator-hudi] leesf opened a new pull request #1460: [HUDI-679]
Make io package Spark free
leesf opened a new pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
Make io package spark free.
## Brief change log
* Introduce SparkSupplier.java
* Remove the usage of TaskContext in io package.
## Verify this pull request
This pull request is a trivial rework / code cleanup without any test coverage.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399700184
##########
File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java
##########
@@ -55,26 +55,27 @@
protected final String partitionPath;
protected final String fileId;
protected final String writeToken;
+ protected final Suppliers suppliers;
public HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath,
- String fileId, HoodieTable<T> hoodieTable) {
+ String fileId, HoodieTable<T> hoodieTable, Suppliers suppliers) {
super(config, instantTime, hoodieTable);
this.partitionPath = partitionPath;
this.fileId = fileId;
- this.writeToken = makeSparkWriteToken();
this.originalSchema = new Schema.Parser().parse(config.getSchema());
this.writerSchema = createHoodieWriteSchema(originalSchema);
this.timer = new HoodieTimer().startTimer();
this.writeStatus = (WriteStatus) ReflectionUtils.loadClass(config.getWriteStatusClassName(),
!hoodieTable.getIndex().isImplicitWithStorage(), config.getWriteStatusFailureFraction());
+ this.suppliers = suppliers;
+ this.writeToken = makeSparkWriteToken();
Review comment:
rename to just `makeWriteToken()`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua merged pull request #1460: [HUDI-679] Make
io package Spark free
Posted by GitBox <gi...@apache.org>.
yanghua merged pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1460:
[HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605428358
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=h1) Report
> Merging [#1460](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/1713f686f86e8c2f0a908c313cca9b595c6aed33&el=desc) will **decrease** coverage by `0.08%`.
> The diff coverage is `97.36%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1460 +/- ##
============================================
- Coverage 67.66% 67.58% -0.09%
Complexity 261 261
============================================
Files 342 348 +6
Lines 16510 16670 +160
Branches 1684 1693 +9
============================================
+ Hits 11172 11266 +94
- Misses 4599 4665 +66
Partials 739 739
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hudi/execution/MergeOnReadLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL01lcmdlT25SZWFkTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `64.70% <66.66%> (ø)` | `0.00 <0.00> (ø)` | |
| [...g/apache/hudi/client/SparkTaskContextSupplier.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrVGFza0NvbnRleHRTdXBwbGllci5qYXZh) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [.../hudi/execution/CopyOnWriteLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0NvcHlPbldyaXRlTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `80.76% <100.00%> (+0.37%)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQXBwZW5kSGFuZGxlLmphdmE=) | `84.17% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQ3JlYXRlSGFuZGxlLmphdmE=) | `84.61% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllTWVyZ2VIYW5kbGUuamF2YQ==) | `79.31% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllV3JpdGVIYW5kbGUuamF2YQ==) | `75.00% <100.00%> (+1.66%)` | `0.00 <0.00> (ø)` | |
| [...rg/apache/hudi/io/storage/HoodieParquetWriter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVQYXJxdWV0V3JpdGVyLmphdmE=) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...he/hudi/io/storage/HoodieStorageWriterFactory.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTdG9yYWdlV3JpdGVyRmFjdG9yeS5qYXZh) | `93.75% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| ... and [22 more](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=footer). Last update [1713f68...07f073a](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399737990
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/SparkTaskContextDetailSupplier.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.spark.TaskContext;
+
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * Spark Supplier.
+ */
+public interface SparkTaskContextDetailSupplier<T> extends Supplier<T>, Serializable {
Review comment:
Just online seeing the latest changes comes from @leesf . Yes, it seems this is a better abstraction.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1460:
[HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605428358
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=h1) Report
> Merging [#1460](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/1713f686f86e8c2f0a908c313cca9b595c6aed33&el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `97.36%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1460 +/- ##
============================================
- Coverage 67.66% 67.60% -0.07%
Complexity 261 261
============================================
Files 342 348 +6
Lines 16510 16670 +160
Branches 1684 1693 +9
============================================
+ Hits 11172 11270 +98
- Misses 4599 4661 +62
Partials 739 739
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hudi/execution/MergeOnReadLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL01lcmdlT25SZWFkTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `64.70% <66.66%> (ø)` | `0.00 <0.00> (ø)` | |
| [...g/apache/hudi/client/SparkTaskContextSupplier.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrVGFza0NvbnRleHRTdXBwbGllci5qYXZh) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [.../hudi/execution/CopyOnWriteLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0NvcHlPbldyaXRlTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `80.76% <100.00%> (+0.37%)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQXBwZW5kSGFuZGxlLmphdmE=) | `84.17% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQ3JlYXRlSGFuZGxlLmphdmE=) | `84.61% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllTWVyZ2VIYW5kbGUuamF2YQ==) | `79.31% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllV3JpdGVIYW5kbGUuamF2YQ==) | `75.00% <100.00%> (+1.66%)` | `0.00 <0.00> (ø)` | |
| [...rg/apache/hudi/io/storage/HoodieParquetWriter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVQYXJxdWV0V3JpdGVyLmphdmE=) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...he/hudi/io/storage/HoodieStorageWriterFactory.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTdG9yYWdlV3JpdGVyRmFjdG9yeS5qYXZh) | `93.75% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| ... and [22 more](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=footer). Last update [1713f68...07f073a](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1460:
[HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605428358
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=h1) Report
> Merging [#1460](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/1713f686f86e8c2f0a908c313cca9b595c6aed33&el=desc) will **decrease** coverage by `0.08%`.
> The diff coverage is `97.36%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1460 +/- ##
============================================
- Coverage 67.66% 67.58% -0.09%
Complexity 261 261
============================================
Files 342 348 +6
Lines 16510 16670 +160
Branches 1684 1693 +9
============================================
+ Hits 11172 11266 +94
- Misses 4599 4665 +66
Partials 739 739
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hudi/execution/MergeOnReadLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL01lcmdlT25SZWFkTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `64.70% <66.66%> (ø)` | `0.00 <0.00> (ø)` | |
| [...g/apache/hudi/client/SparkTaskContextSupplier.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrVGFza0NvbnRleHRTdXBwbGllci5qYXZh) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [.../hudi/execution/CopyOnWriteLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0NvcHlPbldyaXRlTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `80.76% <100.00%> (+0.37%)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQXBwZW5kSGFuZGxlLmphdmE=) | `84.17% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQ3JlYXRlSGFuZGxlLmphdmE=) | `84.61% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllTWVyZ2VIYW5kbGUuamF2YQ==) | `79.31% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllV3JpdGVIYW5kbGUuamF2YQ==) | `75.00% <100.00%> (+1.66%)` | `0.00 <0.00> (ø)` | |
| [...rg/apache/hudi/io/storage/HoodieParquetWriter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVQYXJxdWV0V3JpdGVyLmphdmE=) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...he/hudi/io/storage/HoodieStorageWriterFactory.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTdG9yYWdlV3JpdGVyRmFjdG9yeS5qYXZh) | `93.75% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| ... and [22 more](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=footer). Last update [1713f68...07f073a](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399660595
##########
File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java
##########
@@ -55,26 +55,32 @@
protected final String partitionPath;
protected final String fileId;
protected final String writeToken;
+ protected final Supplier<Integer> idSupplier;
Review comment:
ditto, IMO, id here is not clear. partition id is better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399660167
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/SparkTaskContextDetailSupplier.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.spark.TaskContext;
+
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * Spark Supplier.
+ */
+public interface SparkTaskContextDetailSupplier<T> extends Supplier<T>, Serializable {
+
+ /**
+ * Supplier to get partition id.
+ */
+ SparkTaskContextDetailSupplier<Integer> PARTITION_SUPPLIER = () -> TaskContext.getPartitionId();
+
+ /**
+ * Supplier to get stage id.
+ */
+ SparkTaskContextDetailSupplier<Integer> STAGE_SUPPLIER = () -> TaskContext.get().stageId();
+
+ /**
+ * Supplier to get task attempt id.
+ */
+ SparkTaskContextDetailSupplier<Long> ATTEMPT_SUPPLIER = () -> TaskContext.get().taskAttemptId();
Review comment:
`ATTEMPT_ID_SUPPLIER `?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399649196
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/SparkSupplier.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.spark.TaskContext;
+
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * Spark Supplier.
+ */
+public interface SparkSupplier<T> extends Supplier<T>, Serializable {
Review comment:
Considering this interface supports some information about `TaskContext`, can we rename to `SparkTaskContextDetailSupplier`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399660338
##########
File path: hudi-client/src/main/java/org/apache/hudi/execution/BulkInsertMapFunction.java
##########
@@ -51,6 +51,7 @@ public BulkInsertMapFunction(String instantTime, HoodieWriteConfig config, Hoodi
@Override
public Iterator<List<WriteStatus>> call(Integer partition, Iterator<HoodieRecord<T>> sortedRecordItr) {
return new CopyOnWriteLazyInsertIterable<>(sortedRecordItr, config, instantTime, hoodieTable,
- fileIDPrefixes.get(partition));
+ fileIDPrefixes.get(partition), hoodieTable.getIdSupplier(), hoodieTable.getStageSupplier(),
Review comment:
`hoodieTable.getIdSupplier()` is not clear here. I suggest we can rename these getter to e.g. `getPartitionIdSupplier`, `getStageId` and `getAttemptId`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on a change in pull request #1460:
[HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399673870
##########
File path: hudi-client/src/main/java/org/apache/hudi/execution/CopyOnWriteLazyInsertIterable.java
##########
@@ -50,15 +51,23 @@
protected final HoodieTable<T> hoodieTable;
protected final String idPrefix;
protected int numFilesWritten;
+ protected Supplier<Integer> idSupplier;
+ protected Supplier<Integer> stageSupplier;
+ protected Supplier<Long> attemptSupplier;
public CopyOnWriteLazyInsertIterable(Iterator<HoodieRecord<T>> sortedRecordItr, HoodieWriteConfig config,
- String instantTime, HoodieTable<T> hoodieTable, String idPrefix) {
+ String instantTime, HoodieTable<T> hoodieTable, String idPrefix,
+ Supplier<Integer> idSupplier, Supplier<Integer> stageSupplier,
+ Supplier<Long> attemptSupplier) {
Review comment:
> Actually, I am not sure if we can package these three args into a DTO structure. Just a thought, you can ignore.
Yes, I think it is better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399660466
##########
File path: hudi-client/src/main/java/org/apache/hudi/execution/CopyOnWriteLazyInsertIterable.java
##########
@@ -50,15 +51,23 @@
protected final HoodieTable<T> hoodieTable;
protected final String idPrefix;
protected int numFilesWritten;
+ protected Supplier<Integer> idSupplier;
+ protected Supplier<Integer> stageSupplier;
+ protected Supplier<Long> attemptSupplier;
public CopyOnWriteLazyInsertIterable(Iterator<HoodieRecord<T>> sortedRecordItr, HoodieWriteConfig config,
- String instantTime, HoodieTable<T> hoodieTable, String idPrefix) {
+ String instantTime, HoodieTable<T> hoodieTable, String idPrefix,
+ Supplier<Integer> idSupplier, Supplier<Integer> stageSupplier,
+ Supplier<Long> attemptSupplier) {
Review comment:
Actually, I am not sure if we can package these three args into a DTO structure. Just a thought, you can ignore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1460: [HUDI-679] Make io
package Spark free
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605462038
@yanghua Updated this PR to package three suppliers into `Suppliers`. PTAL, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io commented on issue #1460: [HUDI-679]
Make io package Spark free
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605428358
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=h1) Report
> Merging [#1460](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/1713f686f86e8c2f0a908c313cca9b595c6aed33&el=desc) will **decrease** coverage by `0.16%`.
> The diff coverage is `97.67%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1460 +/- ##
============================================
- Coverage 67.66% 67.50% -0.17%
Complexity 261 261
============================================
Files 342 344 +2
Lines 16510 16589 +79
Branches 1684 1694 +10
============================================
+ Hits 11172 11198 +26
- Misses 4599 4652 +53
Partials 739 739
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hudi/execution/MergeOnReadLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL01lcmdlT25SZWFkTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `64.70% <66.66%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/client/SparkSupplier.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrU3VwcGxpZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [.../hudi/execution/CopyOnWriteLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0NvcHlPbldyaXRlTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `81.48% <100.00%> (+1.08%)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQXBwZW5kSGFuZGxlLmphdmE=) | `84.17% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQ3JlYXRlSGFuZGxlLmphdmE=) | `84.61% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllTWVyZ2VIYW5kbGUuamF2YQ==) | `79.31% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllV3JpdGVIYW5kbGUuamF2YQ==) | `74.46% <100.00%> (+1.13%)` | `0.00 <0.00> (ø)` | |
| [...rg/apache/hudi/io/storage/HoodieParquetWriter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVQYXJxdWV0V3JpdGVyLmphdmE=) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...he/hudi/io/storage/HoodieStorageWriterFactory.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTdG9yYWdlV3JpdGVyRmFjdG9yeS5qYXZh) | `93.75% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| ... and [19 more](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=footer). Last update [1713f68...bb7268f](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399660372
##########
File path: hudi-client/src/main/java/org/apache/hudi/execution/CopyOnWriteLazyInsertIterable.java
##########
@@ -50,15 +51,23 @@
protected final HoodieTable<T> hoodieTable;
protected final String idPrefix;
protected int numFilesWritten;
+ protected Supplier<Integer> idSupplier;
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1460: [HUDI-679] Make io
package Spark free
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605404425
@vinothchandar @yanghua Please take a look when you are free. Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on a change in pull request #1460:
[HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399651507
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/SparkSupplier.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.spark.TaskContext;
+
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * Spark Supplier.
+ */
+public interface SparkSupplier<T> extends Supplier<T>, Serializable {
+ SparkSupplier<Integer> PARTITION_SUPPLIER = () -> TaskContext.getPartitionId();
Review comment:
> Can we add empty line to split this definition? Additionally, add some comments?
sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1460: [HUDI-679] Make io
package Spark free
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605560443
@yanghua @vinothchandar Thanks for your review, just updated the PR to address your comments. PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1460:
[HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605428358
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=h1) Report
> Merging [#1460](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/1713f686f86e8c2f0a908c313cca9b595c6aed33&el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `97.36%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1460 +/- ##
============================================
- Coverage 67.66% 67.60% -0.07%
Complexity 261 261
============================================
Files 342 348 +6
Lines 16510 16670 +160
Branches 1684 1693 +9
============================================
+ Hits 11172 11270 +98
- Misses 4599 4661 +62
Partials 739 739
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hudi/execution/MergeOnReadLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL01lcmdlT25SZWFkTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `64.70% <66.66%> (ø)` | `0.00 <0.00> (ø)` | |
| [...g/apache/hudi/client/SparkTaskContextSupplier.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrVGFza0NvbnRleHRTdXBwbGllci5qYXZh) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [.../hudi/execution/CopyOnWriteLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0NvcHlPbldyaXRlTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `80.76% <100.00%> (+0.37%)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQXBwZW5kSGFuZGxlLmphdmE=) | `84.17% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQ3JlYXRlSGFuZGxlLmphdmE=) | `84.61% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllTWVyZ2VIYW5kbGUuamF2YQ==) | `79.31% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllV3JpdGVIYW5kbGUuamF2YQ==) | `75.00% <100.00%> (+1.66%)` | `0.00 <0.00> (ø)` | |
| [...rg/apache/hudi/io/storage/HoodieParquetWriter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVQYXJxdWV0V3JpdGVyLmphdmE=) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...he/hudi/io/storage/HoodieStorageWriterFactory.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTdG9yYWdlV3JpdGVyRmFjdG9yeS5qYXZh) | `93.75% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| ... and [22 more](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=footer). Last update [1713f68...07f073a](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399700058
##########
File path: hudi-client/src/main/java/org/apache/hudi/execution/MergeOnReadLazyInsertIterable.java
##########
@@ -35,8 +36,9 @@
public class MergeOnReadLazyInsertIterable<T extends HoodieRecordPayload> extends CopyOnWriteLazyInsertIterable<T> {
public MergeOnReadLazyInsertIterable(Iterator<HoodieRecord<T>> sortedRecordItr, HoodieWriteConfig config,
- String instantTime, HoodieTable<T> hoodieTable, String idPfx) {
- super(sortedRecordItr, config, instantTime, hoodieTable, idPfx);
+ String instantTime, HoodieTable<T> hoodieTable, String idPfx,
Review comment:
side point: we should fix method arg formatting consistently between intellij and checkstyle. Keep seeing these sort of whitespace changes in PRs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399649258
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/SparkSupplier.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.spark.TaskContext;
+
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * Spark Supplier.
+ */
+public interface SparkSupplier<T> extends Supplier<T>, Serializable {
+ SparkSupplier<Integer> PARTITION_SUPPLIER = () -> TaskContext.getPartitionId();
Review comment:
Can we add empty line to split this definition? Additionally, add some comments?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1460:
[HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605428358
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=h1) Report
> Merging [#1460](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/1713f686f86e8c2f0a908c313cca9b595c6aed33&el=desc) will **decrease** coverage by `0.16%`.
> The diff coverage is `97.67%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1460 +/- ##
============================================
- Coverage 67.66% 67.50% -0.17%
Complexity 261 261
============================================
Files 342 344 +2
Lines 16510 16589 +79
Branches 1684 1694 +10
============================================
+ Hits 11172 11198 +26
- Misses 4599 4652 +53
Partials 739 739
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hudi/execution/MergeOnReadLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL01lcmdlT25SZWFkTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `64.70% <66.66%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/client/SparkSupplier.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrU3VwcGxpZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [.../hudi/execution/CopyOnWriteLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0NvcHlPbldyaXRlTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=) | `81.48% <100.00%> (+1.08%)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQXBwZW5kSGFuZGxlLmphdmE=) | `84.17% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQ3JlYXRlSGFuZGxlLmphdmE=) | `84.61% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllTWVyZ2VIYW5kbGUuamF2YQ==) | `79.31% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllV3JpdGVIYW5kbGUuamF2YQ==) | `74.46% <100.00%> (+1.13%)` | `0.00 <0.00> (ø)` | |
| [...rg/apache/hudi/io/storage/HoodieParquetWriter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVQYXJxdWV0V3JpdGVyLmphdmE=) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...he/hudi/io/storage/HoodieStorageWriterFactory.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTdG9yYWdlV3JpdGVyRmFjdG9yeS5qYXZh) | `93.75% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
| ... and [19 more](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=footer). Last update [1713f68...bb7268f](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on a change in pull request #1460:
[HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399651495
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/SparkSupplier.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.spark.TaskContext;
+
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * Spark Supplier.
+ */
+public interface SparkSupplier<T> extends Supplier<T>, Serializable {
Review comment:
sounds reasonable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1460: [HUDI-679] Make io
package Spark free
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605434298
@yanghua Updated this PR to address your comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399699941
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/Suppliers.java
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import java.io.Serializable;
+
+/**
+ * A bundle of Suppliers.
+ */
+public class Suppliers implements Serializable {
Review comment:
then this can go away.. I feel this is additional abstraction, that we may not need..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399699794
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/SparkTaskContextDetailSupplier.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.spark.TaskContext;
+
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * Spark Supplier.
+ */
+public interface SparkTaskContextDetailSupplier<T> extends Supplier<T>, Serializable {
Review comment:
I am not sure this abstraction is at the right level.. Should this have to be `Supplier`.. I think we can just have three methods that return `Supplier<Integer>` and `Supplier<Long>` and pass just one argument through the code path i.e the `SparkTaskContextSuppler` instance..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399660132
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/SparkTaskContextDetailSupplier.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.spark.TaskContext;
+
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * Spark Supplier.
+ */
+public interface SparkTaskContextDetailSupplier<T> extends Supplier<T>, Serializable {
+
+ /**
+ * Supplier to get partition id.
+ */
+ SparkTaskContextDetailSupplier<Integer> PARTITION_SUPPLIER = () -> TaskContext.getPartitionId();
+
+ /**
+ * Supplier to get stage id.
+ */
+ SparkTaskContextDetailSupplier<Integer> STAGE_SUPPLIER = () -> TaskContext.get().stageId();
Review comment:
`STAGE_ID_SUPPLIER `?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399660094
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/SparkTaskContextDetailSupplier.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.spark.TaskContext;
+
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * Spark Supplier.
+ */
+public interface SparkTaskContextDetailSupplier<T> extends Supplier<T>, Serializable {
+
+ /**
+ * Supplier to get partition id.
+ */
+ SparkTaskContextDetailSupplier<Integer> PARTITION_SUPPLIER = () -> TaskContext.getPartitionId();
Review comment:
`PARTITION_ID_SUPPLIER `?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1460: [HUDI-679] Make io package Spark free
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399699506
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/SparkTaskContextDetailSupplier.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.spark.TaskContext;
+
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * Spark Supplier.
+ */
+public interface SparkTaskContextDetailSupplier<T> extends Supplier<T>, Serializable {
Review comment:
Should be interface be something generic like `WriteTaskContextSupplier` which is extended by `SparkTaskContextSupplier` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services