You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/12/09 04:01:58 UTC
[GitHub] [flink-ml] weibozhao opened a new pull request, #189: [FLINK-30348] Refine Transformer for RandomSplitter
weibozhao opened a new pull request, #189:
URL: https://github.com/apache/flink-ml/pull/189
## This pull request refines Transformer for RandomSplitter
## Brief change log
- *Adds API setSeed() for RandomSplitter Transformer in Java and Python*
- *Adds ut for this new API.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on pull request #189: [FLINK-30348] Add HasSeed param for RandomSplitter
Posted by GitBox <gi...@apache.org>.
weibozhao commented on PR #189:
URL: https://github.com/apache/flink-ml/pull/189#issuecomment-1371875726
>
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 merged pull request #189: [FLINK-30348] Add HasSeed param for RandomSplitter
Posted by GitBox <gi...@apache.org>.
zhipeng93 merged PR #189:
URL: https://github.com/apache/flink-ml/pull/189
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on pull request #189: [FLINK-30348] Add HasSeed param for RandomSplitter
Posted by GitBox <gi...@apache.org>.
weibozhao commented on PR #189:
URL: https://github.com/apache/flink-ml/pull/189#issuecomment-1373185876
> @weibozhao Thanks for the update :) Can you also update the PR title as well as the commit message to make it more informative? (e.g., `Add HasSeed param for RandomSplitter`)
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on pull request #189: [FLINK-30348] Refine Transformer for RandomSplitter
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on PR #189:
URL: https://github.com/apache/flink-ml/pull/189#issuecomment-1371870055
Thanks for the PR. LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on a diff in pull request #189: [FLINK-30348] Refine Transformer for RandomSplitter
Posted by GitBox <gi...@apache.org>.
weibozhao commented on code in PR #189:
URL: https://github.com/apache/flink-ml/pull/189#discussion_r1057045262
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/randomsplitter/RandomSplitter.java:
##########
@@ -83,11 +84,12 @@ public Table[] transform(Table... inputs) {
private static class SplitterOperator extends AbstractStreamOperator<Row>
implements OneInputStreamOperator<Row, Row> {
- private final Random random = new Random(0);
+ private final Random random;
OutputTag<Row>[] outputTag;
final double[] fractions;
- public SplitterOperator(OutputTag<Row>[] outputTag, Double[] weights) {
+ public SplitterOperator(OutputTag<Row>[] outputTag, Double[] weights, long seed) {
+ random = new Random(seed);
Review Comment:
Sounds good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on a diff in pull request #189: [FLINK-30348] Refine Transformer for RandomSplitter
Posted by GitBox <gi...@apache.org>.
weibozhao commented on code in PR #189:
URL: https://github.com/apache/flink-ml/pull/189#discussion_r1057047880
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/RandomSplitterTest.java:
##########
@@ -95,7 +96,7 @@ public void testOutputSchema() {
@Test
public void testWeights() throws Exception {
Table data = getTable(1000);
- RandomSplitter splitter = new RandomSplitter().setWeights(2.0, 1.0, 2.0);
+ RandomSplitter splitter = new RandomSplitter().setWeights(2.0, 1.0, 2.0).setSeed(0);
Review Comment:
The default seed can good enough to generate split results that meet statistical expectations. I have remove the setSeed(0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on pull request #189: [FLINK-30348] Refine Transformer for RandomSplitter
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on PR #189:
URL: https://github.com/apache/flink-ml/pull/189#issuecomment-1347615471
Hi @weibozhao, The failure thrown in the CI of this PR has been fixed by a recently merged PR. Could you please rebase this PR onto the latest master code to ensure the tests can pass?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on a diff in pull request #189: [FLINK-30348] Add HasSeed param for RandomSplitter
Posted by GitBox <gi...@apache.org>.
weibozhao commented on code in PR #189:
URL: https://github.com/apache/flink-ml/pull/189#discussion_r1062181069
##########
docs/content/docs/operators/feature/randomsplitter.md:
##########
@@ -31,9 +31,10 @@ An AlgoOperator which splits a table into N tables according to the given weight
### Parameters
-| Key | Default | Type | Required | Description |
-|:--------|:-------------|:---------|:---------|:-------------------------------|
-| weights | `[1.0, 1.0]` | Double[] | no | The weights of data splitting. |
+| Key | Default | Type | Required | Description |
+|:--------|:-------------|:---------|:---------|:-------------------------------------------------------------------------------------------------------------------------------------|
+| weights | `[1.0, 1.0]` | Double[] | no | The weights of data splitting. |
+| seed | `null` | Long | no | The random seed. This parameter works only when the parallelism is consistent and each worker read the same data in the same order. |
Review Comment:
sounds good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #189: [FLINK-30348] Refine Transformer for RandomSplitter
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #189:
URL: https://github.com/apache/flink-ml/pull/189#discussion_r1046592801
##########
docs/content/docs/operators/feature/randomsplitter.md:
##########
@@ -34,6 +34,7 @@ An AlgoOperator which splits a table into N tables according to the given weight
| Key | Default | Type | Required | Description |
|:--------|:-------------|:---------|:---------|:-------------------------------|
| weights | `[1.0, 1.0]` | Double[] | no | The weights of data splitting. |
+| seed | `null` | Long | no | The random seed. |
Review Comment:
It might be better to describe the conditions required to reproduce random split results. For example, if users set the same random seed but changed the parallelism of the upstream operator, can they still expect to get the same split result?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/randomsplitter/RandomSplitter.java:
##########
@@ -83,11 +84,12 @@ public Table[] transform(Table... inputs) {
private static class SplitterOperator extends AbstractStreamOperator<Row>
implements OneInputStreamOperator<Row, Row> {
- private final Random random = new Random(0);
+ private final Random random;
OutputTag<Row>[] outputTag;
final double[] fractions;
- public SplitterOperator(OutputTag<Row>[] outputTag, Double[] weights) {
+ public SplitterOperator(OutputTag<Row>[] outputTag, Double[] weights, long seed) {
+ random = new Random(seed);
Review Comment:
It might be better to avoid having random values behaving the same on each subtask, e.g., always assigning the first element to the first output table regardless of the id of the subtask. You may check [`RowGenerator.open()`](https://github.com/apache/flink-ml/blob/master/flink-ml-benchmark/src/main/java/org/apache/flink/ml/benchmark/datagenerator/common/RowGenerator.java#L52) for how to create randoms from both initial seed and subtask id.
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/RandomSplitterTest.java:
##########
@@ -95,7 +96,7 @@ public void testOutputSchema() {
@Test
public void testWeights() throws Exception {
Table data = getTable(1000);
- RandomSplitter splitter = new RandomSplitter().setWeights(2.0, 1.0, 2.0);
+ RandomSplitter splitter = new RandomSplitter().setWeights(2.0, 1.0, 2.0).setSeed(0);
Review Comment:
It seems that the default seed is not good enough to generate split results that meet statistical expectations. Can we modify the random splitting behavior to improve the default splitting behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #189: [FLINK-30348] Refine Transformer for RandomSplitter
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on code in PR #189:
URL: https://github.com/apache/flink-ml/pull/189#discussion_r1062179082
##########
docs/content/docs/operators/feature/randomsplitter.md:
##########
@@ -31,9 +31,10 @@ An AlgoOperator which splits a table into N tables according to the given weight
### Parameters
-| Key | Default | Type | Required | Description |
-|:--------|:-------------|:---------|:---------|:-------------------------------|
-| weights | `[1.0, 1.0]` | Double[] | no | The weights of data splitting. |
+| Key | Default | Type | Required | Description |
+|:--------|:-------------|:---------|:---------|:-------------------------------------------------------------------------------------------------------------------------------------|
+| weights | `[1.0, 1.0]` | Double[] | no | The weights of data splitting. |
+| seed | `null` | Long | no | The random seed. This parameter works only when the parallelism is consistent and each worker read the same data in the same order. |
Review Comment:
How about using the following the descriptions:
'''
The random seed. This parameter guarantees reproduciable output only when the paralleism is unchanged and each worker reads the same data in the same order.
'''
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on pull request #189: [FLINK-30348] Refine Transformer for RandomSplitter
Posted by GitBox <gi...@apache.org>.
weibozhao commented on PR #189:
URL: https://github.com/apache/flink-ml/pull/189#issuecomment-1371647105
>
> Hi @weibozhao, thanks for the update. Could you please modify the description section of this PR and its corresponding Jira ticket, explaining the changes made in this PR and why we should add them to Flink ML?
I have modified the description of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on pull request #189: [FLINK-30348] Refine Transformer for RandomSplitter
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on PR #189:
URL: https://github.com/apache/flink-ml/pull/189#issuecomment-1369450594
Hi @weibozhao, thanks for the update. Could you please modify the description section of this PR and its corresponding Jira ticket, explaining the changes made in this PR and why we should add them to Flink ML?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #189: [FLINK-30348] Refine Transformer for RandomSplitter
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on code in PR #189:
URL: https://github.com/apache/flink-ml/pull/189#discussion_r1062179082
##########
docs/content/docs/operators/feature/randomsplitter.md:
##########
@@ -31,9 +31,10 @@ An AlgoOperator which splits a table into N tables according to the given weight
### Parameters
-| Key | Default | Type | Required | Description |
-|:--------|:-------------|:---------|:---------|:-------------------------------|
-| weights | `[1.0, 1.0]` | Double[] | no | The weights of data splitting. |
+| Key | Default | Type | Required | Description |
+|:--------|:-------------|:---------|:---------|:-------------------------------------------------------------------------------------------------------------------------------------|
+| weights | `[1.0, 1.0]` | Double[] | no | The weights of data splitting. |
+| seed | `null` | Long | no | The random seed. This parameter works only when the parallelism is consistent and each worker read the same data in the same order. |
Review Comment:
How about update the descriptions as:
'''
The random seed. This parameter guarantees reproduciable output only when the paralleism is unchanged and each worker reads the same data in the same order.
'''
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on pull request #189: [FLINK-30348] Refine Transformer for RandomSplitter
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on PR #189:
URL: https://github.com/apache/flink-ml/pull/189#issuecomment-1371872841
@weibozhao Thanks for the update :) Can you also update the PR title as well as the commit message to make it more informative? (e.g., `Add HasSeed param for RandomSplitter`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org