You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2024/11/15 20:38:51 UTC
[I] [Bug]: Certain nullable type resolved to not null coder after #32705 [beam]
Abacn opened a new issue, #33132:
URL: https://github.com/apache/beam/issues/33132
### What happened?
Caught by DataflowTemplates validation for Beam 2.61.0rc1: https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/2014/checks?check_run_id=33057418339
Was able to bisect:
good: 20d0f6e5a85c6b5738098e13e212337d86f49412
bad: c243491254896e039e912662e4cfbe4bd38c766f
To reproduce,
1. checkout https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/2014
2. on beam repo, after making any change, run
On Beam repo, after making any change (e.g. add log)
```
git cherry-pick c53a2017ba6e22816c8346fc8ce50142d30e3f81
./gradlew :sdks:java:core:publishToMavenLocal -Ppublishing
./gradlew :runners:direct-java:publishToMavenLocal -Ppublishing
```
3. On DataflowTemplate repo, run
mvn test -pl v2/dataplex -Dtest=DataplexBigQueryToGcsTest#testE2E_mainPathWithAllStepsEnabled
### Issue Priority
Priority: 1 (data loss / total loss of function)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Certain nullable type resolved to not null coder after #32705 [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33132:
URL: https://github.com/apache/beam/issues/33132#issuecomment-2916185337
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Certain nullable type resolved to not null coder after #32705 [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #33132: [Bug]: Certain nullable type resolved to not null coder after #32705
URL: https://github.com/apache/beam/issues/33132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Certain nullable type resolved to not null coder after #32705 [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on issue #33132:
URL: https://github.com/apache/beam/issues/33132#issuecomment-2487333449
Can you explain what the bug was (other than some tests started failing)? It's not clear to me from this description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Certain nullable type resolved to not null coder after #32705 [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #33132:
URL: https://github.com/apache/beam/issues/33132#issuecomment-2483944068
This has been fixed via https://github.com/apache/beam/pull/33133 - leaving open since we had to revert the PR from @reuvenlax and we probably want the right fix going forward here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Certain nullable type resolved to not null coder after #32705 [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33132:
URL: https://github.com/apache/beam/issues/33132#issuecomment-2498539681
> I see. So the expected result would be that ValueType should be NullableCoder<SchemaCoder>?
> […](#)
> On Wed, Nov 20, 2024 at 7:46 AM Yi Hu ***@***.***> wrote: The bug was that it used to be able to encode elements of KV<KeyType, @nullable ValueType> where ValueType uses a SchemaCoder, but now it cannot. — Reply to this email directly, view it on GitHub <[#33132 (comment)](https://github.com/apache/beam/issues/33132#issuecomment-2488937300)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAYJVPZUBOO6P52ZRX6FP32BSVF5AVCNFSM6AAAAABR35SLKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBYHEZTOMZQGA> . You are receiving this because you were assigned.Message ID: <apache/beam/issues/33132/2488937300 ***@***.***>
Yeah this would be a solution. I was wondering how it worked before the change. Did it use SerializableCoder?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Certain nullable type resolved to not null coder after #32705 [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on issue #33132:
URL: https://github.com/apache/beam/issues/33132#issuecomment-2492564892
I see. So the expected result would be that ValueType should be
NullableCoder<SchemaCoder>?
On Wed, Nov 20, 2024 at 7:46 AM Yi Hu ***@***.***> wrote:
> The bug was that it used to be able to encode elements of KV<KeyType,
> @Nullable ValueType> where ValueType uses a SchemaCoder, but now it
> cannot.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/issues/33132#issuecomment-2488937300>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVPZUBOO6P52ZRX6FP32BSVF5AVCNFSM6AAAAABR35SLKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBYHEZTOMZQGA>
> .
> You are receiving this because you were assigned.Message ID:
> <apache/beam/issues/33132/2488937300 ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Certain nullable type resolved to not null coder after #32705 [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33132:
URL: https://github.com/apache/beam/issues/33132#issuecomment-2488937300
The bug was that it used to be able to encode elements of KV<KeyType, @Nullable ValueType> where ValueType uses a SchemaCoder, but now it cannot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Certain nullable type resolved to not null coder after #32705 [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33132:
URL: https://github.com/apache/beam/issues/33132#issuecomment-3016666757
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Certain nullable type resolved to not null coder after #32705 [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33132:
URL: https://github.com/apache/beam/issues/33132#issuecomment-2480002050
A simple test: succeeded prior to the change, failing after the change:
```java
package org.apache.beam.runners.direct;
import com.google.auto.value.AutoValue;
import java.io.Serializable;
import org.apache.beam.sdk.coders.DefaultCoder;
import org.apache.beam.sdk.schemas.AutoValueSchema;
import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
import org.checkerframework.checker.nullness.qual.Nullable;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@RunWith(JUnit4.class)
public class NullableEvaluationTest {
@Rule
public TestPipeline p = TestPipeline.create();
@Test
public void testConstruction() {
// Either annotated with @Nullable or not does not matter
p.apply(Create.of(KV.<String, @Nullable BigQueryTablePartition>of("123", null)))
.apply(
"MapFileNames",
MapElements.into(TypeDescriptors.kvs(
TypeDescriptor.of(BigQueryTablePartition.class),
TypeDescriptor.of(String.class)))
.via((SerializableFunction<KV<String, BigQueryTablePartition>, KV<BigQueryTablePartition, String>>) kv -> KV.of(kv.getValue(), kv.getKey())));
p.run();
}
/** BigQuery table partition metadata. */
@AutoValue
@DefaultCoder(SchemaCoder.class)
@DefaultSchema(AutoValueSchema.class)
public abstract static class BigQueryTablePartition implements Serializable {
public abstract String getPartitionName();
public static Builder builder() {
return new AutoValue_NullableEvaluationTest_BigQueryTablePartition.Builder();
}
/** Builder for {@link BigQueryTablePartition}. */
@AutoValue.Builder
public abstract static class Builder {
public abstract Builder setPartitionName(String value);
public abstract BigQueryTablePartition build();
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org