You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/02 23:12:41 UTC
[GitHub] [beam] TheNeuralBit opened a new pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
TheNeuralBit opened a new pull request #12764:
URL: https://github.com/apache/beam/pull/12764
This PR adds support for `beam:logical_type:micros_instant:v1` in Python and Java. It also adds logic for translating logical types which do not use the argument to portable Beam schemas.
In Python, `micros_instant:v1` is the default mapping for the microsecond-precision `beam.utils.timestamp.Timestamp` type when inferring a schema.
The change is tested via unit tests in Python and Java, as well as a new entry in `standard_coders_test.yaml`.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on a change in pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on a change in pull request #12764:
URL: https://github.com/apache/beam/pull/12764#discussion_r487248777
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
I see. Thanks!
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
I see. Thanks!
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
I see. Thanks!
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
I see. Thanks!
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
I see. Thanks!
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
I see. Thanks!
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
I see. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12764:
URL: https://github.com/apache/beam/pull/12764#discussion_r483313450
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
+
+ public MicrosInstant() {
+ this.schema = Schema.builder().addInt64Field("seconds").addInt32Field("micros").build();
+ }
+
+ @Override
+ public Row toBaseType(Instant input) {
Review comment:
That's a good question.. I don't think we have an answer for it documented anywhere. I think we should probably handle null checking outside of `toBaseType` and `toInputType`, so that we only do it when the type is marked `nullable`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
# [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=h1) Report
> Merging [#12764](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `57.02%`.
[](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12764 +/- ##
==========================================
+ Coverage 40.23% 40.29% +0.05%
==========================================
Files 455 455
Lines 53726 53841 +115
==========================================
+ Hits 21619 21695 +76
- Misses 32107 32146 +39
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `41.54% <45.45%> (+0.71%)` | :arrow_up: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `49.02% <65.55%> (+13.43%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/timestamp.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdGltZXN0YW1wLnB5) | `50.25% <0.00%> (+3.55%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=footer). Last update [9f4d54a...25618a6](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #12764:
URL: https://github.com/apache/beam/pull/12764#discussion_r483322224
##########
File path: sdks/python/apache_beam/typehints/schemas.py
##########
@@ -279,3 +301,184 @@ def schema_from_element_type(element_type): # (type) -> schema_pb2.Schema
def named_fields_from_element_type(
element_type): # (type) -> typing.List[typing.Tuple[unicode, type]]
return named_fields_from_schema(schema_from_element_type(element_type))
+
+
+# Registry of typings for a schema by UUID
+class LogicalTypeRegistry(object):
+ def __init__(self):
+ self.by_urn = {}
+ self.by_logical_type = {}
+ self.by_language_type = {}
+
+ def add(self, urn, logical_type):
+ self.by_urn[urn] = logical_type
+ self.by_logical_type[logical_type] = urn
+ self.by_language_type[logical_type.language_type()] = logical_type
+
+ def get_logical_type_by_urn(self, urn):
+ return self.by_urn.get(urn, None)
+
+ def get_urn_by_logial_type(self, logical_type):
+ return self.by_logical_type.get(logical_type, None)
+
+ def get_logical_type_by_language_type(self, representation_type):
+ return self.by_language_type.get(representation_type, None)
+
+
+LanguageT = TypeVar('LanguageT')
+RepresentationT = TypeVar('RepresentationT')
+ArgT = TypeVar('ArgT')
+
+
+class LogicalType(Generic[LanguageT, RepresentationT, ArgT]):
+ _known_logical_types = LogicalTypeRegistry()
+
+ @classmethod
+ def urn(cls):
+ # type: () -> unicode
+
+ """Return the URN used to identify this logical type"""
+ raise NotImplementedError()
+
+ @classmethod
+ def language_type(cls):
+ # type: () -> type
+
+ """Return the language type this LogicalType encodes.
+
+ The returned type should match LanguageT"""
+ raise NotImplementedError()
+
+ @classmethod
+ def representation_type(cls):
+ # type: () -> type
+
+ """Return the type of the representation this LogicalType uses to encode the
+ language type.
+
+ The returned type should match RepresentationT"""
+ raise NotImplementedError()
+
+ @classmethod
+ def argument_type(cls):
+ # type: () -> type
+
+ """Return the type of the argument used for variations of this LogicalType.
+
+ The returned type should match ArgT"""
+ raise NotImplementedError(cls)
+
+ def argument(self):
+ # type: () -> ArgT
+
+ """Return the argument for this instance of the LogicalType."""
+ raise NotImplementedError()
+
+ def to_representation_type(value):
+ # type: (LanguageT) -> RepresentationT
+
+ """Convert an instance of LanguageT to RepresentationT."""
+ raise NotImplementedError()
+
+ def to_language_type(value):
+ # type: (RepresentationT) -> LanguageT
+
+ """Convert an instance of RepresentationT to LanguageT."""
+ raise NotImplementedError()
+
+ @classmethod
+ def register_logical_type(cls, logical_type_cls):
+ """Register an implementation of LogicalType."""
+ cls._known_logical_types.add(logical_type_cls.urn(), logical_type_cls)
+
+ @classmethod
+ def from_typing(cls, typ):
+ # type: (type) -> LogicalType
+
+ """Construct an instance of a registered LogicalType implementation given a
+ typing.
+
+ Raises ValueError if no registered LogicalType implementation can encode the
+ given typing."""
+
+ logical_type = cls._known_logical_types.get_logical_type_by_language_type(
+ typ)
+ if logical_type is None:
+ raise ValueError("No logical type registered for typing '%s'" % typ)
+
+ return logical_type._from_typing(typ)
+
+ @classmethod
+ def _from_typing(cls, typ):
+ # type: (type) -> LogicalType
+
+ """Construct an instance of this LogicalType implementation given a typing.
+ """
+ raise NotImplementedError()
+
+ @classmethod
+ def from_runner_api(cls, logical_type_proto):
+ # type: (schema_pb2.LogicalType) -> LogicalType
+
+ """Construct an instance of a registered LogicalType implementation given a
+ proto LogicalType.
+
+ Raises ValueError if no LogicalType registered for the given URN.
+ """
+ logical_type = cls._known_logical_types.get_logical_type_by_urn(
+ logical_type_proto.urn)
+ if logical_type is None:
+ raise ValueError(
+ "No logical type registered for URN '%s'" % logical_type_proto.urn)
+ # TODO(bhulette): Use argument
+ return logical_type()
+
+
+class NoArgumentLogicalType(LogicalType[LanguageT, RepresentationT, None]):
+ @classmethod
+ def argument_type(cls):
+ # type: () -> type
+ return None
+
+ def argument(self):
+ # type: () -> ArgT
+ return None
+
+ @classmethod
+ def _from_typing(cls, typ):
+ # type: (type) -> LogicalType
+
+ # Sicne there's no argument, there can be no additional information encoded
Review comment:
Since.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
# [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=h1) Report
> Merging [#12764](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `57.02%`.
[](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12764 +/- ##
==========================================
+ Coverage 40.23% 40.29% +0.05%
==========================================
Files 455 455
Lines 53726 53841 +115
==========================================
+ Hits 21619 21695 +76
- Misses 32107 32146 +39
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `41.54% <45.45%> (+0.71%)` | :arrow_up: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `49.02% <65.55%> (+13.43%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/timestamp.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdGltZXN0YW1wLnB5) | `50.25% <0.00%> (+3.55%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=footer). Last update [9f4d54a...25618a6](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686580958
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
# [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=h1) Report
> Merging [#12764](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `57.02%`.
[](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12764 +/- ##
==========================================
+ Coverage 40.23% 40.29% +0.05%
==========================================
Files 455 455
Lines 53726 53841 +115
==========================================
+ Hits 21619 21695 +76
- Misses 32107 32146 +39
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `41.54% <45.45%> (+0.71%)` | :arrow_up: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `49.02% <65.55%> (+13.43%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/timestamp.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdGltZXN0YW1wLnB5) | `50.25% <0.00%> (+3.55%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=footer). Last update [9f4d54a...25618a6](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12764:
URL: https://github.com/apache/beam/pull/12764#discussion_r486709165
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
There's an issue with Dataflow deserializing the logical type if the Schema is static, see BEAM-9113. There should probably be a follow-up to fix that though.. I'll file a jira and add a TODO here and in NanosInstant.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-691173231
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #12764:
URL: https://github.com/apache/beam/pull/12764
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-691173231
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-691173231
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #12764:
URL: https://github.com/apache/beam/pull/12764
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
# [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=h1) Report
> Merging [#12764](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/9f4d54a2e60ec45150437c0b050f4c73cca91f36?el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `61.60%`.
[](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12764 +/- ##
==========================================
+ Coverage 40.23% 40.29% +0.05%
==========================================
Files 455 455
Lines 53731 53841 +110
==========================================
+ Hits 21619 21695 +76
- Misses 32112 32146 +34
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `41.54% <45.45%> (+0.71%)` | :arrow_up: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `49.02% <65.55%> (+13.43%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/timestamp.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdGltZXN0YW1wLnB5) | `50.25% <0.00%> (+3.55%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=footer). Last update [9f4d54a...25618a6](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
# [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=h1) Report
> Merging [#12764](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/9f4d54a2e60ec45150437c0b050f4c73cca91f36?el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `61.60%`.
[](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12764 +/- ##
==========================================
+ Coverage 40.23% 40.29% +0.05%
==========================================
Files 455 455
Lines 53731 53841 +110
==========================================
+ Hits 21619 21695 +76
- Misses 32112 32146 +34
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `41.54% <45.45%> (+0.71%)` | :arrow_up: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `49.02% <65.55%> (+13.43%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/timestamp.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdGltZXN0YW1wLnB5) | `50.25% <0.00%> (+3.55%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=footer). Last update [9f4d54a...25618a6](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
# [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=h1) Report
> Merging [#12764](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `57.02%`.
[](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12764 +/- ##
==========================================
+ Coverage 40.23% 40.29% +0.05%
==========================================
Files 455 455
Lines 53726 53841 +115
==========================================
+ Hits 21619 21695 +76
- Misses 32107 32146 +39
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `41.54% <45.45%> (+0.71%)` | :arrow_up: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `49.02% <65.55%> (+13.43%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/timestamp.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdGltZXN0YW1wLnB5) | `50.25% <0.00%> (+3.55%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=footer). Last update [9f4d54a...25618a6](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #12764:
URL: https://github.com/apache/beam/pull/12764
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on a change in pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on a change in pull request #12764:
URL: https://github.com/apache/beam/pull/12764#discussion_r487248777
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
I see. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686085979
R: @robinyq for Java changes
R: @robertwb for Python changes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on a change in pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on a change in pull request #12764:
URL: https://github.com/apache/beam/pull/12764#discussion_r487248777
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
I see. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
# [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=h1) Report
> Merging [#12764](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/9f4d54a2e60ec45150437c0b050f4c73cca91f36?el=desc) will **increase** coverage by `41.81%`.
> The diff coverage is `87.50%`.
[](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12764 +/- ##
===========================================
+ Coverage 40.23% 82.04% +41.81%
===========================================
Files 455 457 +2
Lines 53731 54180 +449
===========================================
+ Hits 21619 44453 +22834
+ Misses 32112 9727 -22385
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (+57.61%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (+53.53%)` | :arrow_up: |
| [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | |
| [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/utils/histogram.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaGlzdG9ncmFtLnB5) | `94.28% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
| ... and [275 more](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=footer). Last update [9f4d54a...fcd88ac](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
# [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=h1) Report
> Merging [#12764](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/9f4d54a2e60ec45150437c0b050f4c73cca91f36?el=desc) will **increase** coverage by `41.81%`.
> The diff coverage is `87.50%`.
[](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12764 +/- ##
===========================================
+ Coverage 40.23% 82.04% +41.81%
===========================================
Files 455 457 +2
Lines 53731 54180 +449
===========================================
+ Hits 21619 44453 +22834
+ Misses 32112 9727 -22385
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (+57.61%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (+53.53%)` | :arrow_up: |
| [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | |
| [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/utils/histogram.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaGlzdG9ncmFtLnB5) | `94.28% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
| ... and [275 more](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=footer). Last update [9f4d54a...fcd88ac](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on a change in pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on a change in pull request #12764:
URL: https://github.com/apache/beam/pull/12764#discussion_r487248777
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
I see. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on a change in pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on a change in pull request #12764:
URL: https://github.com/apache/beam/pull/12764#discussion_r483157879
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
Review comment:
I think this can be a constant. We don't need to initialize it in constructor.
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
+
+ public MicrosInstant() {
+ this.schema = Schema.builder().addInt64Field("seconds").addInt32Field("micros").build();
+ }
+
+ @Override
+ public Row toBaseType(Instant input) {
Review comment:
I see other logical types return handles null input in this function and `toInputType()` as well. Should we also add it here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on a change in pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on a change in pull request #12764:
URL: https://github.com/apache/beam/pull/12764#discussion_r483157390
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MicrosInstant.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.logicaltypes;
+
+import java.time.Instant;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * A timestamp represented as microseconds since the epoch.
+ *
+ * <p>WARNING: This logical type exists solely for interoperability with other type systems such as
+ * SQL and other Beam SDKs. It should never be used in a native Java context where the {@code
+ * java.time.Instant} instances it describes may have higher than microsecond precision. Ignoring
+ * this will likely result in an {@code AssertionError} at pipeline execution time.
+ *
+ * <p>For a more faithful logical type to use with {@code java.time.Instant}, see {@link
+ * NanosInstant}.
+ */
+public class MicrosInstant implements Schema.LogicalType<Instant, Row> {
+ public static final String IDENTIFIER = "beam:logical_type:micros_instant:v1";
+ private final Schema schema;
+
+ public MicrosInstant() {
+ this.schema = Schema.builder().addInt64Field("seconds").addInt32Field("micros").build();
+ }
+
+ @Override
+ public Row toBaseType(Instant input) {
Review comment:
I see other logical types handle null input in this function and `toInputType()` as well. Should we also add it here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
# [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=h1) Report
> Merging [#12764](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `57.02%`.
[](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12764 +/- ##
==========================================
+ Coverage 40.23% 40.29% +0.05%
==========================================
Files 455 455
Lines 53726 53841 +115
==========================================
+ Hits 21619 21695 +76
- Misses 32107 32146 +39
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `41.54% <45.45%> (+0.71%)` | :arrow_up: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `49.02% <65.55%> (+13.43%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/timestamp.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdGltZXN0YW1wLnB5) | `50.25% <0.00%> (+3.55%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=footer). Last update [9f4d54a...25618a6](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-691234584
I think we're at about 5 nines of flake probability on the Python GH actions. Looks like the current failure is BEAM-10768, I'm just going to go ahead and merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
# [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=h1) Report
> Merging [#12764](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/9f4d54a2e60ec45150437c0b050f4c73cca91f36?el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `61.60%`.
[](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12764 +/- ##
==========================================
+ Coverage 40.23% 40.29% +0.05%
==========================================
Files 455 455
Lines 53731 53841 +110
==========================================
+ Hits 21619 21695 +76
- Misses 32112 32146 +34
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `41.54% <45.45%> (+0.71%)` | :arrow_up: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `49.02% <65.55%> (+13.43%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/timestamp.py](https://codecov.io/gh/apache/beam/pull/12764/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdGltZXN0YW1wLnB5) | `50.25% <0.00%> (+3.55%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=footer). Last update [9f4d54a...fcd88ac](https://codecov.io/gh/apache/beam/pull/12764?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #12764:
URL: https://github.com/apache/beam/pull/12764
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686581003
Run PythonDocker PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686100810
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit edited a comment on pull request #12764: [BEAM-10009] Add beam:logical_type:micros_instant:v1
Posted by GitBox <gi...@apache.org>.
TheNeuralBit edited a comment on pull request #12764:
URL: https://github.com/apache/beam/pull/12764#issuecomment-686085979
R: @robinyqiu for Java changes
R: @robertwb for Python changes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org