You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/06/26 07:33:02 UTC
[GitHub] [beam] Dobiasd opened a new pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Dobiasd opened a new pull request #15087:
URL: https://github.com/apache/beam/pull/15087
`GenericRecordToRowFn` in `AvroUtils` does not support the logical Avro type "timestamp-micros". Instead of converting it to `FieldType.DATETIME` (as it does with "timestamp-millis") it just interprets it as a raw LONG.
This pull request fixes that.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] ~Update `CHANGES.md` with noteworthy changes.~
- [ ] ~If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).~
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-874437043
The tests are based on the assumption, that in the conversion chain `Avro (A) -> Beam (B) -> Avro (C)` the schema in C is the same as in A.
For millisecond timestamps this works fine: `Avro (TimestampMillis) -> Beam (DATETIME) -> Avro (TimestampMillis)`
But for microsecond timestamps it does not: `Avro (TimestampMicros) -> Beam (DATETIME) -> Avro (TimestampMillis)`
Thus I've just pushed a commit that removed the (failing) tests for `TimestampMillis` again.
@iemejia Do you think this pull request is good to (squash-)merge despite that lack of additional tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd commented on a change in pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd commented on a change in pull request #15087:
URL: https://github.com/apache/beam/pull/15087#discussion_r659135263
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
##########
@@ -765,8 +765,8 @@ private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundE
if (logicalType instanceof LogicalTypes.Decimal) {
fieldType = FieldType.DECIMAL;
} else if (logicalType instanceof LogicalTypes.TimestampMillis) {
- // TODO: There is a desire to move Beam schema DATETIME to a micros representation. When
Review comment:
This is unrelated to the representation of Beam schema `DATETIME`. The correct TODO comment for this is here: https://github.com/apache/beam/blob/961cf0936d7065bf6089e666fbbc6b83da266df2/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L883-L884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-881497979
I believe that we still need the tests if we decide that this change is valid.
Actually, this conversion `Avro (TimestampMicros) -> Beam (DATETIME) -> Avro (TimestampMillis)` - this is really what actually is expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-874437043
The tests are based on the assumption, that in the conversion chain `Avro (A) -> Beam (B) -> Avro (C)` the schema in C is the same as in A.
For millisecond timestamps this works fine: `Avro (TimestampMillis) -> Beam (DATETIME) -> Avro (TimestampMillis)`
But for microsecond timestamps it does not: `Avro (TimestampMicros) -> Beam (DATETIME) -> Avro (TimestampMillis)`
Thus I've just pushed a commit that removed the (failing) tests for `TimestampMillis` again.
@iemejia Do you think this pull request is good to (squash-)merge despite that lack of additional tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-869953354
@iemejia
Thanks for the review. :+1:
I tried to add tests (see the latest commit), but it actually is quite hard to do correctly (they currently fail), because in the Beam schema (and thus row), the Avro microsecond timestamp is also represented as `FieldType.DATETIME` (same as the millisecond timestamp). So the back conversion, i.e., `AvroUtils.toAvroSchema(beamSchema)` in `testFromBeamSchema` and `AvroUtils.toGenericRecord(getBeamRow() ...);` in `testBeamRowToGenericRecordInferSchema`/`testBeamRowToGenericRecord`, returns an Avro schema/record with milliseconds instead of microseconds, which makes the equality check with the original schema/record (`assertEquals(getAvroSchema(), avroSchema);`/`assertEquals(getGenericRecord(), genericRecord);`) fail.
Since we probably don't want to introduce a new `FieldType.DATETIMEMICROS`, we'd need to somewhat hack around this "source resolution amnesia" in the test, which might create more harm than help.
Another way would be to add an additional, separate test (without back conversion check) just for the microsecond timestamp, but regarding the triviality of the implementation, this too currently seems to me like too much overhead. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd edited a comment on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd edited a comment on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-874437043
The tests are based on the assumption, that in the conversion chain `Avro (A) -> Beam (B) -> Avro (C)` the schema in C is the same as in A.
For millisecond timestamps this works fine: `Avro (TimestampMillis) -> Beam (DATETIME) -> Avro (TimestampMillis)`
But for microsecond timestamps it does not: `Avro (TimestampMicros) -> Beam (DATETIME) -> Avro (TimestampMillis)`
Thus I've just pushed a commit that removed the (failing) tests for `TimestampMicros` again.
@iemejia Do you think this pull request is good to (squash-)merge despite that lack of additional tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-895298513
@aromanenko-dev Sorry, not from my side. Honestly, I don't know what to do now, i.e., if I should try to understand and refactor the tests to support this new scenario, or not.
> Btw, are there any principal objections or difficulties to use a micros representation for DATETIME field?
I have no idea and I was hoping for somebody more experienced with the codebase than me to chime in here. :grimacing:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-870279192
# [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15087](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (75ba634) into [master](https://codecov.io/gh/apache/beam/commit/000ac07f2883d4708b6abbfc935fd004405aae19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (000ac07) will **increase** coverage by `1.22%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15087 +/- ##
==========================================
+ Coverage 82.55% 83.78% +1.22%
==========================================
Files 455 439 -16
Lines 55143 59237 +4094
==========================================
+ Hits 45526 49632 +4106
+ Misses 9617 9605 -12
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `0.00% <0.00%> (-92.60%)` | :arrow_down: |
| [...s/python/apache\_beam/examples/snippets/snippets.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvc25pcHBldHMucHk=) | `76.70% <0.00%> (-12.83%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/kafka.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8va2Fma2EucHk=) | `79.16% <0.00%> (-6.55%)` | :arrow_down: |
| [...n/apache\_beam/runners/direct/test\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9kaXJlY3RfcnVubmVyLnB5) | `37.50% <0.00%> (-4.81%)` | :arrow_down: |
| [test\_config.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGVzdF9jb25maWcucHk=) | `66.66% <0.00%> (-4.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/test/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy90ZXN0L19faW5pdF9fLnB5) | `66.66% <0.00%> (-4.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL19faW5pdF9fLnB5) | `80.00% <0.00%> (-4.62%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/channel\_factory.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvY2hhbm5lbF9mYWN0b3J5LnB5) | `75.00% <0.00%> (-3.95%)` | :arrow_down: |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `75.82% <0.00%> (-3.71%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/sdk\_worker\_main.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlcl9tYWluLnB5) | `72.26% <0.00%> (-3.66%)` | :arrow_down: |
| ... and [457 more](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [22205ee...75ba634](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-912495516
Sorry, I feel I don't have enough understanding of the bigger picture to build/express an opinion on that. I think I should leave it up to you guys.
---
In the meantime, I've found a different (non-apache-beam-based) solution to the problem I was working on when I created this pull request. So for me personally, it does not matter anymore, and I'd also be ok with this PR begin closed without further action.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-895213515
@Dobiasd Kind ping, any updates on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-870279192
# [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15087](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (75ba634) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 75ba634 differs from pull request most recent head 699d30c. Consider uploading reports for the commit 699d30c to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15087 +/- ##
==========================================
- Coverage 83.80% 83.78% -0.02%
==========================================
Files 441 439 -2
Lines 59539 59237 -302
==========================================
- Hits 49894 49632 -262
+ Misses 9645 9605 -40
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.17% <0.00%> (-1.07%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/kafka.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8va2Fma2EucHk=) | `79.16% <0.00%> (-0.84%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3NjaGVtYXMucHk=) | `96.06% <0.00%> (-0.79%)` | :arrow_down: |
| [.../apache\_beam/runners/direct/transform\_evaluator.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdHJhbnNmb3JtX2V2YWx1YXRvci5weQ==) | `89.78% <0.00%> (-0.66%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.47% <0.00%> (-0.43%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.60% <0.00%> (-0.26%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.57% <0.00%> (-0.09%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/dataframe/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL19faW5pdF9fLnB5) | `100.00% <0.00%> (ø)` | |
| [...ython/apache\_beam/examples/kafkataxi/kafka\_taxi.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMva2Fma2F0YXhpL2thZmthX3RheGkucHk=) | `0.00% <0.00%> (ø)` | |
| ... and [3 more](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...699d30c](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-881515402
Well, in this case I'd say that we need to test it in the same way as we do for `Avro(TimestampMillis)` but expecting milliseconds after two-sides conversion which actually can be quite ambiguous from the user point perspective (and that is why I'm asking if it's expected behaviour). Then in this case we need to warn users about that in documentation as well (JavaDoc and website).
Btw, are there any principal objections or difficulties to use a micros representation for DATETIME field?
Afaik, @iemejia is OOO this week but he will be back soon and maybe can add something on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-869953354
@iemejia
Thanks for the review. :+1:
I tried to add tests (see the latest commit), but it actually is quite hard to do correctly (they currently fail), because in the Beam schema (and thus row), the Avro microsecond timestamp is also represented as `FieldType.DATETIME` (same as the millisecond timestamp). So the back conversion, i.e., `AvroUtils.toAvroSchema(beamSchema)` in `testFromBeamSchema` and `AvroUtils.toGenericRecord(getBeamRow() ...);` in `testBeamRowToGenericRecordInferSchema`/`testBeamRowToGenericRecord`, returns an Avro schema/record with milliseconds instead of microseconds, which makes the equality check with the original schema/record (`assertEquals(getAvroSchema(), avroSchema);`/`assertEquals(getGenericRecord(), genericRecord);`) fail.
Since we probably don't want to introduce a new `FieldType.DATETIMEMICROS`, we'd need to somewhat hack around this "source resolution amnesia" in the test, which might create more harm than help.
Another way would be to add an additional, separate test (without back conversion check) just for the microsecond timestamp, but regarding the triviality of the implementation, this too currently seems to me like too much overhead. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd commented on a change in pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd commented on a change in pull request #15087:
URL: https://github.com/apache/beam/pull/15087#discussion_r659135263
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
##########
@@ -765,8 +765,8 @@ private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundE
if (logicalType instanceof LogicalTypes.Decimal) {
fieldType = FieldType.DECIMAL;
} else if (logicalType instanceof LogicalTypes.TimestampMillis) {
- // TODO: There is a desire to move Beam schema DATETIME to a micros representation. When
Review comment:
This is unrelated to the representation of Beam schema `DATETIME`. The correct TODO comment for this is here: https://github.com/apache/beam/blob/961cf0936d7065bf6089e666fbbc6b83da266df2/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L883-L884
So the comment here can (and should) be removed, even independently of the change in this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev edited a comment on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
aromanenko-dev edited a comment on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-912476256
@Dobiasd Kind ping. What do you think about Ryan's suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-900442016
cc: @RyanSkraba Could you take a look, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] RyanSkraba commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
RyanSkraba commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-914112387
Unfortunately, I think we should close this PR and refine the requirements. The strictly "right" thing to do would probably be to implement a Beam date type with a finer granularity :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-870279192
# [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15087](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (75ba634) into [master](https://codecov.io/gh/apache/beam/commit/000ac07f2883d4708b6abbfc935fd004405aae19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (000ac07) will **increase** coverage by `1.22%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15087 +/- ##
==========================================
+ Coverage 82.55% 83.78% +1.22%
==========================================
Files 455 439 -16
Lines 55143 59237 +4094
==========================================
+ Hits 45526 49632 +4106
+ Misses 9617 9605 -12
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `0.00% <0.00%> (-92.60%)` | :arrow_down: |
| [...s/python/apache\_beam/examples/snippets/snippets.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvc25pcHBldHMucHk=) | `76.70% <0.00%> (-12.83%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/kafka.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8va2Fma2EucHk=) | `79.16% <0.00%> (-6.55%)` | :arrow_down: |
| [...n/apache\_beam/runners/direct/test\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9kaXJlY3RfcnVubmVyLnB5) | `37.50% <0.00%> (-4.81%)` | :arrow_down: |
| [test\_config.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGVzdF9jb25maWcucHk=) | `66.66% <0.00%> (-4.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/test/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy90ZXN0L19faW5pdF9fLnB5) | `66.66% <0.00%> (-4.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL19faW5pdF9fLnB5) | `80.00% <0.00%> (-4.62%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/channel\_factory.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvY2hhbm5lbF9mYWN0b3J5LnB5) | `75.00% <0.00%> (-3.95%)` | :arrow_down: |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `75.82% <0.00%> (-3.71%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/sdk\_worker\_main.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlcl9tYWluLnB5) | `72.26% <0.00%> (-3.66%)` | :arrow_down: |
| ... and [457 more](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [22205ee...75ba634](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd closed pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd closed pull request #15087:
URL: https://github.com/apache/beam/pull/15087
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd edited a comment on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd edited a comment on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-874437043
The tests are based on the assumption, that in the conversion chain `Avro (A) -> Beam (B) -> Avro (C)` the schema in C is the same as in A.
For millisecond timestamps this works fine: `Avro (TimestampMillis) -> Beam (DATETIME) -> Avro (TimestampMillis)`
But for microsecond timestamps it does not: `Avro (TimestampMicros) -> Beam (DATETIME) -> Avro (TimestampMillis)`
Thus I've just pushed a commit that removed the (failing) tests for `TimestampMicros` again.
@iemejia Do you think this pull request is good to (squash-)merge despite that lack of additional tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-912476256
@Dobiasd What do you think about Ryan's suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-870279192
# [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15087](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (75ba634) into [master](https://codecov.io/gh/apache/beam/commit/000ac07f2883d4708b6abbfc935fd004405aae19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (000ac07) will **increase** coverage by `1.22%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15087 +/- ##
==========================================
+ Coverage 82.55% 83.78% +1.22%
==========================================
Files 455 439 -16
Lines 55143 59237 +4094
==========================================
+ Hits 45526 49632 +4106
+ Misses 9617 9605 -12
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `0.00% <0.00%> (-92.60%)` | :arrow_down: |
| [...s/python/apache\_beam/examples/snippets/snippets.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvc25pcHBldHMucHk=) | `76.70% <0.00%> (-12.83%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/kafka.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8va2Fma2EucHk=) | `79.16% <0.00%> (-6.55%)` | :arrow_down: |
| [...n/apache\_beam/runners/direct/test\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9kaXJlY3RfcnVubmVyLnB5) | `37.50% <0.00%> (-4.81%)` | :arrow_down: |
| [test\_config.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGVzdF9jb25maWcucHk=) | `66.66% <0.00%> (-4.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/test/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy90ZXN0L19faW5pdF9fLnB5) | `66.66% <0.00%> (-4.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL19faW5pdF9fLnB5) | `80.00% <0.00%> (-4.62%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/channel\_factory.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvY2hhbm5lbF9mYWN0b3J5LnB5) | `75.00% <0.00%> (-3.95%)` | :arrow_down: |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `75.82% <0.00%> (-3.71%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/sdk\_worker\_main.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlcl9tYWluLnB5) | `72.26% <0.00%> (-3.66%)` | :arrow_down: |
| ... and [457 more](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [22205ee...75ba634](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-900442016
cc: @RyanSkraba Could you take a look, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-870279192
# [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15087](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (75ba634) into [master](https://codecov.io/gh/apache/beam/commit/000ac07f2883d4708b6abbfc935fd004405aae19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (000ac07) will **increase** coverage by `1.22%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15087 +/- ##
==========================================
+ Coverage 82.55% 83.78% +1.22%
==========================================
Files 455 439 -16
Lines 55143 59237 +4094
==========================================
+ Hits 45526 49632 +4106
+ Misses 9617 9605 -12
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `0.00% <0.00%> (-92.60%)` | :arrow_down: |
| [...s/python/apache\_beam/examples/snippets/snippets.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvc25pcHBldHMucHk=) | `76.70% <0.00%> (-12.83%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/kafka.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8va2Fma2EucHk=) | `79.16% <0.00%> (-6.55%)` | :arrow_down: |
| [...n/apache\_beam/runners/direct/test\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9kaXJlY3RfcnVubmVyLnB5) | `37.50% <0.00%> (-4.81%)` | :arrow_down: |
| [test\_config.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGVzdF9jb25maWcucHk=) | `66.66% <0.00%> (-4.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/test/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy90ZXN0L19faW5pdF9fLnB5) | `66.66% <0.00%> (-4.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL19faW5pdF9fLnB5) | `80.00% <0.00%> (-4.62%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/channel\_factory.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvY2hhbm5lbF9mYWN0b3J5LnB5) | `75.00% <0.00%> (-3.95%)` | :arrow_down: |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `75.82% <0.00%> (-3.71%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/sdk\_worker\_main.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlcl9tYWluLnB5) | `72.26% <0.00%> (-3.66%)` | :arrow_down: |
| ... and [457 more](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [22205ee...75ba634](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-881153886
/cc @aromanenko-dev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-870279192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-870279192
# [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15087](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (75ba634) into [master](https://codecov.io/gh/apache/beam/commit/22205ee1a84581e9206c5c61bad88a799779b4bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (22205ee) will **not change** coverage.
> The diff coverage is `n/a`.
> :exclamation: Current head 75ba634 differs from pull request most recent head 818d4ed. Consider uploading reports for the commit 818d4ed to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15087 +/- ##
=======================================
Coverage 83.78% 83.78%
=======================================
Files 439 439
Lines 59237 59237
=======================================
Hits 49632 49632
Misses 9605 9605
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.47% <0.00%> (-1.39%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.71% <0.00%> (+0.52%)` | :arrow_up: |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/15087/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `75.82% <0.00%> (+1.09%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [22205ee...818d4ed](https://codecov.io/gh/apache/beam/pull/15087?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-881503501
Yeah, this is expected. It's just that the current test setup, to me at least, looks like it's not prepared for that. Any suggestions for a good approach on how to add a test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-870279192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Dobiasd commented on pull request #15087: [BEAM-12544] Add support for Avro timestamps in microseconds
Posted by GitBox <gi...@apache.org>.
Dobiasd commented on pull request #15087:
URL: https://github.com/apache/beam/pull/15087#issuecomment-868964184
R: @iemejia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org