You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/07/01 03:40:00 UTC
[jira] [Work logged] (BEAM-4700) JDBC driver cannot support
TIMESTAMP data type
[ https://issues.apache.org/jira/browse/BEAM-4700?focusedWorklogId=117935&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-117935 ]
ASF GitHub Bot logged work on BEAM-4700:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Jul/18 03:39
Start Date: 01/Jul/18 03:39
Worklog Time Spent: 10m
Work Description: kennknowles opened a new pull request #5849: [BEAM-4700] Convert Beam Row to Avatica Row in BeamEnumerableCollector
URL: https://github.com/apache/beam/pull/5849
This is a fairly surgical fix that makes timestamps work.
It is not at all a comprehensive solution to the general problem of establishing a proper compatibility between Beam `Row` and Avatica's meta schema stuff. I suspect we can override `Cursor.getAccessors` at the right place to avoid conversion at all. Since there's no meaningful documentation, it will take a minute to figure that out.
------------------------
Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 117935)
Time Spent: 10m
Remaining Estimate: 0h
> JDBC driver cannot support TIMESTAMP data type
> ----------------------------------------------
>
> Key: BEAM-4700
> URL: https://issues.apache.org/jira/browse/BEAM-4700
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
> Reporter: Kenneth Knowles
> Assignee: Kenneth Knowles
> Priority: Blocker
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Avatica allows column representation to be customized, so a timestamp can be stored as a variety of types. Joda ReadableInstant is none of these types: https://github.com/apache/calcite-avatica/blob/acb675de97b9b0743c09368820a770e2ceda05f8/core/src/main/java/org/apache/calcite/avatica/util/AbstractCursor.java#L162
> By default, it seems to be configured to store {{TIMESTAMP}} columns as {{long}} values. If you run the SQL shell and select a {{TIMESTAMP}} column, you get:
> {code}
> ava.lang.ClassCastException: org.joda.time.Instant cannot be cast to java.lang.Number
> at org.apache.beam.repackaged.beam_sdks_java_extensions_sql.org.apache.calcite.avatica.util.AbstractCursor$NumberAccessor.getNumber(AbstractCursor.java:726)
> at org.apache.beam.repackaged.beam_sdks_java_extensions_sql.org.apache.calcite.avatica.util.AbstractCursor$TimestampFromNumberAccessor.getString(AbstractCursor.java:1026)
> at org.apache.beam.repackaged.beam_sdks_java_extensions_sql.org.apache.calcite.avatica.AvaticaResultSet.getString(AvaticaResultSet.java:225)
> at sqlline.Rows$Row.<init>(Rows.java:183)
> {code}
> So, essentially, Beam SQL Shell does not support timestamps.
> We may be able to:
> - override how the accessor for our existing storage is created
> - configure what the column representation is (this doesn't really help, since none of the choices are ours)
> - convert timestamps to longs in BeamEnumerableConverter; not sure how many conversions will be required here
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)