You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/02/08 18:37:30 UTC
[GitHub] [beam] dpcollins-google opened a new pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
dpcollins-google opened a new pull request #13920:
URL: https://github.com/apache/beam/pull/13920
Implement a TableProvider for Pub/Sub Lite, enabling it to be used in BeamSQL
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam
.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-786229336
I will try to start review this PR this week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-783079555
cc @robinyqiu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-791057248
I am trying to triage BEAM-11921 to get it solved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-793057389
@dpcollins-google can you rebase? https://github.com/apache/beam/pull/14148 should fix the failing tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-787669325
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-794225733
Run Java_Examples_Dataflow PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on a change in pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on a change in pull request #13920:
URL: https://github.com/apache/beam/pull/13920#discussion_r584431988
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##########
@@ -949,6 +950,40 @@ public int hashCode() {
getMetadata()
});
}
+
+ @Override
+ public String toString() {
Review comment:
This is FieldType, without it, its just a dump of all the fields in a json-ish structure, which makes it hard to determine at a glance what is going on in debug prints, and makes it not a great method to present to end users when saying "this field is badly formatted". This makes the output very similar to the DDL you would have to type to describe a field of this type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-791023777
The failed tests seem are not flaky:
```
java.lang.NoSuchMethodError: 'void com.sun.tools.javac.util.Log.error(com.sun.tools.javac.util.JCDiagnostic$DiagnosticPosition, java.lang.String, java.lang.Object[])'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-793044516
BEAM-11921 might be get fixed today.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-775356385
R: @amaliujia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-775944964
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on a change in pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #13920:
URL: https://github.com/apache/beam/pull/13920#discussion_r584423788
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsublite/PubsubLiteTopicTable.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.pubsublite;
+
+import com.google.cloud.pubsublite.TopicPath;
+import com.google.cloud.pubsublite.proto.PubSubMessage;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
+import org.apache.beam.sdk.io.gcp.pubsublite.PublisherOptions;
+import org.apache.beam.sdk.io.gcp.pubsublite.PubsubLiteIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollection.IsBounded;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.Row;
+
+class PubsubLiteTopicTable extends SchemaBaseBeamTable {
+ private final TopicPath topic;
+ private final PTransform<PCollection<? extends Row>, PCollection<PubSubMessage>> transform;
+
+ PubsubLiteTopicTable(
+ Schema schema,
+ TopicPath topic,
+ PTransform<PCollection<? extends Row>, PCollection<PubSubMessage>> transform) {
+ super(schema);
+ this.topic = topic;
+ this.transform = transform;
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(PBegin begin) {
+ throw new UnsupportedOperationException(
Review comment:
Is there a test for this?
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##########
@@ -949,6 +950,40 @@ public int hashCode() {
getMetadata()
});
}
+
+ @Override
+ public String toString() {
Review comment:
I am not sure here: without this toString(), what is the default toString output for, say ROW type?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on a change in pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on a change in pull request #13920:
URL: https://github.com/apache/beam/pull/13920#discussion_r586697357
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##########
@@ -949,6 +950,40 @@ public int hashCode() {
getMetadata()
});
}
+
+ @Override
+ public String toString() {
Review comment:
Will add a test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-791030371
https://issues.apache.org/jira/browse/BEAM-11921
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on a change in pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on a change in pull request #13920:
URL: https://github.com/apache/beam/pull/13920#discussion_r584441882
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsublite/PubsubLiteTopicTable.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.pubsublite;
+
+import com.google.cloud.pubsublite.TopicPath;
+import com.google.cloud.pubsublite.proto.PubSubMessage;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
+import org.apache.beam.sdk.io.gcp.pubsublite.PublisherOptions;
+import org.apache.beam.sdk.io.gcp.pubsublite.PubsubLiteIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollection.IsBounded;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.Row;
+
+class PubsubLiteTopicTable extends SchemaBaseBeamTable {
+ private final TopicPath topic;
+ private final PTransform<PCollection<? extends Row>, PCollection<PubSubMessage>> transform;
+
+ PubsubLiteTopicTable(
+ Schema schema,
+ TopicPath topic,
+ PTransform<PCollection<? extends Row>, PCollection<PubSubMessage>> transform) {
+ super(schema);
+ this.topic = topic;
+ this.transform = transform;
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(PBegin begin) {
+ throw new UnsupportedOperationException(
Review comment:
To clarify, you want me to test that you can't read from write tables or write from read tables? Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on a change in pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on a change in pull request #13920:
URL: https://github.com/apache/beam/pull/13920#discussion_r584441882
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsublite/PubsubLiteTopicTable.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.pubsublite;
+
+import com.google.cloud.pubsublite.TopicPath;
+import com.google.cloud.pubsublite.proto.PubSubMessage;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
+import org.apache.beam.sdk.io.gcp.pubsublite.PublisherOptions;
+import org.apache.beam.sdk.io.gcp.pubsublite.PubsubLiteIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollection.IsBounded;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.Row;
+
+class PubsubLiteTopicTable extends SchemaBaseBeamTable {
+ private final TopicPath topic;
+ private final PTransform<PCollection<? extends Row>, PCollection<PubSubMessage>> transform;
+
+ PubsubLiteTopicTable(
+ Schema schema,
+ TopicPath topic,
+ PTransform<PCollection<? extends Row>, PCollection<PubSubMessage>> transform) {
+ super(schema);
+ this.topic = topic;
+ this.transform = transform;
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(PBegin begin) {
+ throw new UnsupportedOperationException(
Review comment:
To clarify, you want me to test that you can't read from write tables or write to read tables? Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia merged pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia merged pull request #13920:
URL: https://github.com/apache/beam/pull/13920
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-788379740
cc @robinyqiu
I think this PR is in good shape (just have one open comment left). So if Robin has no other comments I can merge this PR after the open comment is addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on a change in pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #13920:
URL: https://github.com/apache/beam/pull/13920#discussion_r586696748
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##########
@@ -949,6 +950,40 @@ public int hashCode() {
getMetadata()
});
}
+
+ @Override
+ public String toString() {
Review comment:
@dpcollins-google what do you think of this comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-791026659
This looks to be failing when using java 11 with gradle? When using java 8 with gradle, :sdks:java:core:javadoc fails with confusing errors. Going to try rebasing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-791005638
I took a quick look and this PR LGTM. We can merge after tests pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-790814690
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on a change in pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #13920:
URL: https://github.com/apache/beam/pull/13920#discussion_r585116358
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##########
@@ -949,6 +950,40 @@ public int hashCode() {
getMetadata()
});
}
+
+ @Override
+ public String toString() {
Review comment:
I see. In this case, do you think if it is worth also having simple tests for the new toString() code added (just for MAP,ROW,ARRAY)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on a change in pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #13920:
URL: https://github.com/apache/beam/pull/13920#discussion_r585115874
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsublite/PubsubLiteTopicTable.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.pubsublite;
+
+import com.google.cloud.pubsublite.TopicPath;
+import com.google.cloud.pubsublite.proto.PubSubMessage;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
+import org.apache.beam.sdk.io.gcp.pubsublite.PublisherOptions;
+import org.apache.beam.sdk.io.gcp.pubsublite.PubsubLiteIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollection.IsBounded;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.Row;
+
+class PubsubLiteTopicTable extends SchemaBaseBeamTable {
+ private final TopicPath topic;
+ private final PTransform<PCollection<? extends Row>, PCollection<PubSubMessage>> transform;
+
+ PubsubLiteTopicTable(
+ Schema schema,
+ TopicPath topic,
+ PTransform<PCollection<? extends Row>, PCollection<PubSubMessage>> transform) {
+ super(schema);
+ this.topic = topic;
+ this.transform = transform;
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(PBegin begin) {
+ throw new UnsupportedOperationException(
Review comment:
Yes. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on pull request #13920:
URL: https://github.com/apache/beam/pull/13920#issuecomment-775965131
Run SQL_Java11 PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on a change in pull request #13920: [BEAM-11659] Implement Pub/Sub Lite Table Provider
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on a change in pull request #13920:
URL: https://github.com/apache/beam/pull/13920#discussion_r586715650
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##########
@@ -949,6 +950,40 @@ public int hashCode() {
getMetadata()
});
}
+
+ @Override
+ public String toString() {
Review comment:
Added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org