You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "bzablocki (via GitHub)" <gi...@apache.org> on 2024/06/13 19:18:11 UTC

[PR] Solace Read connector: SolaceIO PTransform init [beam]

bzablocki opened a new pull request, #31594:
URL: https://github.com/apache/beam/pull/31594

   PR for the Solace Read connector (addresses https://github.com/apache/beam/issues/31440).
   Initializes SolaceIO Ptransform. 
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Solace Read connector: SolaceIO PTransform init [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #31594:
URL: https://github.com/apache/beam/pull/31594#discussion_r1638769590


##########
CHANGES.md:
##########
@@ -96,6 +96,8 @@
 
 ## I/Os
 
+* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
+* Support for reading from Solace message broker added (Java) ([#31440](https://github.com/apache/beam/issues/31440)).

Review Comment:
   We don't announce it before we start publish javadoc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Solace Read connector: SolaceIO PTransform init [beam]

Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #31594:
URL: https://github.com/apache/beam/pull/31594#discussion_r1638769892


##########
CHANGES.md:
##########
@@ -96,6 +96,8 @@
 
 ## I/Os
 
+* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
+* Support for reading from Solace message broker added (Java) ([#31440](https://github.com/apache/beam/issues/31440)).

Review Comment:
   ok!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Solace Read connector: SolaceIO PTransform init [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31594:
URL: https://github.com/apache/beam/pull/31594#issuecomment-2166716721

   Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Solace Read connector: SolaceIO PTransform init [beam]

Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #31594:
URL: https://github.com/apache/beam/pull/31594#issuecomment-2166647172

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Solace Read connector: SolaceIO PTransform init [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #31594:
URL: https://github.com/apache/beam/pull/31594


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Solace Read connector: SolaceIO PTransform init [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #31594:
URL: https://github.com/apache/beam/pull/31594#discussion_r1638878135


##########
sdks/java/io/solace/build.gradle:
##########
@@ -30,4 +30,9 @@ ext.summary = """IO to read and write to Solace destinations (queues and topics)
 
 dependencies {
     implementation enforcedPlatform(library.java.google_cloud_platform_libraries_bom)
-}
\ No newline at end of file
+    implementation library.java.vendored_guava_32_1_2_jre
+    implementation project(path: ":sdks:java:core", configuration: "shadow")
+    implementation library.java.joda_time
+    implementation library.java.solace
+    implementation library.java.vendored_guava_32_1_2_jre

Review Comment:
   ```suggestion
   ```
   



##########
sdks/java/io/solace/src/main/java/org/apache/beam/sdk/io/solace/SolaceIO.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.solace;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.auto.value.AutoValue;
+import com.solacesystems.jcsmp.BytesXMLMessage;
+import com.solacesystems.jcsmp.JCSMPFactory;
+import com.solacesystems.jcsmp.Queue;
+import com.solacesystems.jcsmp.Topic;
+import org.apache.beam.sdk.io.solace.broker.SempClientFactory;
+import org.apache.beam.sdk.io.solace.broker.SessionServiceFactory;
+import org.apache.beam.sdk.io.solace.data.Solace;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Instant;
+
+public class SolaceIO {
+
+  private static final boolean DEFAULT_DEDUPLICATE_RECORDS = false;
+
+  /** Get a {@link Topic} object from the topic name. */
+  static Topic topicFromName(String topicName) {
+    return JCSMPFactory.onlyInstance().createTopic(topicName);
+  }
+
+  /** Get a {@link Queue} object from the queue name. */
+  static Queue queueFromName(String queueName) {
+    return JCSMPFactory.onlyInstance().createQueue(queueName);
+  }
+
+  @AutoValue
+  public abstract static class Read<T> extends PTransform<PBegin, PCollection<T>> {
+
+    /** Set the queue name to read from. Use this or the `from(Topic)` method. */
+    public Read<T> from(Solace.Queue queue) {
+      return toBuilder().setQueue(queueFromName(queue.getName())).build();
+    }
+
+    /** Set the topic name to read from. Use this or the `from(Queue)` method. */
+    public Read<T> from(Solace.Topic topic) {
+      return toBuilder().setTopic(topicFromName(topic.getName())).build();
+    }
+
+    /**
+     * The timestamp function, used for estimating the watermark, mapping the record T to an {@link
+     * Instant}
+     *
+     * <p>Optional when using the no-arg {@link SolaceIO#read()} method. Defaults to {@link
+     * SolaceIO#SENDER_TIMESTAMP_FUNCTION}. When using the {@link SolaceIO#read(TypeDescriptor,
+     * SerializableFunction, SerializableFunction)} method, the function mapping from T to {@link
+     * Instant} has to be passed as an argument.
+     */
+    public Read<T> withTimestampFn(SerializableFunction<T, Instant> timestampFn) {
+      checkState(
+          timestampFn != null,
+          "SolaceIO.Read: timestamp function must be set or use the"
+              + " `Read.readSolaceRecords()` method");
+      return toBuilder().setTimestampFn(timestampFn).build();
+    }
+
+    /**
+     * Optional. Sets the maximum number of connections to the broker. The actual number of sessions
+     * is determined by this and the number set by the runner. If not set, the number of sessions is
+     * determined by the runner. The number of connections created follows this logic:
+     * `numberOfConnections = min(maxNumConnections, desiredNumberOfSplits)`, where the
+     * `desiredNumberOfSplits` is set by the runner.
+     */
+    public Read<T> withMaxNumConnections(Integer maxNumConnections) {
+      return toBuilder().setMaxNumConnections(maxNumConnections).build();
+    }
+
+    /**
+     * Optional, default: false. Set to deduplicate messages based on the {@link
+     * BytesXMLMessage#getApplicationMessageId()} of the incoming {@link BytesXMLMessage}. If the
+     * field is null, then the {@link BytesXMLMessage#getReplicationGroupMessageId()} will be used,
+     * which is always set by Solace.
+     */
+    public Read<T> withDeduplicateRecords(boolean deduplicateRecords) {
+      return toBuilder().setDeduplicateRecords(deduplicateRecords).build();
+    }
+
+    /**
+     * Set a factory that creates a {@link org.apache.beam.sdk.io.solace.broker.SempClientFactory}.
+     *
+     * <p>The factory `create()` method is invoked in each instance of an {@link
+     * org.apache.beam.sdk.io.solace.read.UnboundedSolaceReader}. Created {@link
+     * org.apache.beam.sdk.io.solace.broker.SempClient} has to communicate with broker management
+     * API. It must support operations such as:
+     *
+     * <ul>
+     *   <li>query for outstanding backlog bytes in a Queue,
+     *   <li>query for metadata such as access-type of a Queue,
+     *   <li>requesting creation of new Queues.
+     * </ul>
+     *
+     * <p>An existing implementation of the SempClientFactory includes {@link
+     * org.apache.beam.sdk.io.solace.broker.BasicAuthSempClientFactory} which implements connection
+     * to the SEMP with the Basic Authentication method.
+     *
+     * <p>To use it, specify the credentials with the builder methods.
+     *
+     * <p>The format of the host is `[Protocol://]Host[:Port]`
+     *
+     * <pre>{@code
+     * .withSempClientFactory(
+     *         BasicAuthSempClientFactory.builder()
+     *               .host("your-host-name-with-protocol") // e.g. "http://12.34.56.78:8080"
+     *               .username("username")
+     *               .password("password")
+     *               .vpnName("vpn-name")
+     *               .build())
+     * }</pre>
+     */
+    public Read<T> withSempClientFactory(SempClientFactory sempClientFactory) {
+      checkState(sempClientFactory != null, "SolaceIO.Read: sempClientFactory must not be null.");
+      return toBuilder().setSempClientFactory(sempClientFactory).build();
+    }
+
+    /**
+     * Set a factory that creates a {@link SessionService}.
+     *
+     * <p>The factory `create()` method is invoked in each instance of an {@link
+     * org.apache.beam.sdk.io.solace.read.UnboundedSolaceReader}. Created {@link SessionService} has
+     * to be able to:
+     *
+     * <ul>
+     *   <li>initialize a connection with the broker,
+     *   <li>check liveliness of the connection,
+     *   <li>close the connection,
+     *   <li>create a {@link org.apache.beam.sdk.io.solace.broker.MessageReceiver}.
+     * </ul>
+     *
+     * <p>An existing implementation of the SempClientFactory includes {@link
+     * org.apache.beam.sdk.io.solace.broker.BasicAuthJcsmpSessionService} which implements the Basic
+     * Authentication to Solace. *
+     *
+     * <p>To use it, specify the credentials with the builder methods. *
+     *
+     * <p>The host is the IPv4 or IPv6 or host name of the appliance. IPv5 addresses must be encoded

Review Comment:
   ```suggestion
        * <p>The host is the IPv4 or IPv6 or host name of the appliance. IPv6 addresses must be encoded
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org