You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hop.apache.org by ba...@apache.org on 2026/10/08 18:32:09 UTC
(hop) branch main updated: Issue #8708 : Add Beam MQTT, Debezium, Elasticsearch, Splunk and Snowflake IO (#8716)
This is an automated email from the ASF dual-hosted git repository.
bamaer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 93eca9610c Issue #8708 : Add Beam MQTT, Debezium, Elasticsearch, Splunk and Snowflake IO (#8716)
93eca9610c is described below
commit 93eca9610cb486a0aeb9e6e7002effebfd070516
Author: Matt Casters <ma...@gmail.com>
AuthorDate: Thu Oct 8 20:32:03 2026 +0200
Issue #8708 : Add Beam MQTT, Debezium, Elasticsearch, Splunk and Snowflake IO (#8716)
* Docs: add triage table for umbrella issue 8708
Track all 18 sub-issues of the Beam engine / runners / Beam IO umbrella
with a decision, effort rating and rationale each, plus the open
questions that block wave 4.
Refs #8708
* Beam tests: add single-transform pipeline test base
PipelineTestBase runs a pipeline but discards the engine and leaves the
output folder untouched, so tests extending it can neither read the output
nor assert on a row count. Every Beam transform handler in umbrella #8708
needs exactly those two assertions.
Adds SingleTransformPipelineTestBase with helpers to clear stale output,
read the output files, lines, text and per-shard bytes, and read a named
Beam counter off the engine, plus a self-test that exercises all of them.
Two behaviours the helpers have to respect, both found by the self-test:
- Beam shards file output, so a batch arrives as several
name-00000-of-000NN files. A helper assuming a single file silently
drops rows.
- Which counter a transform reports depends on its role: the file input
counts 'input' and 'written', the file output counts 'read' and
'output'. The metric name is a parameter rather than hard coded.
The engine publishes metrics from a background timer, so a read straight
after waitUntilFinished can come back empty; the counter helpers poll
instead of racing, and the negative tests use a non-polling read so they
fail fast.
Refs #8708
* Fixes #5064: return BigQuery RECORD fields as JSON strings
The type derivation did AvroType.valueOf("RECORD"), which threw and
failed the entire read for any table with a nested or repeated field.
Map RECORD and STRUCT to a String and render the value as JSON.
A field the table schema does not mention (common for a fromQuery
result) also threw "Unable to find field"; it now falls back to the Hop
type declared in the transform and logs a warning.
Nested values are rendered with GenericData.toString(), not Jackson:
Jackson serialises a GenericRecord as its internal bean structure
({"schema":...,"elementType":...}) rather than its contents, which is
what an earlier attempt produced. A test guards against that.
Fixes #5064
* Fixes #2416: add BigQuery query location and skip-validation options
BigQueryIO.TypedRead already offers withQueryLocation() and
withoutValidation(); neither was reachable from Hop. Both are exposed on
the BigQuery input transform, passed through to the read, and default to
off so a transform saved before this change behaves exactly as before.
Applies only when a query is set, since a plain table read has neither a
query location nor a query to validate.
Fixes #2416
* Fixes #2337: add compression support to the Beam file output transform
TextIO.Write.withCompression() already supports it; nothing in Hop
exposed it. Adds a compression option to the transform, the dialog and
the PTransform, defaulting to blank so existing transforms are unchanged.
Beam's AUTO is read-only, it fails a write with "AUTO is not supported for
writing", so AUTO is resolved against the file suffix here instead of
being passed through. With no recognisable suffix it degrades to
uncompressed rather than failing the job.
Covered end to end on the Direct runner: the tests run a real pipeline
and check the gzip magic on disk, then gunzip it and check all 100 rows
survive with all 10 fields.
Fixes #2337
* Fixes #2355: add a gcpTempLocation option to the Dataflow run configuration
BigQuery load jobs and the other GCP IOs need a Cloud Storage path in
GcpOptions.gcpTempLocation, which is a separate Beam option from the
general tempLocation. GcpOptions is already on the plugin classpath, so
this is a new run configuration option plus the wiring.
Only set when non-blank, so an existing run configuration keeps exactly
the behaviour it had.
The test builds PipelineOptions directly rather than going through
getPipelineOptions(), because that call asks the Dataflow client for GCP
application default credentials, which a unit test cannot have.
Fixes #2355
* Fixes #2675: fix Kafka Avro with a schema registry, in two parts
Packaging: the Confluent serializers were only on the compile classpath,
so the Hop fat jar never carried them. FatJarBuilder stages from
plugins/engines/beam/lib, which did not exist at all, and a Flink task
manager failed with ClassNotFoundException:
io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe. A
copy-dependencies execution now stages the io.confluent jars there at
prepare-package.
No new dependency declarations: the jars already resolve transitively at
the version the hop-libs BOM manages, and declaring them explicitly
breaks resolution because they are not on Maven Central and the project
only configures central.
Producer: an Avro message field with no schema made the transform call
AvroCoder.of(null), which throws a bare NullPointerException from inside
Beam's determinism checker before the pipeline is submitted. The check
now names the field and says what to do about it.
The staging test is verified to fail without the copy-dependencies
execution. The generated jars are covered by the root *.jar gitignore.
Fixes #2675
* Docs: tick off the wave 2 sub-issues in the 8708 triage table
Records the commit for each of the five issues implemented so far.
The 12 remaining rows stay Planned and #2613 stays Blocked on scope.
Refs #8708
* Fixes #2379: add a Beam handler for the Add Sequence transform
Without a handler the transform fell through to BeamGenericTransformHandler,
which runs the Hop transform inside a ParDo per worker, so every worker
generated its own sequence starting at the same number.
The handler funnels all rows onto one key and keeps the counter in Beam
state, which is what makes it a single pipeline-wide sequence. A plain
field would restart per worker, and state is per key, so the keying and
the state belong together.
Three things worth remembering for any future stateful Beam DoFn here:
- @StateId is a field-only annotation and belongs on a final StateSpec
field. The State itself arrives as a @ProcessElement parameter, not as
a field; a field for it is never injected and stays null.
- A stateful ParDo needs a KvCoder input, so the rows are re-keyed after
Flatten, which leaves a PCollection<Iterable<HopRow>>.
- The row meta the converter passes already carries the sequence field,
because AddSequenceMeta.getFields() added it. Adding another one grows
the row past what the output transform reads and silently drops the
value, which is what an earlier version did.
The database sequence mode cannot be expressed on Beam and now fails with
an explanation rather than emitting duplicates. A trade-off worth
stating: the transform runs at parallelism one, which is what a globally
increasing sequence costs.
The end-to-end test is verified to fail without the handler registered.
Fixes #2379
* Fixes #2275: let the Beam window transform window on a key
Beam can only group keyed elements, so windowing per key means keying the
rows, windowing the KV collection, grouping, and dropping the key again.
The key is derived from a field that is already on the row, so no extra
column appears in the output.
An empty key field keeps the previous global windowing, which is what a
transform saved before this change gets.
Two details that cost time and are easy to get wrong again:
- WithKeys.of is overloaded: one takes a constant key, the other a
SerializableFunction. HopKeyFn is the latter, and the call site names the
type explicitly so the function overload is selected.
- A Window<HopRow> cannot be applied to PCollection<KV<String, HopRow>>.
Window is a PTransform, not a WindowFn, so the window function is taken
from window.getWindowFn() and re-wrapped for the keyed element type.
A missing key field now fails the pipeline naming the field, rather than
windowing everything together and looking like it worked. Note that a
valid key field and no key field produce the same row count for a GLOBAL
window, so the end-to-end test proves the keyed branch is reached via that
failure case rather than by row count alone.
The Beam window dialog gets a Key field text box; TextVar has no
setToolTip, the SWT method is setToolTipText.
Fixes #2275
* Fixes #2040: add a Beam partition transform
Wraps Beam's Partition transform, giving Beam pipelines the partitioning
option the local engine has. This also gives the existing SinglePartitionFn
a caller: it was on the classpath with nothing referencing it.
Two modes: Single sends everything to one partition, Key hashes a field so
rows sharing a value land together.
The part worth reading twice: Beam returns a PCollectionList, one collection
per partition, and the partition for a given row is only decided while the
pipeline runs, so the next transform cannot branch on it. Reading partition
0 therefore drops every row that went to the others - 88 of 100 in the test.
The partitions are flattened back into one stream instead. The end-to-end
test asserts the row count and was verified to fail (12 of 100) against the
single-partition version, so this regression cannot come back quietly.
KeyedPartitionFn carries the value meta as JSON rather than as an IValueMeta,
because Beam serialises the partition function to the workers and an
IValueMeta does not survive that. The meta is rebuilt on first use. A null
key and a zero partition count are both guarded rather than left to throw
inside the runner, and the modulo uses floorMod because Math.abs returns a
negative number for Integer.MIN_VALUE, which would give an invalid index.
Fixes #2040
* Answers #2358: the Avro row transport already exists, only half the
transforms need work
The issue asks whether adding a handler is worth the effort. Investigated it
rather than guessing, and the answer is that the effort splits: two of the
four Avro transforms need nothing, and two need real work. That is a larger
change than the original triage assumed, which had called it handler code
only.
The transport is already done. HopRowCoder handles GenericRecord explicitly,
writing the schema as JSON next to the binary payload - which is exactly
what Avro input needs, since the schema is only known once the file has been
read. Someone anticipated this. HopRowCoderAvroTest now covers the round trip
including a nested record, and pins a detail that is easy to trip over: the
coder is type-tagged, not Java serialization, so a row must hold Long rather
than Integer or it is rejected before the Avro field is reached.
AvroEncode and AvroDecode are pure per-row conversions with no cross-worker
state, which is what the generic handler provides. They work as they are.
AvroFileInput and AvroOutput cannot work there. The input is not a source: it
reads the filename to open from a field in an incoming row, so on Beam there
is no input collection, the generic handler injects one dummy HopRow, and the
transform fails asking for a filename field. The output opens a single writer
and holds it for every row, so every worker would write to the same file.
Both need a handler built on AvroIO, which is already on the classpath. One
complication for whoever picks this up: hop-tech-avro is not a dependency of
the Beam module, so the meta classes cannot be referenced at compile time
without adding it.
Recorded in the triage table. No handler written yet.
Answers #2358
* Answers #2357: add the ParquetIO dependency and record what a handler needs
The issue asks whether a handler is worth the effort, so this investigates
rather than implements.
The missing dependency was real: ParquetIO ships in
beam-sdks-java-io-parquet, not beam-sdks-java-core, and the build fails to
compile ParquetIOViabilityTest without it.
The triage expected the only remaining work to be handler code. That is not
quite right, though the mismatch is in our favour. ParquetIO is Avro-typed:
read(schema) and sink(schema) both take an org.apache.avro.Schema rather
than a Parquet MessageType, and yield GenericRecord. That looks like an
obstacle but is actually a clean seam, because ParquetOutput.buildSchema()
already builds SchemaBuilder.record("ApacheHopParquetSchema") and derives
the Parquet type from it. Both sides already agree on the schema, so a
handler has no translation layer to invent.
Verified along the way:
- AvroCoder is in org.apache.beam.sdk.extensions.avro.coders, not
org.apache.beam.sdk.coders.
- GenericRecord.equals is reference-based, so AvroCoder reports
consistentWithEquals() false even for value-equal records. Irrelevant here
and not a handler decision, since ParquetIO sets the coder itself.
- Packaging needs no change. The Parquet jars do not appear in the beam
plugin's lib/ because that directory is scoped to io.confluent by the
#2675 staging execution, but Beam SDKs are unpacked into lib/core, which
is already on the classpath; lib-beam is legacy and no longer produced.
The transforms still need a handler: ParquetFileInput reads its filename
from an incoming row field and ParquetFileOutput holds a single writer, so
neither works through the generic handler. And hop-tech-parquet is not a
Beam module dependency, so the meta classes cannot be referenced at compile
time - the same constraint as #2358.
151 tests pass. Recorded in the triage table; no handler written yet.
Answers #2357
* Issue #8708 : Add Beam MQTT, Debezium, Elasticsearch, Splunk and Snowflake IO
Sources reject an incoming hop before they expand. Snowflake copies through a
GCS stage and does not implement Snowpipe or a streaming read. The Direct
runner run configuration can set the GCP temp location.
* Fix review findings in Beam IO and partition transforms (#8708)
- Bind stage-confluent-serializers execution to generate-resources and clean lib/ directory on mvn clean
- Exclude redundant bcprov-jdk15on dependency in Snowflake IO
- Use RowDataUtil.resizeArray and indexed field assignment in AddSequenceFn
- Improve KeyedPartitionFn hashing for byte arrays and BigDecimals with scale normalization and IValueMeta fallback
- Refactor BeamPartitionDialog to use GuiCompositeWidgets.addScrolledComposite and @GuiWidgetElement annotations
- Remove duplicate license header in BeamDirectPipelineRunConfiguration
- Add unit tests for KeyedPartitionFn hashing and BeamPartitionDialog UI interactions
* Fix 32 checkstyle AvoidStarImport violations in beam engine tests (#8708)
* Fix action and toolbar item filter evaluation and filter ID generation (#8708)
- In GuiRegistry, use guiPluginClassName directly instead of calling getClass().getName() on it, preventing filter IDs from erroneously starting with 'java.lang.String'.
- In BaseGuiContextHandler, resolve the target filter object from the context handler instance (via getter, field, or self-reference) before falling back to getInstance() and reflection construction. This fixes NullPointerException when evaluating action filters in scenes/tests or when getInstance() returns null.
- Add unit tests for filter ID registration in GuiRegistryTest and resolution logic in BaseGuiContextHandlerTest.
* Fix missing test imports and assertions in hop-engines-beam (#8708)
* Issue #8708 : Fix Beam Add Sequence, keyed windows, compression suffixes, and packaged jars.
* Issue #8708 : Run Add Sequence with a configuration transform through the generic Beam handler
The Beam handler ignored the configuration transform from #5193 and numbered
rows from the typed start, increment, and maximum. The generic handler passes
the info stream to the Hop transform.
---------
Co-authored-by: Bart Maertens <ba...@know.bi>
---
NOTICE | 21 +
assemblies/static/src/main/resources/LICENSE | 58 ++
assemblies/static/src/main/resources/hop | 4 +
.../apache/hop/core/gui/plugin/GuiRegistry.java | 4 +-
.../apache/hop/core/row/value/ValueMetaBase.java | 24 +-
.../hop/core/gui/plugin/GuiRegistryTest.java | 55 ++
.../hop/core/row/value/ValueMetaBaseTest.java | 43 +
.../transforms/icons/beam-debezium-input.svg | 23 +
.../transforms/icons/beam-elasticsearch-input.svg | 25 +
.../transforms/icons/beam-elasticsearch-output.svg | 25 +
.../images/transforms/icons/beam-mqtt-input.svg | 26 +
.../images/transforms/icons/beam-mqtt-output.svg | 26 +
.../images/transforms/icons/beam-partition.svg | 869 +++++++++++++++++++++
.../transforms/icons/beam-snowflake-input.svg | 24 +
.../transforms/icons/beam-snowflake-output.svg | 24 +
.../images/transforms/icons/beam-splunk-output.svg | 24 +
docs/hop-user-manual/modules/ROOT/nav.adoc | 10 +
.../pipeline/beam/developing-beam-transforms.adoc | 68 ++
.../beam-dataflow-pipeline-engine.adoc | 3 +
.../beam-direct-pipeline-engine.adoc | 1 +
.../beam-flink-pipeline-engine.adoc | 6 +
.../pages/pipeline/transforms/addsequence.adoc | 13 +
.../pipeline/transforms/beambigqueryinput.adoc | 6 +
.../pipeline/transforms/beamdebeziuminput.adoc | 136 ++++
.../transforms/beamelasticsearchinput.adoc | 55 ++
.../transforms/beamelasticsearchoutput.adoc | 56 ++
.../pages/pipeline/transforms/beamfileoutput.adoc | 5 +
.../pages/pipeline/transforms/beammqttinput.adoc | 56 ++
.../pages/pipeline/transforms/beammqttoutput.adoc | 55 ++
.../pages/pipeline/transforms/beampartition.adoc | 85 ++
.../pipeline/transforms/beamsnowflakeinput.adoc | 70 ++
.../pipeline/transforms/beamsnowflakeoutput.adoc | 68 ++
.../pipeline/transforms/beamsplunkoutput.adoc | 63 ++
.../ROOT/pages/pipeline/transforms/beamwindow.adoc | 3 +
.../partials/beam-elasticsearch-compatibility.adoc | 31 +
plugins/engines/beam/pom.xml | 118 +++
plugins/engines/beam/src/assembly/assembly.xml | 18 +
.../org/apache/hop/beam/core/BeamDefaults.java | 6 +
.../apache/hop/beam/core/coder/HopRowCoder.java | 12 +
.../hop/beam/core/coder/SplunkWriteErrorCoder.java | 71 ++
.../org/apache/hop/beam/core/fn/AddSequenceFn.java | 144 ++++
.../hop/beam/core/fn/BQSchemaAndRecordToHopFn.java | 27 +-
.../hop/beam/core/fn/ElasticsearchJsonFn.java | 85 ++
.../java/org/apache/hop/beam/core/fn/HopKeyFn.java | 68 ++
.../org/apache/hop/beam/core/fn/HopToMqttFn.java | 91 +++
.../apache/hop/beam/core/fn/HopToSnowflakeRow.java | 88 +++
.../org/apache/hop/beam/core/fn/HopToSplunkFn.java | 89 +++
.../org/apache/hop/beam/core/fn/MqttToHopFn.java | 61 ++
.../apache/hop/beam/core/fn/SnowflakeCsvToHop.java | 81 ++
.../apache/hop/beam/core/fn/SnowflakeValues.java | 185 +++++
.../hop/beam/core/fn/SplunkWriteFailureFn.java | 50 ++
.../hop/beam/core/partition/KeyedPartitionFn.java | 99 +++
.../beam/core/transform/BeamBQInputTransform.java | 15 +
.../core/transform/BeamDebeziumInputTransform.java | 82 ++
.../transform/BeamElasticsearchInputTransform.java | 48 ++
.../BeamElasticsearchOutputTransform.java | 73 ++
.../beam/core/transform/BeamMqttConnection.java | 116 +++
.../core/transform/BeamMqttInputTransform.java | 53 ++
.../core/transform/BeamMqttOutputTransform.java | 54 ++
.../beam/core/transform/BeamOutputTransform.java | 79 +-
.../transform/BeamSnowflakeInputTransform.java | 40 +
.../transform/BeamSnowflakeOutputTransform.java | 41 +
.../core/transform/BeamSplunkOutputTransform.java | 75 ++
.../hop/beam/engines/BeamPipelineEngine.java | 12 +
.../BeamDataFlowPipelineRunConfiguration.java | 33 +
.../direct/BeamDirectPipelineRunConfiguration.java | 49 +-
.../hop/beam/pipeline/BeamSourceTopology.java | 52 ++
.../HopPipelineMetaToBeamPipelineConverter.java | 6 +-
.../handler/BeamAddSequenceTransformHandler.java | 193 +++++
.../hop/beam/transforms/bq/BeamBQInputDialog.java | 41 +
.../hop/beam/transforms/bq/BeamBQInputMeta.java | 48 ++
.../transforms/debezium/BeamDebeziumInput.java | 41 +
.../transforms/debezium/BeamDebeziumInputData.java | 22 +
.../debezium/BeamDebeziumInputDialog.java | 91 +++
.../transforms/debezium/BeamDebeziumInputMeta.java | 459 +++++++++++
.../elasticsearch/BeamElasticsearchConfig.java | 160 ++++
.../elasticsearch/BeamElasticsearchInput.java | 41 +
.../elasticsearch/BeamElasticsearchInputData.java | 21 +
.../BeamElasticsearchInputDialog.java | 93 +++
.../elasticsearch/BeamElasticsearchInputMeta.java | 282 +++++++
.../elasticsearch/BeamElasticsearchOutput.java | 41 +
.../elasticsearch/BeamElasticsearchOutputData.java | 21 +
.../BeamElasticsearchOutputDialog.java | 93 +++
.../elasticsearch/BeamElasticsearchOutputMeta.java | 256 ++++++
.../hop/beam/transforms/io/BeamOutputDialog.java | 30 +
.../hop/beam/transforms/io/BeamOutputMeta.java | 26 +-
.../hop/beam/transforms/kafka/BeamProduceMeta.java | 36 +-
.../hop/beam/transforms/mqtt/BeamMqttInput.java | 41 +
.../beam/transforms/mqtt/BeamMqttInputData.java | 22 +
.../beam/transforms/mqtt/BeamMqttInputDialog.java | 88 +++
.../beam/transforms/mqtt/BeamMqttInputMeta.java | 315 ++++++++
.../hop/beam/transforms/mqtt/BeamMqttOutput.java | 41 +
.../beam/transforms/mqtt/BeamMqttOutputData.java | 22 +
.../beam/transforms/mqtt/BeamMqttOutputDialog.java | 88 +++
.../beam/transforms/mqtt/BeamMqttOutputMeta.java | 281 +++++++
.../beam/transforms/partition/BeamPartition.java | 49 ++
.../transforms/partition/BeamPartitionData.java | 23 +
.../transforms/partition/BeamPartitionDialog.java | 99 +++
.../transforms/partition/BeamPartitionMeta.java | 219 ++++++
.../transforms/snowflake/BeamSnowflakeInput.java | 42 +
.../snowflake/BeamSnowflakeInputData.java | 22 +
.../snowflake/BeamSnowflakeInputDialog.java | 167 ++++
.../snowflake/BeamSnowflakeInputMeta.java | 442 +++++++++++
.../transforms/snowflake/BeamSnowflakeOutput.java | 42 +
.../snowflake/BeamSnowflakeOutputData.java | 22 +
.../snowflake/BeamSnowflakeOutputDialog.java | 89 +++
.../snowflake/BeamSnowflakeOutputMeta.java | 456 +++++++++++
.../beam/transforms/snowflake/SnowflakeField.java | 36 +
.../beam/transforms/snowflake/SnowflakeSpec.java | 197 +++++
.../beam/transforms/splunk/BeamSplunkOutput.java | 41 +
.../transforms/splunk/BeamSplunkOutputData.java | 22 +
.../transforms/splunk/BeamSplunkOutputDialog.java | 88 +++
.../transforms/splunk/BeamSplunkOutputMeta.java | 356 +++++++++
.../beam/transforms/window/BeamWindowDialog.java | 24 +
.../hop/beam/transforms/window/BeamWindowMeta.java | 124 ++-
.../java/org/apache/hop/beam/util/BeamConst.java | 1 +
.../src/main/resources/beam-debezium-input.svg | 23 +
.../main/resources/beam-elasticsearch-input.svg | 25 +
.../main/resources/beam-elasticsearch-output.svg | 25 +
.../beam/src/main/resources/beam-mqtt-input.svg | 26 +
.../beam/src/main/resources/beam-mqtt-output.svg | 26 +
.../beam/src/main/resources/beam-partition.svg | 869 +++++++++++++++++++++
.../src/main/resources/beam-snowflake-input.svg | 24 +
.../src/main/resources/beam-snowflake-output.svg | 24 +
.../beam/src/main/resources/beam-splunk-output.svg | 24 +
.../beam/src/main/resources/dependencies.xml | 1 +
.../dataflow/messages/messages_en_US.properties | 2 +
.../direct/messages/messages_en_US.properties | 18 +
.../bq/messages/messages_en_US.properties | 4 +
.../debezium/messages/messages_en_US.properties | 49 ++
.../messages/messages_en_US.properties | 43 +
.../io/messages/messages_en_US.properties | 2 +
.../mqtt/messages/messages_en_US.properties | 45 ++
.../partition/messages/messages_en_US.properties | 30 +
.../snowflake/messages/messages_en_US.properties | 67 ++
.../splunk/messages/messages_en_US.properties | 44 ++
.../window/messages/messages_en_US.properties | 2 +
.../hop/beam/core/coder/HopRowCoderAvroTest.java | 145 ++++
.../hop/beam/core/coder/HopRowCoderTest.java | 11 +
.../beam/core/fn/BQSchemaAndRecordToHopFnTest.java | 239 ++++++
.../hop/beam/core/fn/BeamMqttConversionTest.java | 216 +++++
.../hop/beam/core/fn/SnowflakeValuesTest.java | 156 ++++
.../beam/core/partition/BeamPartitionFnsTest.java | 207 +++++
.../BeamDataFlowPipelineRunConfigurationTest.java | 84 ++
.../BeamDirectPipelineRunConfigurationTest.java | 78 ++
.../beam/io/parquet/ParquetIOViabilityTest.java | 150 ++++
.../hop/beam/pipeline/BeamSourceConverterTest.java | 292 +++++++
.../fatjar/FatJarKafkaAvroStagingTest.java | 170 ++++
.../BeamAddSequenceTransformHandlerTest.java | 235 ++++++
.../beam/transform/AddSequencePipelineTest.java | 198 +++++
.../BeamOutputCompressionPipelineTest.java | 226 ++++++
.../transform/SingleTransformPipelineTestBase.java | 265 +++++++
.../SingleTransformPipelineTestBaseTest.java | 250 ++++++
.../beam/transforms/bq/BeamBQInputMetaTest.java | 15 +
.../debezium/BeamDebeziumInputDialogTest.java | 161 ++++
.../debezium/BeamDebeziumInputMetaTest.java | 206 +++++
.../debezium/BeamDebeziumInputTransformTest.java | 377 +++++++++
.../elasticsearch/BeamElasticsearchConfigTest.java | 108 +++
.../elasticsearch/BeamElasticsearchDialogTest.java | 154 ++++
.../elasticsearch/BeamElasticsearchIOTest.java | 447 +++++++++++
.../BeamElasticsearchInputMetaTest.java | 121 +++
.../elasticsearch/BeamElasticsearchJsonFnTest.java | 155 ++++
.../BeamElasticsearchOptionsTest.java | 100 +++
.../BeamElasticsearchOutputMetaTest.java | 84 ++
.../elasticsearch/BeamElasticsearchPluginTest.java | 73 ++
.../BeamElasticsearchSerializationTest.java | 78 ++
.../beam/transforms/kafka/BeamProduceMetaTest.java | 136 ++++
.../beam/transforms/mqtt/BeamMqttBrokerTest.java | 441 +++++++++++
.../beam/transforms/mqtt/BeamMqttDialogTest.java | 208 +++++
.../beam/transforms/mqtt/BeamMqttGraphTest.java | 394 ++++++++++
.../hop/beam/transforms/mqtt/BeamMqttMetaTest.java | 167 ++++
.../partition/BeamPartitionDialogTest.java | 127 +++
.../partition/BeamPartitionMetaTest.java | 94 +++
.../partition/BeamPartitionPipelineTest.java | 194 +++++
.../snowflake/BeamSnowflakeInputDialogTest.java | 119 +++
.../snowflake/BeamSnowflakeInputMetaTest.java | 298 +++++++
.../snowflake/BeamSnowflakeOutputDialogTest.java | 113 +++
.../snowflake/BeamSnowflakeOutputMetaTest.java | 296 +++++++
.../transforms/splunk/BeamSplunkDialogTest.java | 124 +++
.../beam/transforms/splunk/BeamSplunkIOTest.java | 179 +++++
.../splunk/BeamSplunkOutputMetaTest.java | 196 +++++
.../window/BeamWindowKeyedPipelineTest.java | 187 +++++
.../window/BeamWindowKeyedTriggerTest.java | 122 +++
.../beam/transforms/window/BeamWindowMetaTest.java | 105 +++
.../src/test/resources/beam-bq-input-transform.xml | 2 +
...sform.xml => beam-debezium-input-transform.xml} | 39 +-
.../beam-elasticsearch-input-transform.xml | 23 +
.../beam-elasticsearch-output-transform.xml | 23 +
.../test/resources/beam-mqtt-input-transform.xml | 31 +
.../test/resources/beam-mqtt-output-transform.xml | 30 +
.../resources/beam-snowflake-input-transform.xml | 43 +
.../resources/beam-snowflake-output-transform.xml | 38 +
.../resources/beam-splunk-output-transform.xml | 33 +
.../beam-window-global-keyed-transform.xml | 41 +
.../test/resources/beam-window-keyed-transform.xml | 41 +
.../ui/hopgui/context/BaseGuiContextHandler.java | 98 ++-
.../hopgui/context/BaseGuiContextHandlerTest.java | 234 ++++++
197 files changed, 20086 insertions(+), 104 deletions(-)
diff --git a/NOTICE b/NOTICE
index ee0041e676..bc933354b8 100644
--- a/NOTICE
+++ b/NOTICE
@@ -22,3 +22,24 @@ See: https://calcite.apache.org/
This product bundles Unity Catalog client libraries (transitive dependency of
Delta Lake) under the Apache License, Version 2.0.
See: https://github.com/unitycatalog/unitycatalog
+
+Apache Beam
+Copyright 2016-2018 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+Based on source code originally developed by
+Google (http://www.google.com/).
+
+This product includes software developed at
+Google (http://www.google.com/).
+
+Elasticsearch
+Copyright 2009-2021 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).
+
+This product includes software developed by
+Joda.org (http://www.joda.org/).
diff --git a/assemblies/static/src/main/resources/LICENSE b/assemblies/static/src/main/resources/LICENSE
index 0cc099aa9d..d1b9f0f252 100644
--- a/assemblies/static/src/main/resources/LICENSE
+++ b/assemblies/static/src/main/resources/LICENSE
@@ -571,6 +571,12 @@ This product bundles beam-sdks-java-io-amazon-web-services2 2.74.0, which is ava
This product bundles beam-sdks-java-io-azure 2.74.0, which is available under a
"Apache 2.0" license. For details, see LICENSE.
+This product bundles beam-sdks-java-io-debezium 2.74.0, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
+This product bundles beam-sdks-java-io-elasticsearch 2.74.0, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
This product bundles beam-sdks-java-io-google-cloud-platform 2.74.0, which is available under a
"Apache 2.0" license. For details, see LICENSE.
@@ -586,6 +592,15 @@ This product bundles beam-sdks-java-io-hcatalog 2.74.0, which is available under
This product bundles beam-sdks-java-io-kafka 2.74.0, which is available under a
"Apache 2.0" license. For details, see LICENSE.
+This product bundles beam-sdks-java-io-mqtt 2.74.0, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
+This product bundles beam-sdks-java-io-snowflake 2.74.0, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
+This product bundles beam-sdks-java-io-splunk 2.74.0, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
This product bundles beam-sdks-java-io-synthetic 2.74.0, which is available under a
"Apache 2.0" license. For details, see LICENSE.
@@ -760,6 +775,12 @@ This product bundles concurrent-trees 2.4.0, which is available under a
This product bundles config 1.4.1, which is available under a
"Apache 2.0" license. For details, see LICENSE.
+This product bundles connect-api 3.9.0, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
+This product bundles connect-transforms 3.9.0, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
This product bundles conscrypt-openjdk-uber 2.5.2, which is available under a
"Apache 2.0" license. For details, see LICENSE.
@@ -808,6 +829,15 @@ This product bundles datasketches-memory 3.0.2, which is available under a
This product bundles dd-plist 1.29, which is available under a
"The MIT" license. For details, see licenses/LICENSE-MIT.
+This product bundles debezium-api 3.1.3.Final, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
+This product bundles debezium-connector-postgres 3.1.3.Final, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
+This product bundles debezium-core 3.1.3.Final, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
This product bundles dec 0.1.2, which is available under a
"The MIT" license. For details, see licenses/LICENSE-MIT.
@@ -955,6 +985,9 @@ This product bundles ejml-simple 0.39, which is available under a
This product bundles ejml-zdense 0.39, which is available under a
"Apache 2.0" license. For details, see LICENSE.
+This product bundles elasticsearch-rest-client 7.17.22, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
This product bundles encoder 1.4.0, which is available under a
"3-clause BSD" license. For details, see licenses/LICENSE-BSD3.
@@ -1333,6 +1366,15 @@ This product bundles hadoop-yarn-common 3.4.2, which is available under a
This product bundles hamcrest 2.1, which is available under a
"3-clause BSD" license. For details, see licenses/LICENSE-BSD3.
+This product bundles hawtbuf 1.11, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
+This product bundles hawtdispatch 1.22, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
+This product bundles hawtdispatch-transport 1.22, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
This product bundles HdrHistogram 2.1.12, which is available under a
"Creative Commons" license. For details, see licenses/LICENSE-CC0.
@@ -1378,6 +1420,9 @@ This product bundles hsqldb 2.7.4, which is available under a
This product bundles http-client-spi 2.20.162, which is available under a
"Apache 2.0" license. For details, see LICENSE.
+This product bundles httpasyncclient 4.1.4, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
This product bundles httpclient 4.5.14, which is available under a
"Apache 2.0" license. For details, see LICENSE.
@@ -1387,6 +1432,9 @@ This product bundles httpclient5 5.6.4, which is available under a
This product bundles httpcore 4.4.16, which is available under a
"Apache 2.0" license. For details, see LICENSE.
+This product bundles httpcore-nio 4.4.12, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
This product bundles httpcore5 5.4.3, which is available under a
"Apache 2.0" license. For details, see LICENSE.
@@ -1600,6 +1648,10 @@ This product bundles javax.servlet-api 3.1.0, which is available under a
This product bundles javax.servlet-api 4.0.1, which is available under a
"Common Development and Distribution 1.1" license. For details, see licenses/LICENSE-CDDL11.
+This product bundles javax.ws.rs-api 2.1.1, which is available under an
+"Eclipse Public 2.0" license. For details, see licenses/LICENSE-EPL20.
+Source code is available at https://github.com/eclipse-ee4j/jaxrs-api.
+
This product bundles jaxb-api 2.2.11, which is available under a
"Common Development and Distribution 1.1" license. For details, see licenses/LICENSE-CDDL11.
@@ -2206,6 +2258,9 @@ This product bundles moshi 1.15.2, which is available under a
This product bundles moshi-kotlin 1.13.0, which is available under a
"Apache 2.0" license. For details, see LICENSE.
+This product bundles mqtt-client 1.15, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
This product bundles msal4j 1.16.1, which is available under a
"The MIT" license. For details, see licenses/LICENSE-MIT.
@@ -2935,6 +2990,9 @@ This product bundles snmp4j 3.9.7, which is available under a
This product bundles snowball-stemmer 1.3.0.581.1, which is available under a
"2-clause BSD" license. For details, see licenses/LICENSE-BSD2.
+This product bundles snowflake-jdbc 4.3.0, which is available under a
+"Apache 2.0" license. For details, see LICENSE.
+
This product bundles sns 2.20.162, which is available under a
"Apache 2.0" license. For details, see LICENSE.
diff --git a/assemblies/static/src/main/resources/hop b/assemblies/static/src/main/resources/hop
index fb7b48035f..5dc5f16522 100755
--- a/assemblies/static/src/main/resources/hop
+++ b/assemblies/static/src/main/resources/hop
@@ -121,6 +121,10 @@ fi
if [ -d "plugins/engines/beam/lib-beam" ]; then
CLASSPATH="${CLASSPATH}:plugins/engines/beam/lib-beam/*"
fi
+# snowflake-jdbc is here, not in lib/core: the jar embeds its own zstd classes.
+if [ -d "plugins/engines/beam/lib" ]; then
+ CLASSPATH="${CLASSPATH}:plugins/engines/beam/lib/*"
+fi
if [ -d "plugins/engines/beam" ]; then
# Hop Beam plugin jars at the plugin root (exclude subdirs via non-recursive glob is fine for /*.jar intent;
# shell * includes jars and may pick non-jars; match historical lib/* style)
diff --git a/core/src/main/java/org/apache/hop/core/gui/plugin/GuiRegistry.java b/core/src/main/java/org/apache/hop/core/gui/plugin/GuiRegistry.java
index 297aa9620e..d6bc3bfbef 100644
--- a/core/src/main/java/org/apache/hop/core/gui/plugin/GuiRegistry.java
+++ b/core/src/main/java/org/apache/hop/core/gui/plugin/GuiRegistry.java
@@ -546,7 +546,7 @@ public class GuiRegistry {
itemFilter.setGuiPluginClassName(guiPluginClassName);
itemFilter.setGuiPluginMethodName(method.getName());
itemFilter.setClassLoader(classLoader);
- itemFilter.setId(guiPluginClassName.getClass().getName() + "." + method.getName());
+ itemFilter.setId(guiPluginClassName + "." + method.getName());
List<GuiToolbarItemFilter> itemFilters =
toolbarItemFiltersMap.computeIfAbsent(filter.parentId(), k -> new ArrayList<>());
@@ -790,7 +790,7 @@ public class GuiRegistry {
actionFilter.setGuiPluginClassName(guiPluginClassName);
actionFilter.setGuiPluginMethodName(method.getName());
actionFilter.setClassLoader(classLoader);
- actionFilter.setId(guiPluginClassName.getClass().getName() + "." + method.getName());
+ actionFilter.setId(guiPluginClassName + "." + method.getName());
List<GuiActionFilter> actionFilters =
contextActionFiltersMap.computeIfAbsent(af.parentId(), k -> new ArrayList<>());
diff --git a/core/src/main/java/org/apache/hop/core/row/value/ValueMetaBase.java b/core/src/main/java/org/apache/hop/core/row/value/ValueMetaBase.java
index 5226157413..4f88008dc0 100644
--- a/core/src/main/java/org/apache/hop/core/row/value/ValueMetaBase.java
+++ b/core/src/main/java/org/apache/hop/core/row/value/ValueMetaBase.java
@@ -388,6 +388,7 @@ public class ValueMetaBase implements IValueMeta {
}
}
}
+ valueMeta.setIndex(index);
break;
case STORAGE_TYPE_BINARY_STRING:
@@ -443,6 +444,12 @@ public class ValueMetaBase implements IValueMeta {
}
valueMeta.setLenientStringToNumber(
"Y".equalsIgnoreCase(XmlHandler.getTagValue(node, "lenient_string_to_number")));
+ // Absent on metadata written before string encoding was stored. Leaving it unset keeps the
+ // previous "no encoding" behavior.
+ String stringEncoding = XmlHandler.getTagValue(node, "string_encoding");
+ if (!Utils.isEmpty(stringEncoding)) {
+ valueMeta.setStringEncoding(stringEncoding);
+ }
}
/**
@@ -4024,6 +4031,9 @@ public class ValueMetaBase implements IValueMeta {
"date_format_timezone",
dateFormatTimeZone != null ? dateFormatTimeZone.getID() : null));
xml.append(XmlHandler.addTagValue("lenient_string_to_number", lenientStringToNumber));
+ if (!Utils.isEmpty(stringEncoding)) {
+ xml.append(XmlHandler.addTagValue("string_encoding", stringEncoding));
+ }
xml.append(XmlHandler.closeTag(XML_META_TAG));
@@ -4800,9 +4810,9 @@ public class ValueMetaBase implements IValueMeta {
// If there are no encodings set, then we're certain we don't have to
// convert as well.
//
- if (getStringEncoding() != null
- && getStringEncoding().equals(storageMetadata.getStringEncoding())
- || getStringEncoding() == null && storageMetadata.getStringEncoding() == null) {
+ if ((getStringEncoding() != null
+ && getStringEncoding().equals(storageMetadata.getStringEncoding()))
+ || (getStringEncoding() == null && storageMetadata.getStringEncoding() == null)) {
// However, perhaps the conversion mask changed since we read the
// binary string?
@@ -4856,7 +4866,15 @@ public class ValueMetaBase implements IValueMeta {
} else {
identicalFormat = false;
}
+ } else {
+ // String and the other non-numeric types: matching encodings mean the stored bytes
+ // already have this field's format.
+ identicalFormat = true;
}
+ } else {
+ // The bytes were stored in another encoding, so they have to be decoded with the storage
+ // metadata before this field can use them.
+ identicalFormat = false;
}
}
}
diff --git a/core/src/test/java/org/apache/hop/core/gui/plugin/GuiRegistryTest.java b/core/src/test/java/org/apache/hop/core/gui/plugin/GuiRegistryTest.java
index b723e938f2..b1baa78207 100644
--- a/core/src/test/java/org/apache/hop/core/gui/plugin/GuiRegistryTest.java
+++ b/core/src/test/java/org/apache/hop/core/gui/plugin/GuiRegistryTest.java
@@ -19,6 +19,7 @@ package org.apache.hop.core.gui.plugin;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -26,6 +27,10 @@ import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.List;
+import org.apache.hop.core.action.GuiContextActionFilter;
+import org.apache.hop.core.gui.plugin.action.GuiActionFilter;
+import org.apache.hop.core.gui.plugin.toolbar.GuiToolbarElementFilter;
+import org.apache.hop.core.gui.plugin.toolbar.GuiToolbarItemFilter;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -117,6 +122,46 @@ class GuiRegistryTest {
assertTrue(elements.getChildren().get(0).isIgnored());
}
+ @Test
+ void actionFilterIdUsesPluginClassName() throws Exception {
+ Method method =
+ WidgetSample.class.getDeclaredMethod("sampleActionFilter", String.class, Object.class);
+ GuiContextActionFilter filter = method.getAnnotation(GuiContextActionFilter.class);
+ String className = WidgetSample.class.getName();
+ registry.addGuiActionFilter(className, method, filter, getClass().getClassLoader());
+
+ List<GuiActionFilter> filters = registry.getGuiContextActionFilters(WidgetSample.PARENT_ID);
+ assertNotNull(filters);
+ boolean found = false;
+ for (GuiActionFilter f : filters) {
+ if (f.getId().equals(className + ".sampleActionFilter")) {
+ found = true;
+ break;
+ }
+ }
+ assertTrue(found, "Action filter ID should use plugin class name, not java.lang.String");
+ }
+
+ @Test
+ void toolbarItemFilterIdUsesPluginClassName() throws Exception {
+ Method method = WidgetSample.class.getDeclaredMethod("sampleToolbarFilter", String.class);
+ GuiToolbarElementFilter filter = method.getAnnotation(GuiToolbarElementFilter.class);
+ String className = WidgetSample.class.getName();
+ registry.addGuiToolbarItemFilter(className, method, filter, getClass().getClassLoader());
+
+ List<GuiToolbarItemFilter> filters =
+ registry.getToolbarItemFiltersMap().get(WidgetSample.PARENT_ID);
+ assertNotNull(filters);
+ boolean found = false;
+ for (GuiToolbarItemFilter f : filters) {
+ if (f.getId().equals(className + ".sampleToolbarFilter")) {
+ found = true;
+ break;
+ }
+ }
+ assertTrue(found, "Toolbar item filter ID should use plugin class name, not java.lang.String");
+ }
+
@Test
void tableFieldRegistersTheRowColumns() throws Exception {
String dataClassName = getClass().getName() + "#tables";
@@ -233,6 +278,16 @@ class GuiRegistryTest {
void browse() {
// Only the annotation matters here.
}
+
+ @GuiContextActionFilter(parentId = PARENT_ID)
+ public boolean sampleActionFilter(String actionId, Object context) {
+ return true;
+ }
+
+ @GuiToolbarElementFilter(parentId = PARENT_ID)
+ public boolean sampleToolbarFilter(String toolbarId) {
+ return true;
+ }
}
public static class TableHost {
diff --git a/core/src/test/java/org/apache/hop/core/row/value/ValueMetaBaseTest.java b/core/src/test/java/org/apache/hop/core/row/value/ValueMetaBaseTest.java
index b590e398b3..e07d8a92f7 100644
--- a/core/src/test/java/org/apache/hop/core/row/value/ValueMetaBaseTest.java
+++ b/core/src/test/java/org/apache/hop/core/row/value/ValueMetaBaseTest.java
@@ -42,6 +42,7 @@ import java.io.DataOutputStream;
import java.io.IOException;
import java.math.BigDecimal;
import java.net.InetAddress;
+import java.nio.charset.StandardCharsets;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
@@ -71,6 +72,7 @@ import org.apache.hop.core.logging.HopLoggingEvent;
import org.apache.hop.core.logging.IHopLoggingEventListener;
import org.apache.hop.core.plugins.PluginRegistry;
import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.RowMeta;
import org.apache.hop.core.util.TestUtil;
import org.apache.hop.core.variables.IVariables;
import org.apache.hop.core.variables.Variables;
@@ -439,6 +441,47 @@ class ValueMetaBaseTest {
assertArrayEquals("true".getBytes(), longer.getBinaryString("Y".getBytes()));
}
+ /**
+ * Lazy text is stored as bytes in the file encoding. Metadata XML has to keep that encoding, and
+ * indexed values have to come back as the original objects, or a later read decodes the bytes
+ * with the platform charset and drops the index.
+ */
+ @Test
+ void metaXmlKeepsBinaryStringEncodingAndIndexedValues() throws Exception {
+ ValueMetaString lazy = new ValueMetaString("lazy");
+ lazy.setStorageType(IValueMeta.STORAGE_TYPE_BINARY_STRING);
+ ValueMetaString storage = new ValueMetaString("lazy");
+ storage.setStringEncoding(StandardCharsets.ISO_8859_1.name());
+ lazy.setStorageMetadata(storage);
+
+ ValueMetaString indexed = new ValueMetaString("indexed");
+ indexed.setStorageType(IValueMeta.STORAGE_TYPE_INDEXED);
+ indexed.setIndex(new Object[] {"alpha", "bravo"});
+
+ ValueMetaBinary binary = new ValueMetaBinary("bytes");
+ binary.setStorageType(IValueMeta.STORAGE_TYPE_INDEXED);
+ binary.setIndex(new Object[] {new byte[] {1}, new byte[] {9, -1}});
+
+ RowMeta meta = new RowMeta();
+ meta.addValueMeta(lazy);
+ meta.addValueMeta(indexed);
+ meta.addValueMeta(binary);
+
+ byte[] latin1 = "caf\u00e9".getBytes(StandardCharsets.ISO_8859_1);
+ assertEquals("caf\u00e9", lazy.getString(latin1));
+
+ RowMeta restored =
+ new RowMeta(
+ XmlHandler.getSubNode(
+ XmlHandler.loadXmlString(meta.getMetaXml()), RowMeta.XML_META_TAG));
+ assertEquals(
+ StandardCharsets.ISO_8859_1.name(),
+ restored.getValueMeta(0).getStorageMetadata().getStringEncoding());
+ assertEquals("caf\u00e9", restored.getValueMeta(0).getString(latin1));
+ assertEquals("bravo", restored.getValueMeta(1).getString(1));
+ assertArrayEquals(new byte[] {9, -1}, restored.getValueMeta(2).getBinary(1));
+ }
+
@Test
void testConvertDataFromStringToBooleanWithMask() throws Exception {
ValueMetaBoolean target = new ValueMetaBoolean("b");
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-debezium-input.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-debezium-input.svg
new file mode 100644
index 0000000000..7bccc058a5
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-debezium-input.svg
@@ -0,0 +1,23 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 32 32">
+ <g stroke="#2876bc" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
+ <path fill="#e8f2fc" d="M3 8v16c0 5 17 5 17 0V8"/>
+ <ellipse cx="11.5" cy="8" rx="8.5" ry="4" fill="#ffffff"/>
+ <path fill="none" d="M3 16c0 5 17 5 17 0"/>
+ </g>
+ <path d="M15 23h4l2-5 3 10 3-5h4M27 20l4 3-4 3" fill="none" stroke="#e87524" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-elasticsearch-input.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-elasticsearch-input.svg
new file mode 100644
index 0000000000..185b46fe7b
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-elasticsearch-input.svg
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 32 32">
+ <title>Beam Elasticsearch input</title>
+ <path d="M13 7v14c0 3 16 3 16 0V7" fill="#0f766e" stroke="#115e59" stroke-width="1.3"/>
+ <ellipse cx="21" cy="7" rx="8" ry="4" fill="#2dd4bf" stroke="#115e59" stroke-width="1.3"/>
+ <path d="M13 13c0 3 16 3 16 0M13 18c0 3 16 3 16 0" fill="none" stroke="#99f6e4" stroke-width="1.3"/>
+ <path d="M17 7h8M21 5v4" stroke="#115e59" stroke-width="1.5" stroke-linecap="round"/>
+ <path d="M3 25h11m-4-4 4 4-4 4" fill="none" stroke="#f97316" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"/>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-elasticsearch-output.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-elasticsearch-output.svg
new file mode 100644
index 0000000000..7cc022240a
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-elasticsearch-output.svg
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 32 32">
+ <title>Beam Elasticsearch output</title>
+ <path d="M13 7v14c0 3 16 3 16 0V7" fill="#0f766e" stroke="#115e59" stroke-width="1.3"/>
+ <ellipse cx="21" cy="7" rx="8" ry="4" fill="#2dd4bf" stroke="#115e59" stroke-width="1.3"/>
+ <path d="M13 13c0 3 16 3 16 0M13 18c0 3 16 3 16 0" fill="none" stroke="#99f6e4" stroke-width="1.3"/>
+ <path d="M17 7h8M21 5v4" stroke="#115e59" stroke-width="1.5" stroke-linecap="round"/>
+ <path d="M14 25H3m4-4-4 4 4 4" fill="none" stroke="#f97316" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"/>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-mqtt-input.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-mqtt-input.svg
new file mode 100644
index 0000000000..cce6ff208e
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-mqtt-input.svg
@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" width="64" height="64" viewBox="0 0 64 64">
+ <g fill="none" stroke="#0099cc" stroke-width="3" stroke-linecap="round" stroke-linejoin="round">
+ <path d="M13 16h12m14 0h12M32 10v8"/>
+ <rect x="25" y="18" width="14" height="14" rx="4"/>
+ <circle cx="9" cy="16" r="4"/><circle cx="55" cy="16" r="4"/><circle cx="32" cy="6" r="4"/>
+ <path d="M27 32h10M32 37v18m-7-7 7 7 7-7"/>
+ </g>
+ <path d="M5 58h16m22 0h16" stroke="#7e57c2" stroke-width="4" stroke-linecap="round"/>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-mqtt-output.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-mqtt-output.svg
new file mode 100644
index 0000000000..9eced8d7a3
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-mqtt-output.svg
@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" width="64" height="64" viewBox="0 0 64 64">
+ <g fill="none" stroke="#e87b23" stroke-width="3" stroke-linecap="round" stroke-linejoin="round">
+ <path d="M13 16h12m14 0h12M32 10v8"/>
+ <rect x="25" y="18" width="14" height="14" rx="4"/>
+ <circle cx="9" cy="16" r="4"/><circle cx="55" cy="16" r="4"/><circle cx="32" cy="6" r="4"/>
+ <path d="M27 32h10M32 55V37m-7 7 7-7 7 7"/>
+ </g>
+ <path d="M5 58h16m22 0h16" stroke="#7e57c2" stroke-width="4" stroke-linecap="round"/>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-partition.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-partition.svg
new file mode 100644
index 0000000000..d71e88c386
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-partition.svg
@@ -0,0 +1,869 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+ xmlns="http://www.w3.org/2000/svg"
+ xmlns:xlink="http://www.w3.org/1999/xlink"
+ width="114px"
+ height="153px"
+ viewBox="0 0 114.07453 153.42"
+ version="1.1"
+ id="svg184">
+ <defs
+ id="defs127">
+ <style
+ id="style2">
+ .cls-1{isolation:isolate;}.cls-2{opacity:0.15;}.cls-12,.cls-19,.cls-2{mix-blend-mode:multiply;}.cls-3{fill:url(#linear-gradient);}.cls-16,.cls-17,.cls-4,.cls-9{fill:#fff;}.cls-5{fill:#fee900;}.cls-6{fill:url(#linear-gradient-2);}.cls-7{fill:url(#linear-gradient-3);}.cls-8{opacity:0.65;fill:url(#linear-gradient-4);}.cls-9{opacity:0.9;}.cls-10,.cls-11{opacity:0.75;}.cls-10{fill:url(#linear-gradient-5);}.cls-12{opacity:0.1;}.cls-13{fill:url(#linear-gradient-6);}.cls-14{fill:url( [...]
+ </style>
+ <linearGradient
+ id="linear-gradient"
+ x1="4.69453"
+ y1="74.80493"
+ x2="8.26761"
+ y2="74.80493"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#ffc400"
+ id="stop4"/>
+ <stop
+ offset="0.13695"
+ stop-color="#fecd00"
+ id="stop6"/>
+ <stop
+ offset="0.36528"
+ stop-color="#fed400"
+ id="stop8"/>
+ <stop
+ offset="1"
+ stop-color="#fed600"
+ id="stop10"/>
+ </linearGradient>
+ <linearGradient
+ id="linear-gradient-2"
+ x1="4.5"
+ y1="62.67316"
+ x2="78.96997"
+ y2="62.67316"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#ff0"
+ stop-opacity="0.7"
+ id="stop13"/>
+ <stop
+ offset="0.13194"
+ stop-color="#fff600"
+ stop-opacity="0.66022"
+ id="stop15"/>
+ <stop
+ offset="0.31385"
+ stop-color="#feef00"
+ stop-opacity="0.62592"
+ id="stop17"/>
+ <stop
+ offset="0.54652"
+ stop-color="#feea00"
+ stop-opacity="0.60587"
+ id="stop19"/>
+ <stop
+ offset="1"
+ stop-color="#fee900"
+ stop-opacity="0.6"
+ id="stop21"/>
+ </linearGradient>
+ <linearGradient
+ id="linear-gradient-3"
+ x1="4.50001"
+ y1="52.7647"
+ x2="78.96999"
+ y2="52.7647"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#fff176"
+ stop-opacity="0.7"
+ id="stop24"/>
+ <stop
+ offset="0.05838"
+ stop-color="#ffef5f"
+ stop-opacity="0.68061"
+ id="stop26"/>
+ <stop
+ offset="0.1703"
+ stop-color="#ffed3c"
+ stop-opacity="0.65121"
+ id="stop28"/>
+ <stop
+ offset="0.2969"
+ stop-color="#feeb21"
+ stop-opacity="0.62829"
+ id="stop30"/>
+ <stop
+ offset="0.44466"
+ stop-color="#feea0e"
+ stop-opacity="0.61215"
+ id="stop32"/>
+ <stop
+ offset="0.63299"
+ stop-color="#fee903"
+ stop-opacity="0.60275"
+ id="stop34"/>
+ <stop
+ offset="1"
+ stop-color="#fee900"
+ stop-opacity="0.6"
+ id="stop36"/>
+ </linearGradient>
+ <linearGradient
+ id="linear-gradient-4"
+ x1="4.59729"
+ y1="41.31817"
+ x2="78.87268"
+ y2="41.31817"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#fff"
+ stop-opacity="0.1"
+ id="stop39"/>
+ <stop
+ offset="0.01007"
+ stop-color="#fff"
+ stop-opacity="0.13207"
+ id="stop41"/>
+ <stop
+ offset="0.07239"
+ stop-color="#fff"
+ stop-opacity="0.31249"
+ id="stop43"/>
+ <stop
+ offset="0.14048"
+ stop-color="#fff"
+ stop-opacity="0.47688"
+ id="stop45"/>
+ <stop
+ offset="0.21288"
+ stop-color="#fff"
+ stop-opacity="0.61839"
+ id="stop47"/>
+ <stop
+ offset="0.29078"
+ stop-color="#fff"
+ stop-opacity="0.73733"
+ id="stop49"/>
+ <stop
+ offset="0.37589"
+ stop-color="#fff"
+ stop-opacity="0.83389"
+ id="stop51"/>
+ <stop
+ offset="0.47118"
+ stop-color="#fff"
+ stop-opacity="0.90827"
+ id="stop53"/>
+ <stop
+ offset="0.58234"
+ stop-color="#fff"
+ stop-opacity="0.96063"
+ id="stop55"/>
+ <stop
+ offset="0.72402"
+ stop-color="#fff"
+ stop-opacity="0.99108"
+ id="stop57"/>
+ <stop
+ offset="1"
+ stop-color="#fff"
+ id="stop59"/>
+ </linearGradient>
+ <linearGradient
+ id="linear-gradient-5"
+ x1="4.50001"
+ y1="74.76326"
+ x2="40.30806"
+ y2="74.76326"
+ xlink:href="#linear-gradient"/>
+ <linearGradient
+ id="linear-gradient-6"
+ x1="4.5"
+ y1="96.62164"
+ x2="78.96999"
+ y2="96.62164"
+ xlink:href="#linear-gradient"/>
+ <linearGradient
+ id="linear-gradient-7"
+ x1="4.5"
+ y1="107.85788"
+ x2="78.96997"
+ y2="107.85788"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#ffab00"
+ stop-opacity="0.7"
+ id="stop64"/>
+ <stop
+ offset="0.02342"
+ stop-color="#ffad04"
+ stop-opacity="0.72443"
+ id="stop66"/>
+ <stop
+ offset="0.13953"
+ stop-color="#fdb413"
+ stop-opacity="0.82494"
+ id="stop68"/>
+ <stop
+ offset="0.27082"
+ stop-color="#fcb91e"
+ stop-opacity="0.90329"
+ id="stop70"/>
+ <stop
+ offset="0.42404"
+ stop-color="#fcbd27"
+ stop-opacity="0.95848"
+ id="stop72"/>
+ <stop
+ offset="0.61939"
+ stop-color="#fbbf2c"
+ stop-opacity="0.99059"
+ id="stop74"/>
+ <stop
+ offset="1"
+ stop-color="#fbc02d"
+ id="stop76"/>
+ </linearGradient>
+ <linearGradient
+ id="linear-gradient-8"
+ x1="4.69459"
+ y1="108.51975"
+ x2="78.96999"
+ y2="108.51975"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#fff"
+ stop-opacity="0"
+ id="stop79"/>
+ <stop
+ offset="0.00958"
+ stop-color="#fffffc"
+ stop-opacity="0.01588"
+ id="stop81"/>
+ <stop
+ offset="0.17414"
+ stop-color="#fffad1"
+ stop-opacity="0.2726"
+ id="stop83"/>
+ <stop
+ offset="0.33615"
+ stop-color="#fff6ac"
+ stop-opacity="0.49416"
+ id="stop85"/>
+ <stop
+ offset="0.4916"
+ stop-color="#fef38e"
+ stop-opacity="0.67545"
+ id="stop87"/>
+ <stop
+ offset="0.63917"
+ stop-color="#fef076"
+ stop-opacity="0.81653"
+ id="stop89"/>
+ <stop
+ offset="0.77694"
+ stop-color="#feee65"
+ stop-opacity="0.91757"
+ id="stop91"/>
+ <stop
+ offset="0.90112"
+ stop-color="#feed5b"
+ stop-opacity="0.9787"
+ id="stop93"/>
+ <stop
+ offset="1"
+ stop-color="#feed57"
+ id="stop95"/>
+ <stop
+ offset="1"
+ stop-color="#fef17e"
+ id="stop97"/>
+ <stop
+ offset="1"
+ stop-color="#fef5a1"
+ id="stop99"/>
+ <stop
+ offset="1"
+ stop-color="#fff8be"
+ id="stop101"/>
+ <stop
+ offset="1"
+ stop-color="#fffbd6"
+ id="stop103"/>
+ <stop
+ offset="1"
+ stop-color="#fffde8"
+ id="stop105"/>
+ <stop
+ offset="1"
+ stop-color="#fffef5"
+ id="stop107"/>
+ <stop
+ offset="1"
+ stop-color="#fffffd"
+ id="stop109"/>
+ <stop
+ offset="1"
+ stop-color="#fff"
+ id="stop111"/>
+ </linearGradient>
+ <linearGradient
+ id="linear-gradient-9"
+ x1="48.65752"
+ y1="100.03092"
+ x2="151.53701"
+ y2="100.03092"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#ffca28"
+ id="stop114"/>
+ <stop
+ offset="0.25055"
+ stop-color="#f3931a"
+ id="stop116"/>
+ <stop
+ offset="0.48896"
+ stop-color="#ea670f"
+ id="stop118"/>
+ <stop
+ offset="0.70088"
+ stop-color="#e34707"
+ id="stop120"/>
+ <stop
+ offset="0.87843"
+ stop-color="#df3302"
+ id="stop122"/>
+ <stop
+ offset="1"
+ stop-color="#dd2c00"
+ id="stop124"/>
+ </linearGradient>
+ <linearGradient
+ xlink:href="#linear-gradient"
+ id="linearGradient995"
+ gradientUnits="userSpaceOnUse"
+ x1="4.69453"
+ y1="74.804932"
+ x2="8.2676096"
+ y2="74.804932"/>
+ <linearGradient
+ id="linear-gradient-2-9"
+ x1="4.5"
+ y1="62.673161"
+ x2="78.969971"
+ y2="62.673161"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#ff0"
+ stop-opacity="0.7"
+ id="stop13-1"/>
+ <stop
+ offset="0.13194"
+ stop-color="#fff600"
+ stop-opacity="0.66022"
+ id="stop15-2"/>
+ <stop
+ offset="0.31385"
+ stop-color="#feef00"
+ stop-opacity="0.62592"
+ id="stop17-7"/>
+ <stop
+ offset="0.54652"
+ stop-color="#feea00"
+ stop-opacity="0.60587"
+ id="stop19-0"/>
+ <stop
+ offset="1"
+ stop-color="#fee900"
+ stop-opacity="0.6"
+ id="stop21-9"/>
+ </linearGradient>
+ <linearGradient
+ id="linear-gradient-3-3"
+ x1="4.50001"
+ y1="52.764702"
+ x2="78.969994"
+ y2="52.764702"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#fff176"
+ stop-opacity="0.7"
+ id="stop24-6"/>
+ <stop
+ offset="0.05838"
+ stop-color="#ffef5f"
+ stop-opacity="0.68061"
+ id="stop26-0"/>
+ <stop
+ offset="0.1703"
+ stop-color="#ffed3c"
+ stop-opacity="0.65121"
+ id="stop28-6"/>
+ <stop
+ offset="0.2969"
+ stop-color="#feeb21"
+ stop-opacity="0.62829"
+ id="stop30-2"/>
+ <stop
+ offset="0.44466"
+ stop-color="#feea0e"
+ stop-opacity="0.61215"
+ id="stop32-6"/>
+ <stop
+ offset="0.63299"
+ stop-color="#fee903"
+ stop-opacity="0.60275"
+ id="stop34-1"/>
+ <stop
+ offset="1"
+ stop-color="#fee900"
+ stop-opacity="0.6"
+ id="stop36-8"/>
+ </linearGradient>
+ <linearGradient
+ id="linear-gradient-4-7"
+ x1="4.59729"
+ y1="41.318169"
+ x2="78.872681"
+ y2="41.318169"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#fff"
+ stop-opacity="0.1"
+ id="stop39-9"/>
+ <stop
+ offset="0.01007"
+ stop-color="#fff"
+ stop-opacity="0.13207"
+ id="stop41-2"/>
+ <stop
+ offset="0.07239"
+ stop-color="#fff"
+ stop-opacity="0.31249"
+ id="stop43-0"/>
+ <stop
+ offset="0.14048"
+ stop-color="#fff"
+ stop-opacity="0.47688"
+ id="stop45-2"/>
+ <stop
+ offset="0.21288"
+ stop-color="#fff"
+ stop-opacity="0.61839"
+ id="stop47-3"/>
+ <stop
+ offset="0.29078"
+ stop-color="#fff"
+ stop-opacity="0.73733"
+ id="stop49-7"/>
+ <stop
+ offset="0.37589"
+ stop-color="#fff"
+ stop-opacity="0.83389"
+ id="stop51-5"/>
+ <stop
+ offset="0.47118"
+ stop-color="#fff"
+ stop-opacity="0.90827"
+ id="stop53-9"/>
+ <stop
+ offset="0.58234"
+ stop-color="#fff"
+ stop-opacity="0.96063"
+ id="stop55-2"/>
+ <stop
+ offset="0.72402"
+ stop-color="#fff"
+ stop-opacity="0.99108"
+ id="stop57-2"/>
+ <stop
+ offset="1"
+ stop-color="#fff"
+ id="stop59-8"/>
+ </linearGradient>
+ <linearGradient
+ id="linear-gradient-5-9"
+ x1="4.50001"
+ y1="74.76326"
+ x2="40.30806"
+ y2="74.76326"
+ xlink:href="#linear-gradient"/>
+ <linearGradient
+ id="linear-gradient-6-7"
+ x1="4.5"
+ y1="96.621643"
+ x2="78.969994"
+ y2="96.621643"
+ xlink:href="#linear-gradient"/>
+ <linearGradient
+ id="linear-gradient-7-3"
+ x1="4.5"
+ y1="107.85788"
+ x2="78.969971"
+ y2="107.85788"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#ffab00"
+ stop-opacity="0.7"
+ id="stop64-6"/>
+ <stop
+ offset="0.02342"
+ stop-color="#ffad04"
+ stop-opacity="0.72443"
+ id="stop66-1"/>
+ <stop
+ offset="0.13953"
+ stop-color="#fdb413"
+ stop-opacity="0.82494"
+ id="stop68-2"/>
+ <stop
+ offset="0.27082"
+ stop-color="#fcb91e"
+ stop-opacity="0.90329"
+ id="stop70-9"/>
+ <stop
+ offset="0.42404"
+ stop-color="#fcbd27"
+ stop-opacity="0.95848"
+ id="stop72-3"/>
+ <stop
+ offset="0.61939"
+ stop-color="#fbbf2c"
+ stop-opacity="0.99059"
+ id="stop74-1"/>
+ <stop
+ offset="1"
+ stop-color="#fbc02d"
+ id="stop76-9"/>
+ </linearGradient>
+ <linearGradient
+ id="linear-gradient-8-4"
+ x1="4.6945901"
+ y1="108.51975"
+ x2="78.969994"
+ y2="108.51975"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#fff"
+ stop-opacity="0"
+ id="stop79-7"/>
+ <stop
+ offset="0.00958"
+ stop-color="#fffffc"
+ stop-opacity="0.01588"
+ id="stop81-8"/>
+ <stop
+ offset="0.17414"
+ stop-color="#fffad1"
+ stop-opacity="0.2726"
+ id="stop83-4"/>
+ <stop
+ offset="0.33615"
+ stop-color="#fff6ac"
+ stop-opacity="0.49416"
+ id="stop85-5"/>
+ <stop
+ offset="0.4916"
+ stop-color="#fef38e"
+ stop-opacity="0.67545"
+ id="stop87-0"/>
+ <stop
+ offset="0.63917"
+ stop-color="#fef076"
+ stop-opacity="0.81653"
+ id="stop89-3"/>
+ <stop
+ offset="0.77694"
+ stop-color="#feee65"
+ stop-opacity="0.91757"
+ id="stop91-6"/>
+ <stop
+ offset="0.90112"
+ stop-color="#feed5b"
+ stop-opacity="0.9787"
+ id="stop93-1"/>
+ <stop
+ offset="1"
+ stop-color="#feed57"
+ id="stop95-0"/>
+ <stop
+ offset="1"
+ stop-color="#fef17e"
+ id="stop97-6"/>
+ <stop
+ offset="1"
+ stop-color="#fef5a1"
+ id="stop99-3"/>
+ <stop
+ offset="1"
+ stop-color="#fff8be"
+ id="stop101-2"/>
+ <stop
+ offset="1"
+ stop-color="#fffbd6"
+ id="stop103-0"/>
+ <stop
+ offset="1"
+ stop-color="#fffde8"
+ id="stop105-6"/>
+ <stop
+ offset="1"
+ stop-color="#fffef5"
+ id="stop107-1"/>
+ <stop
+ offset="1"
+ stop-color="#fffffd"
+ id="stop109-5"/>
+ <stop
+ offset="1"
+ stop-color="#fff"
+ id="stop111-5"/>
+ </linearGradient>
+ <linearGradient
+ id="linear-gradient-9-4"
+ x1="48.65752"
+ y1="100.03092"
+ x2="151.537"
+ y2="100.03092"
+ gradientUnits="userSpaceOnUse">
+ <stop
+ offset="0"
+ stop-color="#ffca28"
+ id="stop114-7"/>
+ <stop
+ offset="0.25055"
+ stop-color="#f3931a"
+ id="stop116-6"/>
+ <stop
+ offset="0.48896"
+ stop-color="#ea670f"
+ id="stop118-5"/>
+ <stop
+ offset="0.70088"
+ stop-color="#e34707"
+ id="stop120-6"/>
+ <stop
+ offset="0.87843"
+ stop-color="#df3302"
+ id="stop122-9"/>
+ <stop
+ offset="1"
+ stop-color="#dd2c00"
+ id="stop124-3"/>
+ </linearGradient>
+ </defs>
+ <title
+ id="title129">beam-logo_full-color
+ </title>
+ <rect
+ style="opacity:1;fill:#c8c8c8;fill-opacity:1;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="rect997"
+ width="62.222397"
+ height="88.318657"
+ x="26.189129"
+ y="29.346682"
+ ry="0"/>
+ <path
+ style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ d="M 88.41153,29.346682 110.05006,10.029879 V 135.96058 L 88.41153,117.66534 Z"
+ id="path1001"/>
+ <path
+ style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ d="M 98.069926,21.174189 V 126.02357"
+ id="path1003"/>
+ <path
+ style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ d="M 88.783001,54.32851 109.86432,49.313571"
+ id="path1007"/>
+ <path
+ style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ d="m 88.597263,86.647009 20.895577,3.900508"
+ id="path1009"/>
+ <path
+ style="fill:none;stroke:#000000;stroke-width:1.00224137px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ d="M 26.189129,29.148714 4.5020958,9.7886135 V 136.00159 L 26.189129,117.66534 Z"
+ id="path1001-3"/>
+ <path
+ style="fill:none;stroke:#000000;stroke-width:1.00224137px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ d="M 16.509088,20.957905 V 126.0423"
+ id="path1003-6"/>
+ <path
+ style="fill:none;stroke:#000000;stroke-width:1.00224137px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ d="M 25.81682,54.186536 4.688253,49.160356"
+ id="path1007-7"/>
+ <path
+ style="fill:none;stroke:#000000;stroke-width:1.00224137px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ d="M 26.002984,86.577475 5.0605653,90.486722"
+ id="path1009-5"/>
+ <g
+ class="cls-1"
+ id="g182"
+ transform="matrix(0.51641342,0,0,0.51641342,29.702902,34.435194)"
+ style="isolation:isolate">
+ <g
+ id="Layer_1"
+ >
+ <image
+ class="cls-2"
+ width="40"
+ height="624"
+ transform="matrix(0.24,0,0,0.24,0.19453,0)"
+ xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACoAAAJxCAYAAAA0FSBGAAAACXBIWXMAAC4jAAAuIwF4pT92AAAMGElEQVR4Xu3W2YucVRrH8eO+xXF3UERcUBBU3C6ECI6KQrula1PichUEBcUNxCsTQlA0XuYiUTAaLxRBcyEuuZCIGtPatXRFJekoNCI6oCRK5g+o+X2f97yZ6srbdo1GcZhfw0O9dd5znudT55z3vJ0Gg0H6X4hFO/xVYtEO44T+DiljkGNe2xg5FotFO1ShBiOxKqVDyxjkGGmb1/+34BftMIorMW+00mHEIMfWf6TDq6K8v7//CH5c9KIdRnFRXNG+Mh3xVSsdSXw9kY4idO/o4Sjby36MGeQfcAD6d0EzsgSWOBCfttIx7dvSsf1703FqW/LJ7en4XSNBG/foQ1/GMLZEl+BxsGMhN2TgnI [...]
+ id="image131"
+ style="opacity:0.15;mix-blend-mode:multiply"/>
+ <rect
+ class="cls-3"
+ x="4.69453"
+ y="3"
+ width="3.5730801"
+ height="143.60986"
+ id="rect133"
+ style="fill:url(#linearGradient995)"/>
+ <path
+ class="cls-4"
+ d="m 149.85752,119.0673 a 31.95182,31.95182 0 0 0 -4.48779,-9.56738 30.16459,30.16459 0 0 0 -7.59473,-7.5459 34.29159,34.29159 0 0 0 -10.80029,-4.88232 q 10.25684,-4.33887 15.584,-12.23047 A 31.00862,31.00862 0 0 0 147.88488,67.08732 41.11393,41.11393 0 0 0 144.482,49.67863 30.92584,30.92584 0 0 0 134.668,37.59611 44.12955,44.12955 0 0 0 118.98538,30.54386 86.44973,86.44973 0 0 0 98.07529,28.226 H 48.85752 v 143.60985 h 52.66992 a 75.21864,75.21864 0 0 0 20.86084,-2.6 [...]
+ transform="translate(-44.16299,-25.22599)"
+ id="path135"
+ style="fill:#ffffff"/>
+ <polygon
+ class="cls-5"
+ points="4.5,8.252 78.97,31.595 "
+ id="polygon137"
+ style="fill:#fee900"/>
+ <polygon
+ class="cls-6"
+ points="78.97,50.541 40.308,74.805 "
+ id="polygon139"
+ style="fill:url(#linear-gradient-2-9)"/>
+ <polygon
+ class="cls-5"
+ points="40.308,74.805 78.97,50.541 78.97,31.595 4.5,8.252 4.5,52.332 "
+ id="polygon141"
+ style="fill:#fee900"/>
+ <polygon
+ class="cls-7"
+ points="78.97,31.595 4.5,8.252 4.5,52.332 4.5,52.925 4.5,97.277 78.97,50.541 "
+ id="polygon143"
+ style="fill:url(#linear-gradient-3-3)"/>
+ <polygon
+ class="cls-8"
+ points="78.873,30.603 4.597,39.263 4.597,43.373 78.873,52.033 "
+ id="polygon145"
+ style="opacity:0.65;fill:url(#linear-gradient-4-7)"/>
+ <path
+ class="cls-9"
+ d="m 114.62478,66.54416 a 8.5081,8.5081 0 0 0 8.50819,8.50819 V 58.036 a 8.50823,8.50823 0 0 0 -8.50819,8.50816 z"
+ transform="translate(-44.16299,-25.22599)"
+ id="path147"
+ style="opacity:0.9;fill:#ffffff"/>
+ <polygon
+ class="cls-10"
+ points="4.5,52.291 4.5,52.883 4.5,97.236 40.308,74.763 "
+ id="polygon149"
+ style="opacity:0.75;fill:url(#linear-gradient-5-9)"/>
+ <g
+ class="cls-11"
+ id="g155"
+ style="opacity:0.75">
+ <image
+ class="cls-12"
+ width="348"
+ height="407"
+ transform="matrix(0.24,0,0,0.24,0,45.83246)"
+ xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAV4AAAGZCAYAAADBxlkWAAAACXBIWXMAAC4jAAAuIwF4pT92AAAgAElEQVR4Xu2dC5glVXXv9703N++Ym5DkYqIXgySCM0P3OWcGNEwEnygKjiD4GEUMOsQQBIxAeASbwLx67+oeW4aEERwFFB0zBgWJYjBwFeRhlMFXkAheFd8KvsBHEu//X3tX7+rq86hz+pzT5/HP9/2+WlWnutsMM79v9VqrVpmf//znRgghRP9oeYMQQoju0vIGIYQQ3aXlDUIIIbpLyxvaAf/334q0+hohhBg3Wt7Q9IvriLYMrb6vEEKMMi1vqPtFeYkGpoz5783I7pOAhRDjTssbFtyck2ZRrMca8z+aUbxfAhZCjCstb0hvaiBcCnWXl+ovpBzWgPD5rnB/XsQSsBBi3Gh9Q66kkM [...]
+ id="image151"
+ style="opacity:0.1;mix-blend-mode:multiply"/>
+ <polygon
+ class="cls-13"
+ points="4.5,97.277 4.5,140.911 78.97,118.015 78.97,99.068 4.5,52.332 "
+ id="polygon153"
+ style="fill:url(#linear-gradient-6-7)"/>
+ </g>
+ <polygon
+ class="cls-14"
+ points="78.97,118.015 78.97,99.068 40.308,74.805 4.5,97.277 4.5,140.911 "
+ id="polygon157"
+ style="fill:url(#linear-gradient-7-3)"/>
+ <polygon
+ class="cls-15"
+ points="78.97,97.805 4.695,106.465 4.695,110.575 78.97,119.235 "
+ id="polygon159"
+ style="opacity:0.7;fill:url(#linear-gradient-8-4)"/>
+ <path
+ class="cls-16"
+ d="m 114.62478,133.74574 a 8.5081,8.5081 0 0 0 8.50819,8.50819 v -17.01637 a 8.50823,8.50823 0 0 0 -8.50819,8.50818 z"
+ transform="translate(-44.16299,-25.22599)"
+ id="path161"
+ style="opacity:0.6;fill:#ffffff"/>
+ <path
+ class="cls-17"
+ d="m 149.85752,119.0673 a 31.95182,31.95182 0 0 0 -4.48779,-9.56738 30.16459,30.16459 0 0 0 -7.59473,-7.5459 34.29159,34.29159 0 0 0 -10.80029,-4.88232 q 10.25684,-4.33887 15.584,-12.23047 A 31.00862,31.00862 0 0 0 147.88488,67.08732 41.11393,41.11393 0 0 0 144.482,49.67863 30.92584,30.92584 0 0 0 134.668,37.59611 44.12955,44.12955 0 0 0 118.98538,30.54386 86.44973,86.44973 0 0 0 98.07529,28.226 H 48.85752 v 4.54242 l 72.89409,23.71047 a 2,2 0 0 1 1.38139,1.90191 v 16 [...]
+ transform="translate(-44.16299,-25.22599)"
+ id="path163"
+ style="fill:#ffffff;stroke:#ffffff;stroke-width:0.40000001px;stroke-miterlimit:10"/>
+ <g
+ class="cls-18"
+ id="g171"
+ style="opacity:0.85">
+ <image
+ class="cls-19"
+ width="467"
+ height="638"
+ transform="matrix(0.24,0,0,0.24,1.99453,0.3)"
+ xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAdcAAAKCCAYAAACDL5MYAAAACXBIWXMAAC4jAAAuIwF4pT92AAAgAElEQVR4Xuy9DZhkVXUuvO/35eZecxPvF0luTIKJMeSC/Mz0qTMzQjAR1ECMGkFRg/8oP6KCIGpQCDYwQ0/X3rt6GAZkRBhBBRxABBRUfoLyqyKCgiKKmETQmCuiJtHovZ93vXutXbXr1Dl1qnq6u6q6l8/zPlV9avfg03X2fs9a613vMr/85S+NQqFQKBSKhUPtAoVCoVAoFMOhdoFCoVAoFIrhULtAoVAoFArFcKhdsBSg//0nhUKx46jbawqFYmlQu2C+qDsESg8GhUIxL9TtrUFRt68VCsVgqF0wKPoR5bQx/49CoRg/7AhJ150JCsVKRu2COnRttgoifakx/2+K7QqFYslQ3H8p6s [...]
+ id="image165"
+ style="opacity:0.2;mix-blend-mode:multiply"/>
+ <path
+ class="cls-20"
+ d="m 149.85752,119.0673 a 31.95182,31.95182 0 0 0 -4.48779,-9.56738 30.16459,30.16459 0 0 0 -7.59473,-7.5459 34.29159,34.29159 0 0 0 -10.80029,-4.88232 q 10.25684,-4.33887 15.584,-12.23047 A 31.00862,31.00862 0 0 0 147.88488,67.08732 41.11393,41.11393 0 0 0 144.482,49.67863 30.92584,30.92584 0 0 0 134.668,37.59611 44.12955,44.12955 0 0 0 118.98538,30.54386 86.44973,86.44973 0 0 0 98.07529,28.226 H 48.85752 v 5.54242 l 72.89409,22.71047 a 2,2 0 0 1 1.38139,1.90191 [...]
+ transform="translate(-44.16299,-25.22599)"
+ id="path167"
+ style="fill:#ff6d00"/>
+ <path
+ class="cls-21"
+ d="m 149.85752,119.0673 a 31.95182,31.95182 0 0 0 -4.48779,-9.56738 30.16459,30.16459 0 0 0 -7.59473,-7.5459 34.29159,34.29159 0 0 0 -10.80029,-4.88232 q 10.25684,-4.33887 15.584,-12.23047 A 31.00862,31.00862 0 0 0 147.88488,67.08732 41.11393,41.11393 0 0 0 144.482,49.67863 30.92584,30.92584 0 0 0 134.668,37.59611 44.12955,44.12955 0 0 0 118.98538,30.54386 86.44973,86.44973 0 0 0 98.07529,28.226 H 48.85752 v 5.54242 l 72.89409,22.71047 a 2,2 0 0 1 1.38139,1.90191 [...]
+ transform="translate(-44.16299,-25.22599)"
+ id="path169"
+ style="fill:none;stroke:url(#linear-gradient-9-4);stroke-width:0.40000001px;stroke-miterlimit:10"/>
+ </g>
+ <path
+ class="cls-22"
+ d="m 122.38828,169.17277 a 43.55411,43.55411 0 0 0 15.68262,-7.94043 34.29293,34.29293 0 0 0 9.86328,-13.11816 45.37613,45.37613 0 0 0 3.40283,-18.19727 39.65034,39.65034 0 0 0 -1.47949,-10.84961 31.95182,31.95182 0 0 0 -4.48779,-9.56738 30.16459,30.16459 0 0 0 -7.59473,-7.5459 34.29159,34.29159 0 0 0 -10.80029,-4.88232 q 10.25684,-4.33887 15.584,-12.23047 a 31.09,31.09 0 0 0 2.85913,-5.29706 L 122.7587,56.885 h -0.32648 a 1.98459,1.98459 0 0 1 0.70074,1.49573 v 16.27 [...]
+ transform="translate(-44.16299,-25.22599)"
+ id="path173"
+ style="opacity:0.25;fill:#e64a19"/>
+ <path
+ class="cls-23"
+ d="m 122.38828,169.17277 a 43.55411,43.55411 0 0 0 15.68262,-7.94043 34.29293,34.29293 0 0 0 9.86328,-13.11816 45.37613,45.37613 0 0 0 3.40283,-18.19727 39.65034,39.65034 0 0 0 -1.47949,-10.84961 31.95182,31.95182 0 0 0 -4.48779,-9.56738 30.16459,30.16459 0 0 0 -7.59473,-7.5459 34.29159,34.29159 0 0 0 -10.80029,-4.88232 37.47036,37.47036 0 0 0 11.22754,-7.13892 L 123.30039,75.03092 h -0.21777 a 1.98783,1.98783 0 0 1 -0.97174,1.36945 l -37.34688,23.63049 37.34692,23.63 [...]
+ transform="translate(-44.16299,-25.22599)"
+ id="path175"
+ style="opacity:0.35;fill:#e64a19"/>
+ <path
+ class="cls-24"
+ d="m 122.38828,169.17277 a 43.55411,43.55411 0 0 0 15.68262,-7.94043 34.19423,34.19423 0 0 0 9.51678,-12.37769 L 123.0452,124.31217 h -0.251 a 1.9864,1.9864 0 0 1 0.33881,1.09381 v 16.275 a 2,2 0 0 1 -1.38135,1.90192 l -72.8941,22.71051 v 5.54248 h 52.66992 a 75.21864,75.21864 0 0 0 20.8608,-2.66312 z"
+ transform="translate(-44.16299,-25.22599)"
+ id="path177"
+ style="opacity:0.5;fill:#e64a19"/>
+ <path
+ class="cls-25"
+ d="m 122.38828,169.17277 a 43.55411,43.55411 0 0 0 15.68262,-7.94043 c 0.63562,-0.5191 1.242,-1.06281 1.83325,-1.6192 l -17.01062,-17.01062 h -0.0124 a 1.97661,1.97661 0 0 1 -1.12952,0.98035 l -72.8941,22.71051 v 5.54248 h 52.66992 a 75.21864,75.21864 0 0 0 20.86084,-2.66309 z"
+ transform="translate(-44.16299,-25.22599)"
+ id="path179"
+ style="opacity:0.8;fill:#e64a19"/>
+ </g>
+ </g>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-snowflake-input.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-snowflake-input.svg
new file mode 100644
index 0000000000..7b63d4ba83
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-snowflake-input.svg
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" width="64" height="64" viewBox="0 0 64 64">
+ <g fill="none" stroke="#2f6fad" stroke-width="3" stroke-linecap="round" stroke-linejoin="round">
+ <rect x="10" y="14" width="44" height="36" rx="4"/>
+ <path d="M18 28h20M18 36h12"/>
+ <path d="M42 40l8 6-8 6"/>
+ </g>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-snowflake-output.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-snowflake-output.svg
new file mode 100644
index 0000000000..5d55e9c0e4
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-snowflake-output.svg
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" width="64" height="64" viewBox="0 0 64 64">
+ <g fill="none" stroke="#c47b2b" stroke-width="3" stroke-linecap="round" stroke-linejoin="round">
+ <rect x="10" y="14" width="44" height="36" rx="4"/>
+ <path d="M18 28h20M18 36h12"/>
+ <path d="M42 40l8 6-8 6"/>
+ </g>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-splunk-output.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-splunk-output.svg
new file mode 100644
index 0000000000..e70763b846
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/beam-splunk-output.svg
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" width="64" height="64" viewBox="0 0 64 64">
+ <g fill="none" stroke="#65a637" stroke-width="3" stroke-linecap="round" stroke-linejoin="round">
+ <rect x="10" y="14" width="44" height="36" rx="4"/>
+ <path d="M18 28h20M18 36h12"/>
+ <path d="M42 40l8 6-8 6"/>
+ </g>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc b/docs/hop-user-manual/modules/ROOT/nav.adoc
index fdb6387002..39988ce0b9 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -66,6 +66,7 @@ under the License.
**** xref:pipeline/beam/beam-samples-spark.adoc[Apache Spark]
**** xref:pipeline/beam/beam-samples-dataflow.adoc[Google Cloud Dataflow]
*** xref:pipeline/beam/flink-k8s-operator-running-hop-pipeline.adoc[Running a Hop pipeline using the Flink Kubernetes Operator]
+*** xref:pipeline/beam/developing-beam-transforms.adoc[Developing a Beam transform]
** xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark engine]
*** xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems]
*** xref:pipeline/spark/cluster-mapping-walkthrough.adoc[Cluster mapping walk-through]
@@ -109,6 +110,14 @@ under the License.
*** xref:pipeline/transforms/beambigtableoutput.adoc[Beam Bigtable output]
*** xref:pipeline/transforms/beambigqueryinput.adoc[Beam BigQuery Input]
*** xref:pipeline/transforms/beambigqueryoutput.adoc[Beam BigQuery Output]
+*** xref:pipeline/transforms/beamdebeziuminput.adoc[Beam Debezium input]
+*** xref:pipeline/transforms/beamelasticsearchinput.adoc[Beam Elasticsearch input]
+*** xref:pipeline/transforms/beamelasticsearchoutput.adoc[Beam Elasticsearch output]
+*** xref:pipeline/transforms/beammqttinput.adoc[Beam MQTT input]
+*** xref:pipeline/transforms/beammqttoutput.adoc[Beam MQTT output]
+*** xref:pipeline/transforms/beamsnowflakeinput.adoc[Beam Snowflake input]
+*** xref:pipeline/transforms/beamsnowflakeoutput.adoc[Beam Snowflake output]
+*** xref:pipeline/transforms/beamsplunkoutput.adoc[Beam Splunk output]
*** xref:pipeline/transforms/beamfileinput.adoc[Beam file input]
*** xref:pipeline/transforms/beamfileoutput.adoc[Beam file output]
*** xref:pipeline/transforms/beamgcppublisher.adoc[Beam GCP Pub/Sub: publish]
@@ -118,6 +127,7 @@ under the License.
*** xref:pipeline/transforms/binaryfileoutput.adoc[Binary File Output]
*** xref:pipeline/transforms/beamkinesisconsume.adoc[Beam Kinesis consume]
*** xref:pipeline/transforms/beamkinesisproduce.adoc[Beam Kinesis produce]
+*** xref:pipeline/transforms/beampartition.adoc[Beam partition]
*** xref:pipeline/transforms/beamtimestamp.adoc[Beam timestamp]
*** xref:pipeline/transforms/beamwindow.adoc[Beam window]
*** xref:pipeline/transforms/blockingtransform.adoc[Blocking transform]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/beam/developing-beam-transforms.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/beam/developing-beam-transforms.adoc
new file mode 100644
index 0000000000..0e25de10f5
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/beam/developing-beam-transforms.adoc
@@ -0,0 +1,68 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+:documentationPath: /pipeline/beam/
+:language: en_US
+:description: How a Hop transform is turned into an Apache Beam pipeline.
+
+= Developing a Beam transform
+
+This page is the path from a Hop pipeline to a Beam graph. It uses the current classes in the Beam engine plugin. A transform that only serializes metadata, or that only passes a unit test written against a hand-built handler call, is not yet a Beam transform.
+
+== From the engine to the graph
+
+`BeamPipelineEngine.prepareExecution()` checks that the run configuration is a `IBeamPipelineEngineRunConfiguration`, then builds a `HopPipelineMetaToBeamPipelineConverter`. `createPipeline()` creates the Beam `Pipeline`, registers `HopRowCoder` for `HopRow`, and walks the Hop pipeline in three steps:
+
+. `handleBeamInputTransforms()` for transforms whose metadata says they are inputs.
+. `handleGenericTransform()` for the transforms in between.
+. `handleBeamOutputTransforms()` for transforms whose metadata says they are outputs.
+
+A handler is resolved in this order:
+
+. An entry in the converter's handler map, keyed by the transform plugin id. This is an explicitly registered handler class.
+. The metadata object itself, when it implements `IBeamPipelineTransformHandler`. MQTT, Debezium, Elasticsearch, Splunk output, and Snowflake input and output use this form.
+. For anything else, `handleGenericTransform()` tries to run the ordinary Hop transform inside a Beam `DoFn`.
+
+The generic fallback is not a compatibility promise. It gives every worker its own copy of the transform. A transform that opens one file, one sequence, or one database session will do that once per worker. Sources that read a filename from an incoming field also fail there, because the generic path has no real input row.
+
+== What a handler receives
+
+`handleBeamInputTransforms()` always calls `handleTransform` with a null predecessor list and a null input `PCollection`. That is the production call. It does not mean the transform has no incoming hops. `BeamSourceTopology.rejectIncomingRows()` treats a non-null list as the caller's topology. A null list is resolved with `PipelineMeta.findPreviousTransforms(transformMeta, false)`, which includes enabled main hops and error hops and skips informational hops. Reject those rows before expa [...]
+
+Outputs receive the real predecessor list and the incoming `PCollection`. A sink that needs exactly one incoming transform checks that list before it expands.
+
+== Metadata, dialog, and worker code
+
+Keep the four roles separate:
+
+* The `*Meta` class holds `@HopMetadataProperty` fields and, for a self-handling transform, implements `IBeamPipelineTransformHandler`. User-visible fields also use `@GuiWidgetElement` inside a group, so the dialog is a `GuiCompositeWidgets` container rather than a hand-built form.
+* The dialog reads and writes that metadata. On OK it calls `getWidgetsContents`. It does not copy fields by hand.
+* The runtime `*Transform` class extends `BaseTransform` and refuses `processRow()` outside a Beam pipeline. Beam execution does not call it.
+* Worker functions, such as a `DoFn` or a Snowflake mapper, call `BeamHop.init()` before they touch Hop metadata. Row metadata travels as XML from `IRowMeta.getMetaXml()`, including storage type and string encoding. Do not rebuild a row meta from a type name alone.
+
+Variables are resolved while the graph is built, on the machine that submits the pipeline. A secret is decrypted there and must not be copied into an error message. Encrypt password fields with `password = true` on `@HopMetadataProperty`.
+
+== Metrics
+
+`BeamPipelineEngine` copies Beam counters onto the Hop transform by counter name. `READ` becomes lines read, `WRITTEN` becomes lines written, `INPUT` becomes lines input, and `OUTPUT` becomes lines output. Increment `WRITTEN` only after a confirmed write. The current IO transforms increment `READ` and `OUTPUT` when a row is converted and handed to Beam. That is not an acknowledgement from the remote system, and lines written stays empty.
+
+== Dependencies and compatibility
+
+A new IO library has to be a runtime dependency of `hop-engines-beam`. The plugin assembly copies runtime dependencies into `lib/core`. A jar that is only on the compiler classpath is missing on a worker. Declare `@Transform` compatibility with the engines that can run the transform. `supportedEngines = {"Beam*"}` matches every Beam engine id, including Spark, so it cannot exclude Spark. `excludedEngines` is enforced by `EngineCompatibilityResolver` when the pipeline is prepared, and it [...]
+
+== What to test
+
+Test the converter path, not only a mapper. For a source, assert that an enabled incoming hop fails before the source expands, and that a disabled or informational hop does not. For a sink, assert the incoming-hop rule. Apply the Beam transform with `Pipeline.create()` when the runner would contact a service you do not have, and do not call `run()` or `waitUntilFinish()` in that test. A serialization test proves the metadata round trip. It does not prove the graph, the worker classpath, [...]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-dataflow-pipeline-engine.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-dataflow-pipeline-engine.adoc
index dbc9ea518e..9d572acf50 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-dataflow-pipeline-engine.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-dataflow-pipeline-engine.adoc
@@ -65,6 +65,9 @@ This is required if you want to run your pipeline using the Dataflow managed ser
Also used when updating an existing pipeline.
|Staging location|Cloud Storage path for staging local files.
Must be a valid Cloud Storage URL, beginning with gs://.
+|GCP temp location|Cloud Storage path used by the transforms which must stage their data in Cloud Storage, for example the BigQuery transforms.
+Must be a valid Cloud Storage URL, beginning with gs://.
+This is a different option from the general `Temp location` below.
|Initial number of workers|The initial number of Google Compute Engine instances to use when executing your pipeline.
This option determines how many workers the Dataflow service starts up when your job begins.
|Maximum number of workers|The maximum number of Compute Engine instances to be made available to your pipeline during execution.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-direct-pipeline-engine.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-direct-pipeline-engine.adoc
index da134b1933..4a07e9662f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-direct-pipeline-engine.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-direct-pipeline-engine.adoc
@@ -48,6 +48,7 @@ From 2.22.0, direct_num_workers = 0 is supported.
When direct_num_workers is set to 0, it will set the number of threads/subprocess to the number of cores of the machine where the pipeline is running.
|User agent|A user agent string as per https://tools.ietf.org/html/rfc2616[RFC2616], describing the pipeline to external services.
|Temp location|Path for temporary files.
+|GCP temp location|Optional `gs://` path for transforms that must stage in Cloud Storage, such as BigQuery. Empty leaves that option unset. It is separate from Temp location.
|Plugins to stage (, delimited)|Comma separated list of plugins.
|Transform plugin classes|List of transform plugin classes.
|XP plugin classes|List of extensions point plugins.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc
index df4ea9377b..c6610c74ba 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc
@@ -170,3 +170,9 @@ bin/flink run \
You can specify a master of `[local]` to run using an embedded Flink engine.
It's primarily used for testing locally.
+
+== Flink job ID
+
+Hop does not show a Flink job ID after a Beam run. Beam 2.74 keeps the attached result as accumulators and a runtime, and the detached result keeps its job client in a private field with no accessor. There is no supported way to read the job ID from `PipelineResult`.
+
+The Beam job name is still available. Hop derives it from the pipeline name. On a Flink cluster, use that name in the Flink UI or the Flink REST API to find the job. The Dataflow runner is different: Hop reports that job ID through the execution detail `dataflow.job.id`.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addsequence.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addsequence.adoc
index fbfb1167ec..d65640d069 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addsequence.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addsequence.adoc
@@ -33,6 +33,19 @@ Also, they are not stored, so the values start back at the same value every time
The start, increment, and maximum can be typed in, or read from another transform. Connect that transform to Add Sequence and select it as the configuration transform. It must return exactly one row. More than one row stops the pipeline with an error. The hop is an info hop, so that row is not part of the main stream. The end field is the maximum value: after it, the sequence returns to the start value. The configuration transform cannot run in multiple copies. If Add Sequence itself run [...]
+=== On the Beam engines
+
+With a typed start, increment, and maximum on a bounded input, the Beam engines produce a single
+sequence across the whole pipeline, with no duplicates, the same as the local engine.
+To achieve this every row is keyed onto one key, so the transform runs at a parallelism of one:
+that is the trade-off for a globally increasing value.
+
+A database sequence, a configuration transform, or an unbounded input (for example Kafka or MQTT)
+runs the transform on each worker instead.
+A database sequence stays unique, because the database hands out the values.
+With a configuration transform or an unbounded input each worker keeps its own counter, so the
+values are not unique across the pipeline.
+
== Options
[options="header"]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryinput.adoc
index fe5a2ca640..b58524b6b5 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryinput.adoc
@@ -36,6 +36,12 @@ The BigQuery input transform reads data from a link:https://cloud.google.com/big
|Data set ID|The BigQuery dataset ID.
|Table ID|The BigQuery table ID.
|Query|The input query, blank means everything from the table.
+|Query location|The BigQuery dataset location that runs the query, for example `EU` or `US`.
+Leave blank to use the location of the job itself.
+Applies to a query only.
+|Skip query validation|Do not validate the query before the job starts.
+Applies to a query only.
+Use this when the query is created or changed by an earlier step, and is therefore not registered when the job is submitted.
|Return fields selection|A list of result fields.
|BQ Field name|The field name in the BigQuery table.
|Rename to... (optional)|The name to be given to a column.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamdebeziuminput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamdebeziuminput.adoc
new file mode 100644
index 0000000000..79483aca4e
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamdebeziuminput.adoc
@@ -0,0 +1,136 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Read database change events using Apache Beam DebeziumIO.
+:page-engines: Hop Engine=no, Single Threaded=no, Native Spark=no, Beam Direct=yes, Beam Spark=no, Beam Flink=yes, Beam Dataflow=yes
+:page-marketplace: hop-engines-beam
+
+= image:transforms/icons/beam-debezium-input.svg[Beam Debezium input Icon, role="image-doc-icon"] Beam Debezium input
+
+== Description
+
+Beam Debezium input is a source transform that reads database change data capture (CDC) events using Apache Beam's `DebeziumIO.readAsJson()`.
+It has no incoming hops and emits one Hop String field per event.
+The database connector runs inside the Beam worker, not inside a Kafka Connect server.
+PostgreSQL is included; other connectors and their drivers must be supplied by the user.
+
+== Options
+
+[cols="1,3",options="header"]
+|===
+|Option |Description
+|Transform name |Unique name within the pipeline.
+|Connector |`PostgreSQL` (bundled) or `Custom` (user-supplied Kafka Connect source connector).
+|Custom connector class |Fully qualified `SourceConnector` implementation, used only for Custom. A missing/incompatible class fails pipeline preparation with a classpath diagnostic.
+|Hostname |Database host, for example `'${CDC_HOST}'`.
+|Port |Positive TCP port (1–65535). Default: 5432.
+|Username |Database account with CDC/replication privileges.
+|Password |Use a secret variable such as `'${CDC_PASSWORD}'`. Hop encrypts literal passwords in transform XML using its standard password handling; this is not a secure credential vault.
+|Connector properties (JSON) |JSON object whose values are strings. Configure database, tables, snapshot mode, slot, publication, SSL and other connector-specific options here. Variables are resolved before parsing. Connection-tab settings cannot be duplicated here. This JSON is stored as text, not encrypted: reference variables for any additional secrets.
+|JSON field |Name of the single output String field. Default: `event`.
+|Maximum records |Optional positive integer, at most 2147483647. Stops after this many events. Blank means no record limit; an idle database can wait indefinitely if no time limit is set.
+|Maximum run time (ms) |Optional positive milliseconds, intended for testing. Beam checks this between polling cycles; connection startup or a blocking poll can exceed it. Blank means no time limit.
+|Polling timeout (ms) |Positive polling interval/timeout. Default: 1000. Blank uses Beam's default.
+|===
+
+All text settings support Hop variables, including the connector selection/class and limits.
+
+== PostgreSQL example
+
+. Enable logical replication on PostgreSQL (`wal_level=logical`), and configure replication slots, publications and permissions according to the Debezium PostgreSQL documentation.
+. In the Connection tab choose PostgreSQL, set the hostname, port and replication account, and use a variable for the password.
+. In Connector properties enter, adapting the names to your database:
++
+[source,json]
+----
+{
+ "database.dbname": "${CDC_DATABASE}",
+ "topic.prefix": "hop-cdc",
+ "plugin.name": "pgoutput",
+ "slot.name": "hop_cdc_slot",
+ "publication.name": "hop_cdc_publication",
+ "table.include.list": "public.orders",
+ "snapshot.mode": "initial"
+}
+----
+. For a finite smoke test, set both Maximum records and Maximum run time. For continuous CDC leave both blank.
+. Connect the output to transforms that can handle an unbounded stream. Add windows/triggers when aggregating or writing to a sink that requires windowed input.
+
+Slot and publication names must be unique/managed appropriately for concurrently running pipelines.
+Network access and the database credentials must be available from every worker.
+No Kafka broker is required for the included PostgreSQL connector with `pgoutput`.
+Other connectors may require an external schema-history service, Kafka broker, or additional configuration.
+
+== Event JSON
+
+The output is the JSON supplied by Beam's built-in `SourceRecordJsonMapper`, not the raw Debezium envelope.
+For an insert it has this shape:
+
+[source,json]
+----
+{
+ "metadata": {
+ "connector": "postgresql",
+ "version": "3.1.3.Final",
+ "name": "hop-cdc",
+ "database": "inventory",
+ "schema": "public",
+ "table": "orders"
+ },
+ "before": null,
+ "after": {"fields": {"id": 1}}
+}
+----
+
+Updates contain before/after values when available; deletes have `after: null`.
+A tombstone is represented by null metadata/before/after.
+The mapper does not retain the raw key, `op`, source offset or `ts_ms` in the JSON.
+Beam uses the record timestamp for event time, independently of the JSON field.
+Nested/special database types follow Beam's serializer; validate these against your own schema before production use.
+
+== Runners and termination
+
+DebeziumIO 2.74.0 uses an *unbounded Splittable DoFn*, even when a record or runtime limit is set.
+The transform enables Beam streaming mode on the pipeline automatically.
+Do not use the limits to assume that a batch-only sink or runner is supported.
+
+* Direct: exercised with the real DebeziumIO, a deterministic Kafka Connect fixture source, JSON row conversion, termination limits and Hop input/written metrics. Suitable for local development.
+* Flink and Dataflow: the streaming runners provide the required unbounded Splittable DoFn execution. Deploy with streaming-compatible downstream transforms and worker dependencies. Live database CDC on these runners was not tested for this change.
+* Beam Spark: rejected before submission. Beam 2.74.0's Spark streaming translator explicitly rejects Splittable DoFn; setting a limit does not make this source bounded.
+* Hop local, Single Threaded and Native Spark: unsupported. The native runtime reports that a Beam engine is required instead of silently returning no rows.
+
+== Dependencies and operational limitations
+
+The Beam plugin distributes `beam-sdks-java-io-debezium` 2.74.0, Debezium PostgreSQL connector/core/API 3.1.3.Final, Kafka Connect API/transforms 3.9.0 and the Hop-managed PostgreSQL JDBC driver.
+Beam DebeziumIO itself does *not* include database connectors.
+The bundled dependencies are Apache-2.0, and PostgreSQL JDBC is BSD-2-Clause.
+They are staged by the Beam plugin assembly into `lib/core`, which is included in Hop's worker fat jar.
+
+For Custom, supply a compatible Debezium 3.1.3.Final connector and its complete driver/runtime dependency set to the Hop classpath and all Beam workers; when using Hop's fat-jar builder, additional jars can be placed in `plugins/engines/beam/lib` before building the fat jar.
+MySQL, SQL Server, Oracle, DB2 and their drivers are not bundled or integration-tested by this transform.
+Review the connector/driver license before redistributing it; some databases require vendor-distributed drivers.
+
+This transform does not expose Beam's external offset-retainer/start-offset API.
+Runner checkpoints and database replication slots are not a general cross-run offset backup or exactly-once guarantee; plan restart/snapshot/replay behavior and idempotent sinks explicitly.
+No live CDC database test was performed for this change.
+Beam logs connector properties (including resolved passwords) at DEBUG level: avoid enabling DEBUG logging for `org.apache.beam.io.debezium` in production.
+
+== See also
+
+* link:https://beam.apache.org/releases/javadoc/current/org/apache/beam/io/debezium/DebeziumIO.html[Apache Beam DebeziumIO]
+* link:https://debezium.io/documentation/reference/3.1/connectors/postgresql.html[Debezium PostgreSQL connector]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamelasticsearchinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamelasticsearchinput.adoc
new file mode 100644
index 0000000000..8db8a0dadd
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamelasticsearchinput.adoc
@@ -0,0 +1,55 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Beam Elasticsearch input uses the Beam ElasticsearchIO connector for JSON documents.
+:page-engines: Hop Engine=no, Single Threaded=no, Native Spark=no, Beam Direct=yes, Beam Spark=yes, Beam Flink=yes, Beam Dataflow=yes
+:page-marketplace: hop-engines-beam
+
+= image:transforms/icons/beam-elasticsearch-input.svg[Beam Elasticsearch input icon, role="image-doc-icon"] Beam Elasticsearch input
+
+== Description
+
+A bounded source that reads JSON documents with Beam's Elasticsearch scroll IO. This transform starts a Beam pipeline without incoming rows and provides a single String column to downstream transforms.
+It is not an incremental change feed and does not parse document properties into columns; use downstream JSON transforms for that.
+
+== Options
+
+All String settings support Hop variables, for example `${ES_HOSTS}` and `${ES_PASSWORD}`.
+Variables are resolved while constructing the Beam graph. Empty required settings, unresolved variables and invalid configuration fail before submission; credentials must also be available on workers through the resolved graph.
+
+[options="header"]
+|===
+|Option|Description
+|Transform name|Unique name within the pipeline.
+|Hosts|Required comma-separated HTTP(S) origins, for example `https://first.example:9200,https://second.example:9200`. Paths, URL credentials, query strings and fragments are rejected.
+|Index|Required index. Input can use an index expression understood by Elasticsearch; output needs a writable target index.
+|Document type (legacy)|Optional legacy type. See the version-specific limits below.
+|Username / Password|Optional basic authentication; supply both or neither. Passwords are stored with Hop's password encryption convention; variable expressions remain expressions. This is not a substitute for restricting pipeline and runner access.
+|JSON field|Required name of the single output String field, default `json`. Each row contains the document `_source` JSON object, not the hit metadata such as `_id`.
+|Query JSON|Optional JSON request body, for example `{"query":{"match_all":{}}}`. Blank reads all documents. The body is also sent to the count endpoint by Beam, so it must be accepted by that endpoint. Elasticsearch performs query semantics validation.
+|Scroll keepalive|Positive scroll context duration, default `5m`; units `ms`, `s`, `m`, `h` and `d` are accepted.
+|===
+
+== Reading and permissions
+
+The connector needs access to the root version response, index statistics, count (when a query is supplied), search and scroll APIs, including scroll cleanup.
+Workers must be able to reach the configured hosts. Runner splitting uses index statistics and Elasticsearch's sliced scroll.
+In Beam 2.74.0 the scroll request implementation does not apply the SDK `withBatchSize()` option to the request, so Hop does not expose a misleading read-batch control.
+There is no point-in-time or continuous polling mode in this transform.
+
+include::partial$beam-elasticsearch-compatibility.adoc[]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamelasticsearchoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamelasticsearchoutput.adoc
new file mode 100644
index 0000000000..31fbf980bb
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamelasticsearchoutput.adoc
@@ -0,0 +1,56 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Beam Elasticsearch output uses the Beam ElasticsearchIO connector for JSON documents.
+:page-engines: Hop Engine=no, Single Threaded=no, Native Spark=no, Beam Direct=yes, Beam Spark=yes, Beam Flink=yes, Beam Dataflow=yes
+:page-marketplace: hop-engines-beam
+
+= image:transforms/icons/beam-elasticsearch-output.svg[Beam Elasticsearch output icon, role="image-doc-icon"] Beam Elasticsearch output
+
+== Description
+
+A sink that writes JSON documents with Beam's Elasticsearch bulk IO. It requires incoming rows and has no downstream row output. It is available only in Beam run configurations, not in the local Hop engine.
+The sink does not create mappings or manage index lifecycle. Ensure the target index and privileges are appropriate before running.
+
+== Options
+
+All String settings support Hop variables, for example `${ES_HOSTS}` and `${ES_PASSWORD}`.
+Variables are resolved while constructing the Beam graph. Empty required settings, unresolved variables and invalid configuration fail before submission; credentials must also be available on workers through the resolved graph.
+
+[options="header"]
+|===
+|Option|Description
+|Transform name|Unique name within the pipeline.
+|Hosts|Required comma-separated HTTP(S) origins, for example `https://first.example:9200,https://second.example:9200`. Paths, URL credentials, query strings and fragments are rejected.
+|Index|Required index. Input can use an index expression understood by Elasticsearch; output needs a writable target index.
+|Document type (legacy)|Optional legacy type. See the version-specific limits below.
+|Username / Password|Optional basic authentication; supply both or neither. Passwords are stored with Hop's password encryption convention; variable expressions remain expressions. This is not a substitute for restricting pipeline and runner access.
+|JSON field|Required incoming String field containing one JSON object per row, default `json`. Null, empty, malformed and non-object JSON fail the pipeline. Multiline JSON is compacted to one line for the bulk protocol. Hop lazy String storage metadata is retained for conversion.
+|Maximum documents per batch|Positive maximum bulk batch document count, default `1000`.
+|Maximum batch bytes|Positive bulk batching byte threshold, default `5242880` (5 MiB). Beam may send smaller batches; one oversized document can exceed the threshold.
+|===
+
+== Delivery and errors
+
+Beam generates the bulk index directives; the payload field is the document, not a user-supplied bulk directive.
+Document IDs are generated by Elasticsearch. There is no explicit ID, upsert, delete or exactly-once setting in this transform.
+Runner retries or reruns can create duplicates, so account for that in your pipeline design.
+Bulk item failures fail the pipeline rather than silently discarding records. A failure can occur after other documents have already been persisted; there is no transaction rollback.
+The Hop written metric counts successful bulk responses. Read/output metrics count rows extracted and submitted to Beam IO, not an exactly-once delivery guarantee.
+
+include::partial$beam-elasticsearch-compatibility.adoc[]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileoutput.adoc
index fd0ca3eaaa..61ebb2498a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileoutput.adoc
@@ -37,6 +37,11 @@ The Beam file output transform writes files using a file definition with the Bea
|File suffix|Text which has to succeed the file name.
|Windowed writes?|Writes a file per window.
Use together with the Beam window transform.
+|Compression|How to compress the output files.
+Leave blank for no explicit compression.
+Choose `AUTO` to derive the codec from the file suffix, for example `.gz`.
+The other options are `UNCOMPRESSED`, `GZIP`, `BZIP2`, `ZIP`, `ZSTD`, `LZO`, `LZOP`, `DEFLATE` and `SNAPPY`.
+With `AUTO` and no recognisable file suffix, the output is written uncompressed.
|File definition to use|The file definition to use stored in a metastore.
|Edit|Edits an existing file definition.
|New|Creates a new file definition.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beammqttinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beammqttinput.adoc
new file mode 100644
index 0000000000..567e9e5e8d
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beammqttinput.adoc
@@ -0,0 +1,56 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Beam MQTT input using Apache Beam MqttIO.
+:page-engines: Hop Engine=no, Single Threaded=no, Native Spark=no, Beam Direct=yes, Beam Spark=yes, Beam Flink=yes, Beam Dataflow=yes
+:page-marketplace: hop-engines-beam
+
+= image:transforms/icons/beam-mqtt-input.svg[Beam MQTT input icon, role="image-doc-icon"] Beam MQTT input
+
+== Description
+
+Read MQTT message payloads as Hop rows using Apache Beam MqttIO. This is a source transform: do not connect an incoming hop. The output contains exactly one payload field. Input topics can contain MQTT subscription wildcards (`+` and `#`).
+
+== Options
+
+[options="header"]
+|===
+|Option|Description
+|Transform name|Unique name in the pipeline.
+|Server URI|Required broker URI, `tcp://host:1883` or `ssl://host:8883`. Do not embed credentials in the URI.
+|Topic|Required subscription filter.
+|Client ID prefix|Optional prefix. Beam appends a UUID and truncates the resulting client ID to 23 characters. Leave empty for a random ID; use a short prefix to preserve random suffix characters.
+|Username|Optional broker username.
+|Password|Optional broker password, requires a username. Masked in the dialog and encrypted with Hop password conventions in saved pipelines. Prefer a secret variable such as `${MQTT_PASSWORD}`.
+|Payload field|Name of the output payload field (default `message`).
+|Payload type|`String` or `Binary`. String decodes UTF-8; Binary preserves arbitrary bytes.
+|Maximum records|Non-negative record limit; zero or empty means unlimited.
+|Maximum read time (seconds)|Non-negative read duration; zero or empty means unlimited. A record limit below `Long.MAX_VALUE` or a nonzero time limit makes the source bounded. If both are configured, the source stops when either limit is reached.
+|===
+
+All text options support Hop variables and are resolved before the Beam graph is submitted. Required options and invalid numeric limits fail before connecting.
+
+== Execution and security
+
+This transform requires a Beam pipeline run configuration; the native Hop engine reports an error. The broker must be reachable from the runner's worker machines, not just Hop GUI.
+
+Beam MqttIO uses MQTT QoS 1 (at least once); retries can produce duplicates. The source is unbounded unless a record or time limit is set. It reads payloads only, not the received topic or other message metadata.
+
+Use `ssl://` for credentials or sensitive data. TLS uses the worker JVM's default trust configuration; install the broker's certificate chain in its trust store. Custom trust stores, client certificates, reconnect controls and QoS selection are not exposed by Beam's connection API and are not dialog options. `tcp://` is not encrypted.
+
+Hop password encryption protects saved metadata from plain-text disclosure; it is not a substitute for a secret manager. Beam's connection specification must carry usable credentials to workers: protect graph serialization, worker storage and transport. Do not enable Beam MqttIO debug logging for sensitive payloads (the upstream writer logs message text at DEBUG).
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beammqttoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beammqttoutput.adoc
new file mode 100644
index 0000000000..127f33d8df
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beammqttoutput.adoc
@@ -0,0 +1,55 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Beam MQTT output using Apache Beam MqttIO.
+:page-engines: Hop Engine=no, Single Threaded=no, Native Spark=no, Beam Direct=yes, Beam Spark=yes, Beam Flink=yes, Beam Dataflow=yes
+:page-marketplace: hop-engines-beam
+
+= image:transforms/icons/beam-mqtt-output.svg[Beam MQTT output icon, role="image-doc-icon"] Beam MQTT output
+
+== Description
+
+Publish one MQTT message for each incoming Hop row using Apache Beam MqttIO. Connect exactly one incoming transform. This is a terminal sink; it does not produce downstream rows. The configured topic must not contain subscription wildcards.
+
+== Options
+
+[options="header"]
+|===
+|Option|Description
+|Transform name|Unique name in the pipeline.
+|Server URI|Required broker URI, `tcp://host:1883` or `ssl://host:8883`. Do not embed credentials in the URI.
+|Topic|Required fixed publish topic.
+|Client ID prefix|Optional prefix. Beam appends a UUID and truncates the resulting client ID to 23 characters. Leave empty for a random ID; use a short prefix to preserve random suffix characters.
+|Username|Optional broker username.
+|Password|Optional broker password, requires a username. Masked in the dialog and encrypted with Hop password conventions in saved pipelines. Prefer a secret variable such as `${MQTT_PASSWORD}`.
+|Payload field|Required incoming field to publish (default `message`).
+|Payload type|`String` or `Binary`. String uses Hop value metadata to convert to text, then encodes UTF-8. Binary requires a Hop Binary field and preserves the bytes. Null payloads fail the pipeline; empty payloads are allowed.
+|Retain messages|Sets the MQTT retained flag. Default is off. A retained publish replaces the broker's retained message; a retained empty payload clears it.
+|===
+
+All text options support Hop variables and are resolved before the Beam graph is submitted. Required options and invalid numeric limits fail before connecting.
+
+== Execution and security
+
+This transform requires a Beam pipeline run configuration; the native Hop engine reports an error. The broker must be reachable from the runner's worker machines, not just Hop GUI.
+
+Beam MqttIO 2.74.0 publishes with MQTT QoS 1 (at least once) and waits for the broker PUBACK inside its writer. That wait is not a Hop callback, so this transform does not expose an acknowledged-write counter. Hop's lines-read and lines-output counts are payload conversion: a row was read, or its field was turned into bytes and handed to MqttIO. Lines written stays empty. A converted payload is not a confirmed publish. QoS and reconnect settings are not dialog options. The MQTT client re [...]
+
+Use `ssl://` for credentials or sensitive data. TLS uses the worker JVM's default trust configuration; install the broker's certificate chain in its trust store. Custom trust stores, client certificates, reconnect controls and QoS selection are not exposed by Beam's connection API and are not dialog options. `tcp://` is not encrypted.
+
+Hop password encryption protects saved metadata from plain-text disclosure; it is not a substitute for a secret manager. Beam's connection specification must carry usable credentials to workers: protect graph serialization, worker storage and transport. Do not enable Beam MqttIO debug logging for sensitive payloads (the upstream writer logs message text at DEBUG).
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beampartition.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beampartition.adoc
new file mode 100644
index 0000000000..ea5ede1b38
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beampartition.adoc
@@ -0,0 +1,85 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: The Beam partition transform assigns each row to a partition, then flattens the partitions back into one stream.
+:page-engines: Hop Engine=no, Single Threaded=no, Native Spark=no, Beam Spark=yes, Beam Flink=yes, Beam Dataflow=yes
+:page-marketplace: hop-engines-beam
+
+= image:transforms/icons/beam-partition.svg[Beam partition Icon, role="image-doc-icon"] Beam partition
+
+== Description
+
+The Beam partition transform decides, before the following transforms run, which partition each row
+belongs to.
+
+This is the counterpart of the partitioning option in the local Hop pipeline engine, and it wraps the
+https://beam.apache.org/documentation/transforms/java/elementwise/partition/[Beam Partition transform].
+
+Partitioning does not change the rows themselves: the same rows with the same columns come out, in
+a possibly different order.
+
+=== What partitioning is for
+
+On a distributed runner, a transform that groups rows, such as a Beam window on a key, shuffles so
+that rows sharing a key end up in the same place.
+
+This transform does not remove that shuffle. It assigns each row to a partition and then flattens
+the partitions back into one stream. A following GroupByKey still shuffles, whether or not it
+groups on the same field.
+
+=== Choosing a partitioning type
+
+Single partition sends every row to one partition.
+It does not serialise the processing that follows. After this transform the rows are one stream
+again, and the runner schedules the rest of the pipeline as it would without this transform.
+It is the default, which means a pipeline saved without a partitioning type does not partition.
+
+Keyed partitions on the value of a field: rows with the same value are sent to the same partition,
+then flattened back into that one stream with every other partition.
+
+NOTE: The partition is chosen by hashing the key, so rows are spread over the partitions in no
+particular order and not evenly. Two keys can share a partition, and some partitions can be
+heavier than others.
+
+== Options
+
+[options="header"]
+|===
+|Option|Description
+|Transform name|Name of the transform, this name has to be unique in a single pipeline.
+|Partitioning type a|
+* Single partition
+* Keyed
+|Key field|Required for the keyed type: the field to partition on. Ignored for the single-partition type.
+|Number of partitions|How many partitions to create. Defaults to 1. Only used for the keyed type.
+|===
+
+NOTE: A key field that is not in the input fails the pipeline with a message naming it, and keyed
+partitioning without a key field is refused too. Neither is silently ignored.
+
+== Notes on the Beam engines
+
+The number of partitions is decided when the pipeline is built.
+Changing it therefore changes the size of the resulting Beam graph, which matters on runners with a
+large number of workers.
+
+After partitioning, Beam hands back one collection per partition.
+Hop recombines them into a single stream before the next transform, because the partition of a given
+row is only decided while the pipeline runs and the next transform cannot branch on it.
+That means partitioning on its own does not filter or split the stream, and the partition does not
+stick for the transform that follows.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamsnowflakeinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamsnowflakeinput.adoc
new file mode 100644
index 0000000000..69237ffa52
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamsnowflakeinput.adoc
@@ -0,0 +1,70 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Beam Snowflake input using a bounded GCS COPY.
+:page-engines: Hop Engine=no, Single Threaded=no, Native Spark=no, Beam Direct=yes, Beam Spark=yes, Beam Flink=yes, Beam Dataflow=yes
+:page-marketplace: hop-engines-beam
+
+= image:transforms/icons/beam-snowflake-input.svg[Beam Snowflake input icon, role="image-doc-icon"] Beam Snowflake input
+
+== Description
+
+Read rows from Snowflake with Apache Beam SnowflakeIO 2.74.0. Snowflake copies the table or query result to a GCS external stage, and Hop turns each CSV row into a Hop row. This transform is a source: it does not accept incoming hops.
+
+The read is bounded. SnowflakeIO rejects a streaming read, and this transform does not offer one. Declared Beam compatibility is not a remote-runner certification. No test in this release runs a pipeline against Snowflake or GCS.
+
+== Options
+
+[options="header"]
+|===
+|Option|Description
+|Transform name|Unique name in the pipeline.
+|Server|Required account host. It must look like `account.snowflakecomputing.com`. The value is not repeated in an error.
+|Username|Required Snowflake user.
+|Password|Used when no private key is set. Masked in the dialog and encrypted with Hop password conventions. Prefer a variable such as `${SNOWFLAKE_PASSWORD}`.
+|Private key|PEM private key. When this is set, Hop uses key-pair authentication and does not send the password. A passphrase is optional. Key files and OAuth are not offered.
+|Database|Optional database identifier.
+|Warehouse|Optional warehouse identifier.
+|Schema|Optional schema identifier.
+|Role|Optional role identifier.
+|Port|Optional port from 1 to 65535. Empty uses the driver default.
+|Staging bucket|Required `gs://` path ending with `/`. S3, Azure, and internal stages are not accepted.
+|Storage integration|Required Snowflake storage integration that can read the bucket.
+|Quotation mark|Optional single CSV quotation character. Empty uses the Beam default apostrophe. A backslash or whitespace is rejected.
+|Table|Table name, `schema.name`, or `database.schema.name`. Set a table or a query, not both.
+|Query|SQL sent to Snowflake as written. Set a query or a table, not both.
+|Fields|Output fields. Name must be a Snowflake identifier. Type is String, Integer, Number, BigNumber, Boolean, Date, Timestamp, or Binary.
+|===
+
+== Field values
+
+An empty CSV field becomes null, including for strings, so an empty string is not preserved. Integer accepts a whole number such as `1.0` and rejects a fraction. Number is an IEEE double. BigNumber keeps the decimal text. Boolean accepts `true`, `false`, `TRUE`, `FALSE`, `1`, and `0`. Date and Timestamp use UTC `yyyy-MM-dd HH:mm:ss.SSS`, and a date-only `yyyy-MM-dd` is midnight UTC. Binary is lowercase hex. A conversion error names the field and does not include the cell text.
+
+== Execution and security
+
+Hop's lines-read and lines-output counts are conversion: a CSV row was read, or it became a Hop row. Lines written stays empty. A converted row is not a confirmed COPY. The transform does not add a TLS switch. The Snowflake JDBC driver negotiates the connection.
+
+Passwords, private keys, passphrases, bucket names, and queries are not copied into Hop error text.
+
+== Limitations
+
+* GCS external stage only. No S3, Azure, or internal stage.
+* No streaming read.
+* The query is user SQL and is sent as written.
+* There is no controlled Snowflake or GCS round trip in the tests. Remote runners have not been certified.
+* A second connector review has not covered this transform.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamsnowflakeoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamsnowflakeoutput.adoc
new file mode 100644
index 0000000000..87ffcee5cb
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamsnowflakeoutput.adoc
@@ -0,0 +1,68 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Beam Snowflake output using a bounded GCS COPY.
+:page-engines: Hop Engine=no, Single Threaded=no, Native Spark=no, Beam Direct=yes, Beam Spark=yes, Beam Flink=yes, Beam Dataflow=yes
+:page-marketplace: hop-engines-beam
+
+= image:transforms/icons/beam-snowflake-output.svg[Beam Snowflake output icon, role="image-doc-icon"] Beam Snowflake output
+
+== Description
+
+Write one bounded Hop collection to a Snowflake table with Apache Beam SnowflakeIO 2.74.0. Hop formats each row as CSV values, SnowflakeIO stages those files in GCS, and Snowflake copies them into the table. Connect exactly one incoming transform. This is a terminal sink; it does not produce downstream rows.
+
+Snowpipe is not used. An unbounded collection is rejected. Declared Beam compatibility is not a remote-runner certification. No test in this release runs a pipeline against Snowflake or GCS.
+
+== Options
+
+[options="header"]
+|===
+|Option|Description
+|Transform name|Unique name in the pipeline.
+|Server|Required account host. It must look like `account.snowflakecomputing.com`. The value is not repeated in an error.
+|Username|Required Snowflake user.
+|Password|Used when no private key is set. Masked in the dialog and encrypted with Hop password conventions. Prefer a variable such as `${SNOWFLAKE_PASSWORD}`.
+|Private key|PEM private key. When this is set, Hop uses key-pair authentication and does not send the password. A passphrase is optional. Key files and OAuth are not offered.
+|Database|Optional database identifier.
+|Warehouse|Optional warehouse identifier.
+|Schema|Optional schema identifier.
+|Role|Optional role identifier.
+|Port|Optional port from 1 to 65535. Empty uses the driver default.
+|Staging bucket|Required `gs://` path ending with `/`. S3, Azure, and internal stages are not accepted.
+|Storage integration|Required Snowflake storage integration that can write the bucket.
+|Quotation mark|Optional single CSV quotation character. Empty uses the Beam default apostrophe. A backslash or whitespace is rejected.
+|Table|Required destination, as name, `schema.name`, or `database.schema.name`.
+|Write disposition|`APPEND` (default), `TRUNCATE`, or `EMPTY`.
+|Create disposition|`CREATE_NEVER` (default) or `CREATE_IF_NEEDED`. Incoming fields are checked either way. A Snowflake column list is sent only when the table may be created. Columns are nullable.
+|===
+
+The column types follow the incoming row. String becomes VARCHAR (with its length when set). Integer becomes NUMBER(38,0), which is the SQL Beam emits for an integer. Number becomes FLOAT, which is an IEEE double and is not a lossless decimal. BigNumber becomes NUMBER, using the Hop length as precision and the Hop precision as scale when the length is set. Boolean becomes BOOLEAN. Date and Timestamp become TIMESTAMP_NTZ in UTC `yyyy-MM-dd HH:mm:ss.SSS`. Binary becomes BINARY hex. Other H [...]
+
+== Execution and security
+
+Hop's lines-read and lines-output counts are conversion: a row was read, or it was formatted for SnowflakeIO. Lines written stays empty. A formatted row is not a confirmed COPY. The transform does not add a TLS switch. The Snowflake JDBC driver negotiates the connection.
+
+Passwords, private keys, passphrases, bucket names, and cell values are not copied into Hop error text. A conversion error names the field only.
+
+== Limitations
+
+* GCS external stage only. No S3, Azure, or internal stage.
+* No Snowpipe and no streaming write. The input collection must be bounded.
+* Number values can lose precision. Use BigNumber for a decimal that must round-trip as text.
+* There is no controlled Snowflake or GCS round trip in the tests. Remote runners have not been certified.
+* A second connector review has not covered this transform.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamsplunkoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamsplunkoutput.adoc
new file mode 100644
index 0000000000..bb2bce7f70
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamsplunkoutput.adoc
@@ -0,0 +1,63 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Beam Splunk output using Apache Beam SplunkIO.
+:page-engines: Hop Engine=no, Single Threaded=no, Native Spark=no, Beam Direct=yes, Beam Spark=yes, Beam Flink=yes, Beam Dataflow=yes
+:page-marketplace: hop-engines-beam
+
+= image:transforms/icons/beam-splunk-output.svg[Beam Splunk output icon, role="image-doc-icon"] Beam Splunk output
+
+== Description
+
+Publish one Splunk HTTP Event Collector (HEC) event for each incoming Hop row using Apache Beam SplunkIO 2.74.0. Connect exactly one incoming transform. This is a terminal sink; it does not produce downstream rows. There is no Splunk input transform: SplunkIO is write-only.
+
+The HEC token is sent as `Authorization: Splunk <token>`. The request URL is the configured base URL plus `/services/collector/event`. Declared Beam compatibility is not a remote-runner certification.
+
+== Options
+
+[options="header"]
+|===
+|Option|Description
+|Transform name|Unique name in the pipeline.
+|HEC URL|Required base URL, `http://host:8088` or `https://host:8088`. No path, user info or query.
+|HEC token|Required token. Masked in the dialog and encrypted with Hop password conventions in saved pipelines. Prefer a secret variable such as `${SPLUNK_HEC_TOKEN}`.
+|Batch count|Optional events per HTTP request. Empty uses the Beam default of 10. A partial batch is flushed on a 2 second timer.
+|Root CA certificate|Optional PEM used while certificate validation stays enabled. A `gs://` path needs the Beam GCS filesystem on the worker.
+|Disable certificate validation|Unchecked by default. HTTPS validates the server certificate and hostname unless this is enabled.
+|Gzip HTTP body|Checked by default, matching SplunkIO. The request body is gzip-compressed.
+|Event field|Required incoming field. Its string value is the HEC `event` body. A null value fails the pipeline.
+|Index|Optional index copied onto every event.
+|Source|Optional source copied onto every event.
+|Source type|Optional source type copied onto every event.
+|Host|Optional host copied onto every event.
+|===
+
+== Execution and security
+
+Hop's lines-read and lines-output counts are conversion: a row was read, or it was turned into a `SplunkEvent` and handed to SplunkIO. Lines written stays empty. A converted event is not a confirmed HEC write. SplunkIO keeps its own HTTP counters inside Beam; this transform does not copy them onto a Hop acknowledged-write counter.
+
+Client errors (HTTP 400 and above, below 500) are not retried. SplunkIO emits them as write errors, and this transform fails the pipeline. Hop's failure text contains the HTTP status only, not the event body or the token. A write that fails without an HTTP status says so, and still omits the event body and the token. Server errors (HTTP 500 and above) are retried by the Google HTTP client for up to 15 minutes (900000 ms) using its default exponential backoff. That retry limit is not a di [...]
+
+HTTPS validates certificates unless *Disable certificate validation* is checked. Leave that unchecked outside a private test.
+
+== Limitations
+
+* Write-only. SplunkIO 2.74.0 has no read transform.
+* The URL accepts `http` and `https` with an optional port. It does not accept a path or embedded credentials.
+* Index, source, source type and host are constants for the transform, not per-row fields.
+* Loopback HEC tests cover a successful post and a single HTTP 400 rejection. A production Splunk server and remote runners have not been certified.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamwindow.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamwindow.adoc
index ddff2f943f..d4257cae8f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamwindow.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamwindow.adoc
@@ -43,8 +43,11 @@ The Beam window transform adds event-time-based window functions using the Beam
|Window start field|The field containing the window start time.
|Window end field|The field containing the window end time.
|Window max field|The field containing the max duration between events.
+|Key field|Optional: the field to window on. Rows are grouped per value of this field, and each group gets its own window. Leave blank to window across the whole stream. The key field is used for grouping only and is not added to the output.
|===
+NOTE: When a key field is given and it does not exist in the input, the pipeline fails with a message naming the missing field rather than silently windowing everything together.
+
== Window Types
=== Fixed
diff --git a/docs/hop-user-manual/modules/ROOT/partials/beam-elasticsearch-compatibility.adoc b/docs/hop-user-manual/modules/ROOT/partials/beam-elasticsearch-compatibility.adoc
new file mode 100644
index 0000000000..a83ae9ddc3
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/partials/beam-elasticsearch-compatibility.adoc
@@ -0,0 +1,31 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+== Elasticsearch and OpenSearch compatibility
+
+These transforms use Apache Beam **2.74.0** `ElasticsearchIO` and its transitive Elasticsearch low-level REST client **7.17.22**.
+The link:https://github.com/apache/beam/blob/v2.74.0/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java[Beam implementation] discovers the backend version from `GET /` and accepts Elasticsearch major versions **5, 6, 7, 8 and 9**; support for 5 and 6 is deprecated upstream.
+This is Beam's version gate, not a certification of every server version, distribution, security configuration or runner.
+
+* For Elasticsearch 5/6 output, supply the legacy document type required by your index. Input may use a type to restrict the search.
+* For Elasticsearch 7 and later, leave document type blank for typeless endpoints; typed paths are deprecated in 7 and removed in 8 and later.
+* OpenSearch is **not a supported native backend** for this connector. Native OpenSearch version responses with major versions 1, 2 or 3 are rejected by Beam's version detection before reading documents.
+* A server or proxy that reports an Elasticsearch-compatible version can bypass that check, but this does **not** establish OpenSearch compatibility. Scroll, index statistics, bulk request/response formats, privileges and authentication must be verified for that exact deployment. Hop does not override or spoof the backend version.
+* Only basic username/password authentication is exposed. AWS SigV4 signing, API keys, bearer tokens, custom HTTP headers and mutual TLS configuration are not provided by these dialogs.
+* HTTPS uses the REST client's normal certificate and hostname verification. There is no trust-all switch. Supply the necessary trusted CA through the worker JVM trust configuration on **every worker**.
+
+The automated tests exercise the real Beam Direct runner and ElasticsearchIO against an in-process HTTP protocol fixture, including version discovery, sliced scroll requests, bulk payloads and item errors.
+They do not use a live Elasticsearch/OpenSearch cluster and do not certify those products or remote runners.
diff --git a/plugins/engines/beam/pom.xml b/plugins/engines/beam/pom.xml
index 902ee473ab..dd34394acf 100644
--- a/plugins/engines/beam/pom.xml
+++ b/plugins/engines/beam/pom.xml
@@ -41,6 +41,13 @@
<type>pom</type>
<scope>import</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hop</groupId>
+ <artifactId>hop-libs-jdbc</artifactId>
+ <version>${project.version}</version>
+ <type>pom</type>
+ <scope>import</scope>
+ </dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-sdk-bom</artifactId>
@@ -298,6 +305,12 @@
</exclusion>
</exclusions>
</dependency>
+ <!-- DebeziumIO provides the CDC source but not a database connector. -->
+ <dependency>
+ <groupId>io.debezium</groupId>
+ <artifactId>debezium-connector-postgres</artifactId>
+ <version>3.1.3.Final</version>
+ </dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-direct-java</artifactId>
@@ -822,6 +835,14 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-sdks-java-io-debezium</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-sdks-java-io-elasticsearch</artifactId>
+ </dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
@@ -1140,6 +1161,45 @@
</exclusion>
</exclusions>
</dependency>
+ <!-- Beam-native IO transforms (#2375, #2409, #2613).
+ Versions are managed by the Beam BOM; the assembly ships these and their
+ runtime dependencies in lib/core. -->
+ <dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-sdks-java-io-mqtt</artifactId>
+ </dependency>
+ <!-- #2378: Snowflake COPY through GCS. JDBC version comes from hop-libs-jdbc (4.3.0),
+ not Beam's 4.0.2. Snowpipe's ingest SDK is omitted because this transform does not
+ call it. -->
+ <dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-sdks-java-io-snowflake</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.snowflake</groupId>
+ <artifactId>snowflake-ingest-sdk</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.bouncycastle</groupId>
+ <artifactId>bcprov-jdk15on</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <!-- #2377: Splunk HEC. Version is managed by the Beam BOM. No Splunk Java SDK. -->
+ <dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-sdks-java-io-splunk</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-synthetic</artifactId>
@@ -1484,6 +1544,13 @@
</exclusion>
</exclusions>
</dependency>
+ <!-- Debezium schema utilities require Connect transforms, which its connector
+ declares provided. Match DebeziumIO's Connect API version. -->
+ <dependency>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>connect-transforms</artifactId>
+ <version>3.9.0</version>
+ </dependency>
<!--
Spark core/streaming are marked provided by beam-runners-spark-3. Hop client-mode
submit (BeamSparkPipelineEngine → spark://master) needs them on the driver classpath
@@ -1587,6 +1654,12 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.hop</groupId>
+ <artifactId>hop-transform-addsequence</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
<dependency>
<groupId>org.apache.hop</groupId>
<artifactId>hop-transform-constant</artifactId>
@@ -1680,10 +1753,55 @@
</exclusion>
</exclusions>
</dependency>
+
+ <!-- ParquetIO is only referenced by ParquetIOViabilityTest. Test scope keeps
+ the Parquet jars out of the Beam assembly. -->
+ <dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-sdks-java-io-parquet</artifactId>
+ <scope>test</scope>
+ <exclusions>
+ <!-- Transitive and unused here, and it conflicts with the jackson version this
+ plugin already manages. -->
+ <exclusion>
+ <groupId>com.google.j2objc</groupId>
+ <artifactId>j2objc-annotations</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
</dependencies>
<build>
<plugins>
+ <plugin>
+ <artifactId>maven-clean-plugin</artifactId>
+ <configuration>
+ <filesets>
+ <fileset>
+ <directory>${project.basedir}/lib</directory>
+ </fileset>
+ </filesets>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>stage-confluent-serializers</id>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <phase>generate-resources</phase>
+ <configuration>
+ <includeGroupIds>io.confluent</includeGroupIds>
+ <outputDirectory>${project.basedir}/lib</outputDirectory>
+ <overWriteReleases>true</overWriteReleases>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
diff --git a/plugins/engines/beam/src/assembly/assembly.xml b/plugins/engines/beam/src/assembly/assembly.xml
index 4aa190324f..7c4d140093 100644
--- a/plugins/engines/beam/src/assembly/assembly.xml
+++ b/plugins/engines/beam/src/assembly/assembly.xml
@@ -41,6 +41,10 @@
<directory>${project.basedir}/src/main/samples</directory>
<outputDirectory>config/projects/samples/</outputDirectory>
</fileSet>
+ <fileSet>
+ <directory>${project.basedir}/lib</directory>
+ <outputDirectory>plugins/engines/beam/lib</outputDirectory>
+ </fileSet>
</fileSets>
<dependencySets>
@@ -83,8 +87,22 @@
<exclude>org.apache.spark:spark-network-shuffle_2.12</exclude>
<exclude>org.apache.spark:spark-unsafe_2.12</exclude>
<exclude>org.apache.spark:spark-tags_2.12</exclude>
+ <!-- snowflake-jdbc embeds com.github.luben.zstd.Zstd. Keep that jar, and the
+ Snowflake IO jar that loads it, off the shared lib/core classpath. -->
+ <exclude>org.apache.beam:beam-sdks-java-io-snowflake</exclude>
+ <exclude>net.snowflake:snowflake-jdbc</exclude>
</excludes>
<outputDirectory>lib/core</outputDirectory>
</dependencySet>
+
+ <dependencySet>
+ <scope>runtime</scope>
+ <useProjectArtifact>false</useProjectArtifact>
+ <includes>
+ <include>org.apache.beam:beam-sdks-java-io-snowflake:jar</include>
+ <include>net.snowflake:snowflake-jdbc:jar</include>
+ </includes>
+ <outputDirectory>plugins/engines/beam/lib</outputDirectory>
+ </dependencySet>
</dependencySets>
</assembly>
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/BeamDefaults.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/BeamDefaults.java
index f7f1c77e5f..da081aa84a 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/BeamDefaults.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/BeamDefaults.java
@@ -31,6 +31,12 @@ public class BeamDefaults {
PUBSUB_MESSAGE_TYPE_STRING, PUBSUB_MESSAGE_TYPE_MESSAGE,
};
+ /** #2040: put every row in one partition, i.e. do not partition at all. */
+ public static final String PARTITION_TYPE_SINGLE = "Single";
+
+ /** #2040: partition on the value of a field, so equal keys end up in the same partition. */
+ public static final String PARTITION_TYPE_KEY = "Key";
+
public static final String WINDOW_TYPE_FIXED = "Fixed";
public static final String WINDOW_TYPE_SLIDING = "Sliding";
public static final String WINDOW_TYPE_SESSION = "Session";
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/coder/HopRowCoder.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/coder/HopRowCoder.java
index ac5244ff8f..cc301d3cd9 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/coder/HopRowCoder.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/coder/HopRowCoder.java
@@ -38,6 +38,8 @@ import org.apache.hop.core.row.IValueMeta;
import org.apache.hop.core.row.value.ValueMetaAvroRecord;
public class HopRowCoder extends AtomicCoder<HopRow> {
+ /** Distinct from {@link IValueMeta#TYPE_INTEGER}, which is a Hop Long. */
+ private static final int INDEX_INTEGER = 1001;
@Override
public void encode(HopRow hopRow, OutputStream outStream) throws IOException {
@@ -121,6 +123,9 @@ public class HopRowCoder extends AtomicCoder<HopRow> {
out.writeLong(lng);
}
break;
+ case INDEX_INTEGER:
+ out.writeInt((Integer) object);
+ break;
case IValueMeta.TYPE_TIMESTAMP:
{
out.writeLong(((Timestamp) object).getTime());
@@ -211,6 +216,9 @@ public class HopRowCoder extends AtomicCoder<HopRow> {
return in.readLong();
}
+ case INDEX_INTEGER:
+ return in.readInt();
+
case IValueMeta.TYPE_TIMESTAMP:
{
Timestamp timestamp = new Timestamp(in.readLong());
@@ -276,6 +284,10 @@ public class HopRowCoder extends AtomicCoder<HopRow> {
if (object instanceof String) {
return IValueMeta.TYPE_STRING;
}
+ if (object instanceof Integer) {
+ // Indexed Hop values store a java.lang.Integer index, not a Hop Long.
+ return INDEX_INTEGER;
+ }
if (object instanceof Long) {
return IValueMeta.TYPE_INTEGER;
}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/coder/SplunkWriteErrorCoder.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/coder/SplunkWriteErrorCoder.java
new file mode 100644
index 0000000000..a20450fabd
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/coder/SplunkWriteErrorCoder.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.coder;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.io.splunk.SplunkWriteError;
+
+/**
+ * Beam's inferred schema coder drops every field of {@link SplunkWriteError}. The class is an
+ * {@code @AutoValue} with {@code @DefaultSchema(AutoValueSchema)}, and that schema only keeps
+ * methods whose names start with {@code get} or {@code is}. {@code statusCode()}, {@code
+ * statusMessage()} and {@code payload()} therefore round-trip as null.
+ */
+public class SplunkWriteErrorCoder extends AtomicCoder<SplunkWriteError> {
+ private static final SplunkWriteErrorCoder INSTANCE = new SplunkWriteErrorCoder();
+ private static final Coder<Integer> STATUS = NullableCoder.of(VarIntCoder.of());
+ private static final Coder<String> TEXT = NullableCoder.of(StringUtf8Coder.of());
+
+ public static SplunkWriteErrorCoder of() {
+ return INSTANCE;
+ }
+
+ private SplunkWriteErrorCoder() {}
+
+ @Override
+ public void encode(SplunkWriteError value, OutputStream outStream) throws IOException {
+ STATUS.encode(value.statusCode(), outStream);
+ TEXT.encode(value.statusMessage(), outStream);
+ TEXT.encode(value.payload(), outStream);
+ }
+
+ @Override
+ public SplunkWriteError decode(InputStream inStream) throws IOException {
+ Integer status = STATUS.decode(inStream);
+ String statusMessage = TEXT.decode(inStream);
+ String payload = TEXT.decode(inStream);
+ SplunkWriteError.Builder builder = SplunkWriteError.newBuilder();
+ if (status != null) {
+ builder.withStatusCode(status);
+ }
+ if (statusMessage != null) {
+ builder.withStatusMessage(statusMessage);
+ }
+ if (payload != null) {
+ builder.withPayload(payload);
+ }
+ return builder.create();
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/AddSequenceFn.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/AddSequenceFn.java
new file mode 100644
index 0000000000..4430c2f25e
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/AddSequenceFn.java
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import java.io.Serial;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
+import org.apache.beam.sdk.state.ValueState;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.values.KV;
+import org.apache.hop.beam.core.BeamHop;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.JsonRowMeta;
+import org.apache.hop.core.row.RowDataUtil;
+import org.apache.hop.pipeline.Pipeline;
+
+/**
+ * Issue #2379: appends a sequence number to every incoming row, with one counter shared by every
+ * worker.
+ *
+ * <p>The counter lives in Beam state rather than in a field, because on a distributed runner each
+ * worker holds its own instance of this class: a plain field would restart at the same value on
+ * every worker and emit duplicates. The handler keys every row on the same {@code null} key, so
+ * there is exactly one state cell for the whole pipeline.
+ *
+ * <p>The input is a {@code KV} because a stateful {@code ParDo} requires one; the key is always
+ * null and carries no information, only the fact that everything belongs to one key.
+ *
+ * <p>A maximum wraps the same way as {@code org.apache.hop.core.Counter}. Once the following value
+ * would pass the maximum, that following value restarts at {@code startAt}. Rows are not dropped.
+ */
+public class AddSequenceFn extends DoFn<KV<Void, HopRow>, HopRow> {
+
+ @Serial private static final long serialVersionUID = 1L;
+
+ private final String transformName;
+ private final String rowMetaJson;
+ private final String valueName;
+ private final long startAt;
+ private final long incrementBy;
+ private final long maxValue;
+
+ // Not final: they are initialised from transformName, which the constructor only assigns in
+ // its body, so a field initialiser would still see null. The repo's other DoFns do the same.
+ private final Counter numErrors = Metrics.counter("main", "BeamAddSequenceErrors");
+ private final Counter inputCounter;
+ private final Counter writtenCounter;
+
+ private transient IRowMeta rowMeta;
+ private transient IValueMeta valueMeta;
+ private transient int fieldIndex;
+
+ /**
+ * The spec for the counter. {@code @StateId} is field-only and the annotated field has to be a
+ * final {@code StateSpec}, which is the form Beam's own javadoc prescribes.
+ */
+ @StateId("counter")
+ private final StateSpec<ValueState<Long>> counterSpec = StateSpecs.value(VarLongCoder.of());
+
+ public AddSequenceFn(
+ String transformName,
+ String rowMetaJson,
+ String valueName,
+ long startAt,
+ long incrementBy,
+ long maxValue) {
+ this.transformName = transformName;
+ this.rowMetaJson = rowMetaJson;
+ this.valueName = valueName;
+ this.startAt = startAt;
+ this.incrementBy = incrementBy;
+ this.maxValue = maxValue;
+ this.inputCounter = Metrics.counter(Pipeline.METRIC_NAME_INPUT, transformName);
+ this.writtenCounter = Metrics.counter(Pipeline.METRIC_NAME_WRITTEN, transformName);
+ }
+
+ @Setup
+ public void setUp() throws HopException {
+ BeamHop.init();
+ // The row meta handed to us already carries the sequence field: the converter built it from
+ // AddSequenceMeta.getFields(). Adding another one here would grow the row to 12 slots while
+ // the output transform only reads 11, which silently drops the value.
+ rowMeta = JsonRowMeta.fromJson(rowMetaJson);
+ fieldIndex = rowMeta.indexOfValue(valueName);
+ if (fieldIndex < 0) {
+ throw new HopException(
+ "The row layout handed to the Add Sequence function has no field '" + valueName + "'");
+ }
+ valueMeta = rowMeta.getValueMeta(fieldIndex);
+ Metrics.counter(Pipeline.METRIC_NAME_INIT, transformName).inc();
+ }
+
+ @ProcessElement
+ public void processElement(ProcessContext context, @StateId("counter") ValueState<Long> counter) {
+ try {
+ inputCounter.inc();
+
+ // Counter.getAndNext returns the current value, then stores the following one, wrapping
+ // that following value back to the start once it passes the maximum.
+ //
+ Long stored = counter.read();
+ long value = stored == null ? startAt : stored;
+ long next = value + incrementBy;
+ if (incrementBy > 0 && maxValue > startAt && next > maxValue) {
+ next = startAt;
+ } else if (incrementBy < 0 && maxValue < startAt && next < maxValue) {
+ next = startAt;
+ }
+ counter.write(next);
+
+ HopRow row = context.element().getValue();
+ Object[] output = RowDataUtil.resizeArray(row.getRow(), rowMeta.size());
+ output[fieldIndex] = value;
+
+ context.output(new HopRow(output));
+ writtenCounter.inc();
+
+ } catch (Exception e) {
+ numErrors.inc();
+ throw new RuntimeException("Error adding a sequence value in " + transformName, e);
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/BQSchemaAndRecordToHopFn.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/BQSchemaAndRecordToHopFn.java
index 4838631712..245d95fafe 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/BQSchemaAndRecordToHopFn.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/BQSchemaAndRecordToHopFn.java
@@ -21,7 +21,9 @@ import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableSchema;
import java.sql.Timestamp;
import java.text.SimpleDateFormat;
+import java.util.Collection;
import java.util.List;
+import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.util.Utf8;
import org.apache.beam.sdk.io.gcp.bigquery.SchemaAndRecord;
@@ -105,7 +107,14 @@ public class BQSchemaAndRecordToHopFn implements SerializableFunction<SchemaAndR
for (int i = 0; i < rowMeta.size(); i++) {
if (valueTypes[i] == 0) {
IValueMeta valueMeta = rowMeta.getValueMeta(i);
- throw new HopRuntimeException("Unable to find field '" + valueMeta.getName() + "'");
+ // #5064: don't throw. A field the table schema doesn't mention (common for a
+ // fromQuery result) or a type we can't derive has to fall back to the Hop type
+ // declared in the transform, not fail the entire read.
+ LOG.warn(
+ "Unable to determine the BigQuery type of field '"
+ + valueMeta.getName()
+ + "', falling back to the Hop type declared in the transform");
+ valueTypes[i] = valueMeta.getType();
}
}
@@ -127,7 +136,17 @@ public class BQSchemaAndRecordToHopFn implements SerializableFunction<SchemaAndR
if (srcData != null) {
switch (valueMeta.getType()) {
case IValueMeta.TYPE_STRING:
- row[index] = srcData.toString();
+ // #5064: a nested RECORD or a REPEATED field arrives as a GenericRecord or a
+ // List, and neither has a useful toString(): a GenericRecord prints an Avro
+ // container wrapper. GenericData.toString() renders the actual field values as
+ // JSON, which is what we hand back. Note this is deliberately NOT Jackson,
+ // which would serialise GenericRecord as its internal bean structure
+ // ({"schema":...,"elementType":...}) instead of its contents.
+ if (srcData instanceof GenericRecord || srcData instanceof Collection) {
+ row[index] = GenericData.get().toString(srcData);
+ } else {
+ row[index] = srcData.toString();
+ }
break;
case IValueMeta.TYPE_INTEGER:
row[index] = (Long) srcData;
@@ -189,6 +208,10 @@ public class BQSchemaAndRecordToHopFn implements SerializableFunction<SchemaAndR
DATE(IValueMeta.TYPE_DATE),
TIME(IValueMeta.TYPE_DATE),
DATETIME(IValueMeta.TYPE_DATE),
+ // #5064: BigQuery nested and repeated fields have no native Hop type. Hand them back as
+ // JSON strings rather than failing the whole read, which is what valueOf() used to do.
+ RECORD(IValueMeta.TYPE_STRING),
+ STRUCT(IValueMeta.TYPE_STRING),
;
private int hopType;
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/ElasticsearchJsonFn.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/ElasticsearchJsonFn.java
new file mode 100644
index 0000000000..188d97881f
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/ElasticsearchJsonFn.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.core.fn;
+
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.hop.beam.core.BeamHop;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.xml.XmlHandler;
+import org.apache.hop.pipeline.Pipeline;
+
+/** Extracts the configured String field using Hop's value conversion. */
+public class ElasticsearchJsonFn extends DoFn<HopRow, String> {
+ private final String transformName;
+ private final String jsonField;
+ private final String rowMetaXml;
+ private transient IRowMeta rowMeta;
+ private transient int fieldIndex;
+
+ public ElasticsearchJsonFn(String transformName, String jsonField, String rowMetaXml) {
+ this.transformName = transformName;
+ this.jsonField = jsonField;
+ this.rowMetaXml = rowMetaXml;
+ }
+
+ @Setup
+ public void setup() throws Exception {
+ BeamHop.init();
+ // Unlike JsonRowMeta, XML retains lazy storage metadata and its character encoding.
+ rowMeta =
+ new RowMeta(
+ XmlHandler.getSubNode(XmlHandler.loadXmlString(rowMetaXml), RowMeta.XML_META_TAG));
+ fieldIndex = rowMeta.indexOfValue(jsonField);
+ if (fieldIndex < 0 || !rowMeta.getValueMeta(fieldIndex).isString()) {
+ throw new HopException("JSON field must exist and have type String: " + jsonField);
+ }
+ Metrics.counter(Pipeline.METRIC_NAME_INIT, transformName).inc();
+ }
+
+ public String extractJson(HopRow row) throws HopException {
+ if (row == null || row.getRow() == null || fieldIndex >= row.getRow().length) {
+ throw new HopException("Missing JSON value in field: " + jsonField);
+ }
+ String value;
+ try {
+ value = rowMeta.getString(row.getRow(), fieldIndex);
+ } catch (Exception e) {
+ throw new HopException("Unable to convert JSON field: " + jsonField);
+ }
+ if (value == null || value.isBlank()) {
+ throw new HopException("JSON value is required in field: " + jsonField);
+ }
+ return org.apache.hop.beam.transforms.elasticsearch.BeamElasticsearchConfig.compactJsonObject(
+ value, "JSON field " + jsonField);
+ }
+
+ @ProcessElement
+ public void processElement(@Element HopRow row, OutputReceiver<String> output) throws Exception {
+ Metrics.counter(Pipeline.METRIC_NAME_READ, transformName).inc();
+ try {
+ output.output(extractJson(row));
+ } catch (Exception e) {
+ Metrics.counter(Pipeline.METRIC_NAME_ERROR, transformName).inc();
+ throw e;
+ }
+ Metrics.counter(Pipeline.METRIC_NAME_OUTPUT, transformName).inc();
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopKeyFn.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopKeyFn.java
new file mode 100644
index 0000000000..56d8d28596
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopKeyFn.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.hop.beam.core.BeamHop;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.JsonRowMeta;
+
+/**
+ * Issue #2275: keys a HopRow on one of its fields so Beam can group it.
+ *
+ * <p>A {@link SerializableFunction} rather than a DoFn because that is what {@code
+ * WithKeys.of(...)} takes. The row passes through unchanged; only the key is added, so no extra
+ * column reaches the output.
+ */
+public class HopKeyFn implements SerializableFunction<HopRow, String> {
+
+ private final String transformName;
+ private final String rowMetaJson;
+ private final int keyIndex;
+
+ private transient IRowMeta rowMeta;
+ private transient IValueMeta keyValueMeta;
+
+ public HopKeyFn(String transformName, String rowMetaJson, int keyIndex) {
+ this.transformName = transformName;
+ this.rowMetaJson = rowMetaJson;
+ this.keyIndex = keyIndex;
+ }
+
+ @Override
+ public String apply(HopRow row) {
+ try {
+ if (rowMeta == null) {
+ // The Direct runner may call this before any @Setup hook could run.
+ BeamHop.init();
+ rowMeta = JsonRowMeta.fromJson(rowMetaJson);
+ keyValueMeta = rowMeta.getValueMeta(keyIndex);
+ }
+
+ Object raw = keyValueMeta.getString(row.getRow()[keyIndex]);
+ // A null key would collapse every such row onto Beam's null key, so normalise it to the
+ // empty string: rows without a key still window together rather than failing.
+ return raw == null ? "" : raw.toString();
+
+ } catch (Exception e) {
+ throw new RuntimeException("Error keying rows in " + transformName, e);
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopToMqttFn.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopToMqttFn.java
new file mode 100644
index 0000000000..4d09d0e2ea
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopToMqttFn.java
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import java.nio.charset.StandardCharsets;
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.hop.beam.core.BeamHop;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.core.exception.HopRuntimeException;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.xml.XmlHandler;
+import org.apache.hop.pipeline.Pipeline;
+
+/** Converts the selected Hop value to MQTT bytes. Null payloads fail, never silently disappear. */
+@RequiredArgsConstructor
+public class HopToMqttFn extends DoFn<HopRow, byte[]> {
+ private final String transformName;
+ private final String payloadField;
+ private final String payloadType;
+ private final String rowMetaXml;
+ private transient IRowMeta rowMeta;
+ private transient int fieldIndex;
+ private transient Counter readCounter;
+ private transient Counter outputCounter;
+ private transient Counter errorCounter;
+
+ @Setup
+ public void setup() {
+ readCounter = Metrics.counter(Pipeline.METRIC_NAME_READ, transformName);
+ outputCounter = Metrics.counter(Pipeline.METRIC_NAME_OUTPUT, transformName);
+ errorCounter = Metrics.counter(Pipeline.METRIC_NAME_ERROR, transformName);
+ try {
+ BeamHop.init();
+ // JsonRowMeta drops storage type, encoding and indexed values. XML keeps them.
+ rowMeta =
+ new RowMeta(
+ XmlHandler.getSubNode(XmlHandler.loadXmlString(rowMetaXml), RowMeta.XML_META_TAG));
+ fieldIndex = rowMeta.indexOfValue(payloadField);
+ if (fieldIndex < 0)
+ throw new HopRuntimeException("MQTT payload field not found: " + payloadField);
+ if ("Binary".equalsIgnoreCase(payloadType)
+ && rowMeta.getValueMeta(fieldIndex).getType() != IValueMeta.TYPE_BINARY)
+ throw new HopRuntimeException(
+ "MQTT binary payload requires a Binary field: " + payloadField);
+ Metrics.counter(Pipeline.METRIC_NAME_INIT, transformName).inc();
+ } catch (Exception e) {
+ errorCounter.inc();
+ throw new HopRuntimeException("Unable to initialize MQTT payload field: " + payloadField, e);
+ }
+ }
+
+ @ProcessElement
+ public void process(@Element HopRow row, OutputReceiver<byte[]> output) {
+ readCounter.inc();
+ try {
+ Object value = row.getRow()[fieldIndex];
+ if (value == null) throw new HopRuntimeException("MQTT payload cannot be null");
+ byte[] bytes =
+ "Binary".equalsIgnoreCase(payloadType)
+ ? rowMeta.getValueMeta(fieldIndex).getBinary(value)
+ : rowMeta.getString(row.getRow(), fieldIndex).getBytes(StandardCharsets.UTF_8);
+ output.output(bytes);
+ // Payload bytes handed to MqttIO. The QoS 1 PUBACK happens later inside Beam and is not
+ // counted.
+ outputCounter.inc();
+ } catch (Exception e) {
+ errorCounter.inc();
+ throw new HopRuntimeException("Unable to convert MQTT payload field: " + payloadField, e);
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopToSnowflakeRow.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopToSnowflakeRow.java
new file mode 100644
index 0000000000..356305193a
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopToSnowflakeRow.java
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.io.snowflake.SnowflakeIO;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.hop.beam.core.BeamHop;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.exception.HopRuntimeException;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.xml.XmlHandler;
+import org.apache.hop.pipeline.Pipeline;
+
+/**
+ * Hop rows become CSV values for SnowflakeIO. lines-read and lines-output count conversion. A
+ * successful COPY is not a Hop callback and is not counted as written.
+ */
+@RequiredArgsConstructor
+public class HopToSnowflakeRow implements SnowflakeIO.UserDataMapper<HopRow> {
+ private final String transformName;
+ private final String rowMetaXml;
+ private transient IRowMeta rowMeta;
+ private transient Counter readCounter;
+ private transient Counter outputCounter;
+ private transient Counter errorCounter;
+
+ @Override
+ public Object[] mapRow(HopRow row) {
+ try {
+ init();
+ } catch (HopException e) {
+ throw new HopRuntimeException(e.getMessage());
+ }
+ readCounter.inc();
+ try {
+ Object[] incoming = row == null ? null : row.getRow();
+ Object[] values = new Object[rowMeta.size()];
+ for (int i = 0; i < rowMeta.size(); i++) {
+ Object value = incoming != null && i < incoming.length ? incoming[i] : null;
+ values[i] = SnowflakeValues.toCsv(rowMeta.getValueMeta(i), value);
+ }
+ outputCounter.inc();
+ return values;
+ } catch (HopException e) {
+ errorCounter.inc();
+ throw new HopRuntimeException(e.getMessage());
+ } catch (RuntimeException e) {
+ errorCounter.inc();
+ throw new HopRuntimeException("Unable to convert Snowflake output row");
+ }
+ }
+
+ private void init() throws HopException {
+ if (rowMeta != null) return;
+ readCounter = Metrics.counter(Pipeline.METRIC_NAME_READ, transformName);
+ outputCounter = Metrics.counter(Pipeline.METRIC_NAME_OUTPUT, transformName);
+ errorCounter = Metrics.counter(Pipeline.METRIC_NAME_ERROR, transformName);
+ try {
+ BeamHop.init();
+ rowMeta =
+ new RowMeta(
+ XmlHandler.getSubNode(XmlHandler.loadXmlString(rowMetaXml), RowMeta.XML_META_TAG));
+ Metrics.counter(Pipeline.METRIC_NAME_INIT, transformName).inc();
+ } catch (Exception e) {
+ errorCounter.inc();
+ throw new HopException("Unable to initialize Snowflake output fields");
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopToSplunkFn.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopToSplunkFn.java
new file mode 100644
index 0000000000..b34aa8ca30
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopToSplunkFn.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.io.splunk.SplunkEvent;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hop.beam.core.BeamHop;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.core.exception.HopRuntimeException;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.xml.XmlHandler;
+import org.apache.hop.pipeline.Pipeline;
+
+/** Converts one Hop field to a Splunk HEC event. Null events fail and are not published. */
+@RequiredArgsConstructor
+public class HopToSplunkFn extends DoFn<HopRow, SplunkEvent> {
+ private final String transformName;
+ private final String eventField;
+ private final String rowMetaXml;
+ private final String host;
+ private final String source;
+ private final String sourceType;
+ private final String index;
+ private transient IRowMeta rowMeta;
+ private transient int fieldIndex;
+ private transient Counter readCounter;
+ private transient Counter outputCounter;
+ private transient Counter errorCounter;
+
+ @Setup
+ public void setup() {
+ readCounter = Metrics.counter(Pipeline.METRIC_NAME_READ, transformName);
+ outputCounter = Metrics.counter(Pipeline.METRIC_NAME_OUTPUT, transformName);
+ errorCounter = Metrics.counter(Pipeline.METRIC_NAME_ERROR, transformName);
+ try {
+ BeamHop.init();
+ rowMeta =
+ new RowMeta(
+ XmlHandler.getSubNode(XmlHandler.loadXmlString(rowMetaXml), RowMeta.XML_META_TAG));
+ fieldIndex = rowMeta.indexOfValue(eventField);
+ if (fieldIndex < 0)
+ throw new HopRuntimeException("Splunk event field not found: " + eventField);
+ Metrics.counter(Pipeline.METRIC_NAME_INIT, transformName).inc();
+ } catch (Exception e) {
+ errorCounter.inc();
+ throw new HopRuntimeException("Unable to initialize Splunk event field: " + eventField, e);
+ }
+ }
+
+ @ProcessElement
+ public void process(@Element HopRow row, OutputReceiver<SplunkEvent> output) {
+ readCounter.inc();
+ try {
+ String event = rowMeta.getString(row.getRow(), fieldIndex);
+ if (event == null) throw new HopRuntimeException("Splunk event cannot be null");
+ SplunkEvent.Builder builder = SplunkEvent.newBuilder().withEvent(event);
+ if (StringUtils.isNotEmpty(host)) builder.withHost(host);
+ if (StringUtils.isNotEmpty(source)) builder.withSource(source);
+ if (StringUtils.isNotEmpty(sourceType)) builder.withSourceType(sourceType);
+ if (StringUtils.isNotEmpty(index)) builder.withIndex(index);
+ output.output(builder.create());
+ // Handed to SplunkIO. HTTP success is not a Hop callback and is not counted as written.
+ outputCounter.inc();
+ } catch (Exception e) {
+ errorCounter.inc();
+ throw new HopRuntimeException("Unable to convert Splunk event field: " + eventField, e);
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/MqttToHopFn.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/MqttToHopFn.java
new file mode 100644
index 0000000000..bce41d61de
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/MqttToHopFn.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import java.nio.charset.StandardCharsets;
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.core.exception.HopRuntimeException;
+import org.apache.hop.pipeline.Pipeline;
+
+/** Converts one MQTT payload to one Hop row, without exposing payloads in logs. */
+@RequiredArgsConstructor
+public class MqttToHopFn extends DoFn<byte[], HopRow> {
+ private final String transformName;
+ private final String payloadType;
+ private transient Counter inputCounter;
+ private transient Counter writtenCounter;
+ private transient Counter errorCounter;
+
+ @Setup
+ public void setup() {
+ inputCounter = Metrics.counter(Pipeline.METRIC_NAME_INPUT, transformName);
+ writtenCounter = Metrics.counter(Pipeline.METRIC_NAME_WRITTEN, transformName);
+ errorCounter = Metrics.counter(Pipeline.METRIC_NAME_ERROR, transformName);
+ Metrics.counter(Pipeline.METRIC_NAME_INIT, transformName).inc();
+ }
+
+ @ProcessElement
+ public void process(@Element byte[] payload, OutputReceiver<HopRow> output) {
+ inputCounter.inc();
+ try {
+ Object value =
+ "Binary".equalsIgnoreCase(payloadType)
+ ? payload
+ : new String(payload, StandardCharsets.UTF_8);
+ output.output(new HopRow(new Object[] {value}));
+ writtenCounter.inc();
+ } catch (Exception e) {
+ errorCounter.inc();
+ throw new HopRuntimeException("Unable to convert MQTT payload", e);
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/SnowflakeCsvToHop.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/SnowflakeCsvToHop.java
new file mode 100644
index 0000000000..c332254d60
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/SnowflakeCsvToHop.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.io.snowflake.SnowflakeIO;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.hop.beam.core.BeamHop;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.xml.XmlHandler;
+import org.apache.hop.pipeline.Pipeline;
+
+/** CSV rows produced by SnowflakeIO become Hop rows. Counters are conversion, not COPY success. */
+@RequiredArgsConstructor
+public class SnowflakeCsvToHop implements SnowflakeIO.CsvMapper<HopRow> {
+ private final String transformName;
+ private final String rowMetaXml;
+ private transient IRowMeta rowMeta;
+ private transient Counter readCounter;
+ private transient Counter outputCounter;
+ private transient Counter errorCounter;
+
+ @Override
+ public HopRow mapRow(String[] parts) throws Exception {
+ init();
+ readCounter.inc();
+ try {
+ if (parts != null && parts.length > rowMeta.size())
+ throw new HopException("Snowflake CSV row has more columns than the field list");
+ Object[] values = new Object[rowMeta.size()];
+ for (int i = 0; i < rowMeta.size(); i++) {
+ String text = parts != null && i < parts.length ? parts[i] : null;
+ values[i] = SnowflakeValues.fromCsv(rowMeta.getValueMeta(i), text);
+ }
+ outputCounter.inc();
+ return new HopRow(values);
+ } catch (HopException e) {
+ errorCounter.inc();
+ throw e;
+ } catch (RuntimeException e) {
+ errorCounter.inc();
+ throw new HopException("Unable to convert Snowflake CSV row");
+ }
+ }
+
+ private void init() throws HopException {
+ if (rowMeta != null) return;
+ readCounter = Metrics.counter(Pipeline.METRIC_NAME_READ, transformName);
+ outputCounter = Metrics.counter(Pipeline.METRIC_NAME_OUTPUT, transformName);
+ errorCounter = Metrics.counter(Pipeline.METRIC_NAME_ERROR, transformName);
+ try {
+ BeamHop.init();
+ rowMeta =
+ new RowMeta(
+ XmlHandler.getSubNode(XmlHandler.loadXmlString(rowMetaXml), RowMeta.XML_META_TAG));
+ Metrics.counter(Pipeline.METRIC_NAME_INIT, transformName).inc();
+ } catch (Exception e) {
+ errorCounter.inc();
+ throw new HopException("Unable to initialize Snowflake input fields");
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/SnowflakeValues.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/SnowflakeValues.java
new file mode 100644
index 0000000000..5c3ca1456a
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/SnowflakeValues.java
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import java.math.BigDecimal;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Date;
+import java.util.HexFormat;
+import java.util.Locale;
+import org.apache.beam.sdk.io.snowflake.data.SnowflakeDataType;
+import org.apache.beam.sdk.io.snowflake.data.datetime.SnowflakeTimestampNTZ;
+import org.apache.beam.sdk.io.snowflake.data.logical.SnowflakeBoolean;
+import org.apache.beam.sdk.io.snowflake.data.numeric.SnowflakeFloat;
+import org.apache.beam.sdk.io.snowflake.data.numeric.SnowflakeInteger;
+import org.apache.beam.sdk.io.snowflake.data.numeric.SnowflakeNumber;
+import org.apache.beam.sdk.io.snowflake.data.text.SnowflakeBinary;
+import org.apache.beam.sdk.io.snowflake.data.text.SnowflakeVarchar;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.value.ValueMetaFactory;
+
+/**
+ * CSV values handed to and from SnowflakeIO. Timestamps are UTC {@code yyyy-MM-dd HH:mm:ss.SSS}. An
+ * empty CSV field is null, including for strings, so an empty string is not preserved. Number
+ * values are IEEE doubles. BigNumber keeps the decimal text.
+ */
+public final class SnowflakeValues {
+ static final DateTimeFormatter TIMESTAMP =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.SSS").withZone(ZoneOffset.UTC);
+ private static final DateTimeFormatter TIMESTAMP_LOCAL =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.SSS");
+
+ private SnowflakeValues() {}
+
+ public static SnowflakeDataType snowflakeType(IValueMeta meta) throws HopException {
+ return switch (meta.getType()) {
+ case IValueMeta.TYPE_STRING ->
+ meta.getLength() > 0 ? SnowflakeVarchar.of(meta.getLength()) : SnowflakeVarchar.of();
+ case IValueMeta.TYPE_INTEGER -> SnowflakeInteger.of();
+ case IValueMeta.TYPE_NUMBER -> SnowflakeFloat.of();
+ case IValueMeta.TYPE_BIGNUMBER -> {
+ // Hop stores the decimal precision in length and the scale in precision.
+ int precision = meta.getLength();
+ int scale = Math.max(0, meta.getPrecision());
+ yield precision > 0 ? SnowflakeNumber.of(precision, scale) : SnowflakeNumber.of();
+ }
+ case IValueMeta.TYPE_BOOLEAN -> SnowflakeBoolean.of();
+ case IValueMeta.TYPE_DATE, IValueMeta.TYPE_TIMESTAMP -> SnowflakeTimestampNTZ.of();
+ case IValueMeta.TYPE_BINARY -> SnowflakeBinary.of();
+ default -> throw new HopException("Unsupported Snowflake field type: " + meta.getTypeDesc());
+ };
+ }
+
+ public static int hopType(String typeName) throws HopException {
+ int type = ValueMetaFactory.getIdForValueMeta(typeName);
+ if (type == IValueMeta.TYPE_NONE
+ || !(type == IValueMeta.TYPE_STRING
+ || type == IValueMeta.TYPE_INTEGER
+ || type == IValueMeta.TYPE_NUMBER
+ || type == IValueMeta.TYPE_BIGNUMBER
+ || type == IValueMeta.TYPE_BOOLEAN
+ || type == IValueMeta.TYPE_DATE
+ || type == IValueMeta.TYPE_TIMESTAMP
+ || type == IValueMeta.TYPE_BINARY)) {
+ throw new HopException("Unsupported Snowflake field type: " + typeName);
+ }
+ return type;
+ }
+
+ /** A string Beam will quote, or null for an empty CSV field. */
+ public static String toCsv(IValueMeta meta, Object value) throws HopException {
+ if (value == null) return null;
+ try {
+ return switch (meta.getType()) {
+ case IValueMeta.TYPE_STRING -> value.toString();
+ case IValueMeta.TYPE_INTEGER -> Long.toString(((Number) value).longValue());
+ case IValueMeta.TYPE_NUMBER ->
+ BigDecimal.valueOf(((Number) value).doubleValue()).stripTrailingZeros().toPlainString();
+ case IValueMeta.TYPE_BIGNUMBER -> decimal(value).toPlainString();
+ case IValueMeta.TYPE_BOOLEAN -> Boolean.TRUE.equals(value) ? "true" : "false";
+ case IValueMeta.TYPE_DATE, IValueMeta.TYPE_TIMESTAMP -> {
+ if (value instanceof Date date) {
+ yield TIMESTAMP.format(date.toInstant());
+ }
+ fail(meta);
+ yield null;
+ }
+ case IValueMeta.TYPE_BINARY -> {
+ if (value instanceof byte[] bytes) {
+ yield HexFormat.of().formatHex(bytes);
+ }
+ fail(meta);
+ yield null;
+ }
+ default -> {
+ fail(meta);
+ yield null;
+ }
+ };
+ } catch (HopException e) {
+ throw e;
+ } catch (RuntimeException e) {
+ fail(meta);
+ return null;
+ }
+ }
+
+ public static Object fromCsv(IValueMeta meta, String text) throws HopException {
+ if (text == null || text.isEmpty()) return null;
+ if (meta.getType() == IValueMeta.TYPE_STRING) return text;
+ String trimmed = text.trim();
+ if (trimmed.isEmpty()) return null;
+ try {
+ return switch (meta.getType()) {
+ case IValueMeta.TYPE_INTEGER ->
+ new BigDecimal(trimmed).stripTrailingZeros().longValueExact();
+ case IValueMeta.TYPE_NUMBER -> finiteDouble(new BigDecimal(trimmed));
+ case IValueMeta.TYPE_BIGNUMBER -> new BigDecimal(trimmed);
+ case IValueMeta.TYPE_BOOLEAN -> bool(trimmed);
+ case IValueMeta.TYPE_DATE -> parseDate(trimmed);
+ case IValueMeta.TYPE_TIMESTAMP -> new java.sql.Timestamp(parseDate(trimmed).getTime());
+ case IValueMeta.TYPE_BINARY -> HexFormat.of().parseHex(trimmed);
+ default -> {
+ fail(meta);
+ yield null;
+ }
+ };
+ } catch (HopException e) {
+ throw e;
+ } catch (RuntimeException e) {
+ fail(meta);
+ return null;
+ }
+ }
+
+ private static BigDecimal decimal(Object value) {
+ if (value instanceof BigDecimal decimal) return decimal;
+ return new BigDecimal(value.toString());
+ }
+
+ private static Double finiteDouble(BigDecimal number) {
+ double value = number.doubleValue();
+ if (!Double.isFinite(value)) throw new IllegalArgumentException("outside double range");
+ return value;
+ }
+
+ private static Boolean bool(String text) {
+ return switch (text.toLowerCase(Locale.ROOT)) {
+ case "true", "1" -> Boolean.TRUE;
+ case "false", "0" -> Boolean.FALSE;
+ default -> throw new IllegalArgumentException("boolean");
+ };
+ }
+
+ private static Date parseDate(String text) {
+ if (text.length() == 10) {
+ return Date.from(LocalDate.parse(text).atStartOfDay(ZoneOffset.UTC).toInstant());
+ }
+ Instant instant = LocalDateTime.parse(text, TIMESTAMP_LOCAL).toInstant(ZoneOffset.UTC);
+ return Date.from(instant);
+ }
+
+ private static void fail(IValueMeta meta) throws HopException {
+ throw new HopException("Unable to convert Snowflake field " + meta.getName());
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/SplunkWriteFailureFn.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/SplunkWriteFailureFn.java
new file mode 100644
index 0000000000..39f9955d1e
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/SplunkWriteFailureFn.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.io.splunk.SplunkWriteError;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.hop.core.exception.HopRuntimeException;
+import org.apache.hop.pipeline.Pipeline;
+
+/**
+ * SplunkIO emits non-transient HEC failures instead of failing the bundle. Hop treats any such
+ * error as a pipeline failure. The status message and payload are omitted so event bodies and
+ * tokens cannot leak into the error text.
+ */
+@RequiredArgsConstructor
+public class SplunkWriteFailureFn extends DoFn<SplunkWriteError, Void> {
+ private final String transformName;
+
+ @ProcessElement
+ public void process(@Element SplunkWriteError error) {
+ Metrics.counter(Pipeline.METRIC_NAME_ERROR, transformName).inc();
+ throw new HopRuntimeException(message(error));
+ }
+
+ /** Status only. The status message and payload can contain the rejected event. */
+ public static String message(SplunkWriteError error) {
+ Integer status = error == null ? null : error.statusCode();
+ if (status == null) {
+ return "Splunk HEC write failed without an HTTP status";
+ }
+ return "Splunk HEC rejected an event with HTTP status " + status;
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/partition/KeyedPartitionFn.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/partition/KeyedPartitionFn.java
new file mode 100644
index 0000000000..d5908e5845
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/partition/KeyedPartitionFn.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.partition;
+
+import java.io.Serial;
+import java.math.BigDecimal;
+import java.util.Arrays;
+import java.util.Objects;
+import org.apache.beam.sdk.transforms.Partition;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.JsonRowMeta;
+
+/**
+ * Issue #2040: assigns a row to a partition based on the value of a key field.
+ *
+ * <p>Rows sharing a key always land in the same partition, which is what makes the following
+ * per-key steps actually group anything.
+ *
+ * <p>The value meta is carried as JSON rather than as an {@link IValueMeta} because the partition
+ * function is serialized to the workers, and an {@link IValueMeta} does not survive that; the JSON
+ * form does, and the meta is rebuilt lazily on first use.
+ */
+public class KeyedPartitionFn implements Partition.PartitionFn<HopRow> {
+ @Serial private static final long serialVersionUID = 95100000000000002L;
+
+ private final String transformName;
+ private final String rowMetaJson;
+ private final int keyIndex;
+
+ private transient IValueMeta keyValueMeta;
+
+ public KeyedPartitionFn(String transformName, String rowMetaJson, int keyIndex) {
+ this.transformName = transformName;
+ this.rowMetaJson = rowMetaJson;
+ this.keyIndex = keyIndex;
+ }
+
+ @Override
+ public int partitionFor(HopRow elem, int numPartitions) {
+ try {
+ if (keyValueMeta == null) {
+ IRowMeta rowMeta = JsonRowMeta.fromJson(rowMetaJson);
+ keyValueMeta = rowMeta.getValueMeta(keyIndex);
+ }
+
+ Object key = elem.getRow()[keyIndex];
+
+ // A null key still has to land somewhere: all such rows share one partition rather than
+ // failing the pipeline.
+ //
+ if (numPartitions < 1) {
+ return 0;
+ }
+ if (key == null) {
+ return 0;
+ }
+
+ // Math.abs is not used: it returns a negative number for Integer.MIN_VALUE, which would
+ // produce an invalid partition index.
+ //
+ int hash;
+ if (key instanceof byte[] bytes) {
+ hash = Arrays.hashCode(bytes);
+ } else if (key instanceof BigDecimal bd) {
+ hash =
+ (bd.compareTo(BigDecimal.ZERO) == 0 ? BigDecimal.ZERO : bd.stripTrailingZeros())
+ .hashCode();
+ } else {
+ try {
+ hash = keyValueMeta != null ? keyValueMeta.hashCode(key) : Objects.hashCode(key);
+ } catch (Exception e) {
+ hash = Objects.hashCode(key);
+ }
+ }
+ return Math.floorMod(hash, numPartitions);
+
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Unable to determine the partition for transform '" + transformName + "'", e);
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamBQInputTransform.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamBQInputTransform.java
index d78a951910..2d4792a572 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamBQInputTransform.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamBQInputTransform.java
@@ -43,6 +43,8 @@ public class BeamBQInputTransform extends PTransform<PBegin, PCollection<HopRow>
private String datasetId;
private String tableId;
private String query;
+ private String queryLocation;
+ private boolean skippingQueryValidation;
private String rowMetaJson;
// Log and count errors.
@@ -60,6 +62,8 @@ public class BeamBQInputTransform extends PTransform<PBegin, PCollection<HopRow>
String datasetId,
String tableId,
String query,
+ String queryLocation,
+ boolean skippingQueryValidation,
String rowMetaJson) {
super(name);
this.transformName = transformName;
@@ -67,6 +71,8 @@ public class BeamBQInputTransform extends PTransform<PBegin, PCollection<HopRow>
this.datasetId = datasetId;
this.tableId = tableId;
this.query = query;
+ this.queryLocation = queryLocation;
+ this.skippingQueryValidation = skippingQueryValidation;
this.rowMetaJson = rowMetaJson;
}
@@ -94,6 +100,15 @@ public class BeamBQInputTransform extends PTransform<PBegin, PCollection<HopRow>
bqTypedRead = BigQueryIO.read(toHopFn).from(tableReference).usingStandardSql();
} else {
bqTypedRead = BigQueryIO.read(toHopFn).fromQuery(query).usingStandardSql();
+
+ // #2416: let the caller point at a specific BigQuery dataset location and skip validation
+ // for a query that is not registered when the job is submitted.
+ if (StringUtils.isNotEmpty(queryLocation)) {
+ bqTypedRead = bqTypedRead.withQueryLocation(queryLocation);
+ }
+ if (skippingQueryValidation) {
+ bqTypedRead = bqTypedRead.withoutValidation();
+ }
}
// Apply the function
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamDebeziumInputTransform.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamDebeziumInputTransform.java
new file mode 100644
index 0000000000..f389676aad
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamDebeziumInputTransform.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.transform;
+
+import org.apache.beam.io.debezium.DebeziumIO;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hop.beam.core.BeamHop;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.coder.HopRowCoder;
+import org.apache.hop.core.exception.HopRuntimeException;
+import org.apache.hop.pipeline.Pipeline;
+
+/** Debezium's real JSON source followed by a single-string Hop row conversion. */
+public class BeamDebeziumInputTransform extends PTransform<PBegin, PCollection<HopRow>> {
+ private final String transformName;
+ private final DebeziumIO.Read<String> read;
+
+ public BeamDebeziumInputTransform(String transformName, DebeziumIO.Read<String> read) {
+ super(transformName);
+ this.transformName = transformName;
+ this.read = read;
+ }
+
+ @Override
+ public PCollection<HopRow> expand(PBegin input) {
+ return input
+ .apply("Debezium CDC", read)
+ .apply("JSON to Hop row", ParDo.of(new JsonToHopRowFn(transformName)))
+ .setCoder(new HopRowCoder());
+ }
+
+ public static class JsonToHopRowFn extends DoFn<String, HopRow> {
+ private final String transformName;
+ private transient Counter inputCounter;
+ private transient Counter writtenCounter;
+
+ public JsonToHopRowFn(String transformName) {
+ this.transformName = transformName;
+ }
+
+ @Setup
+ public void setup() {
+ try {
+ BeamHop.init();
+ inputCounter = Metrics.counter(Pipeline.METRIC_NAME_INPUT, transformName);
+ writtenCounter = Metrics.counter(Pipeline.METRIC_NAME_WRITTEN, transformName);
+ Metrics.counter(Pipeline.METRIC_NAME_INIT, transformName).inc();
+ } catch (Exception e) {
+ Metrics.counter(Pipeline.METRIC_NAME_ERROR, transformName).inc();
+ throw new HopRuntimeException("Unable to initialize Debezium JSON row conversion", e);
+ }
+ }
+
+ @ProcessElement
+ public void processElement(ProcessContext context) {
+ inputCounter.inc();
+ context.output(new HopRow(new Object[] {context.element()}));
+ writtenCounter.inc();
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamElasticsearchInputTransform.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamElasticsearchInputTransform.java
new file mode 100644
index 0000000000..14cf578b1d
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamElasticsearchInputTransform.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.core.transform;
+
+import org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.coder.HopRowCoder;
+import org.apache.hop.beam.core.fn.StringToHopRowFn;
+
+/** Bounded JSON document source backed by Beam's Elasticsearch scroll reader. */
+public class BeamElasticsearchInputTransform extends PTransform<PBegin, PCollection<HopRow>> {
+ private final String transformName;
+ private final ElasticsearchIO.Read read;
+ private final String rowMetaJson;
+
+ public BeamElasticsearchInputTransform(
+ String transformName, ElasticsearchIO.Read read, String rowMetaJson) {
+ this.transformName = transformName;
+ this.read = read;
+ this.rowMetaJson = rowMetaJson;
+ }
+
+ @Override
+ public PCollection<HopRow> expand(PBegin input) {
+ return input
+ .apply("Read documents", read)
+ .apply("JSON to Hop rows", ParDo.of(new StringToHopRowFn(transformName, rowMetaJson)))
+ .setCoder(new HopRowCoder());
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamElasticsearchOutputTransform.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamElasticsearchOutputTransform.java
new file mode 100644
index 0000000000..2a103ca7b1
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamElasticsearchOutputTransform.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.core.transform;
+
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.fn.ElasticsearchJsonFn;
+
+/** JSON document sink backed by Beam's bulk writer. */
+public class BeamElasticsearchOutputTransform extends PTransform<PCollection<HopRow>, PDone> {
+ private final String transformName;
+ private final ElasticsearchIO.Write write;
+ private final String jsonField;
+ private final String rowMetaXml;
+
+ public BeamElasticsearchOutputTransform(
+ String transformName, ElasticsearchIO.Write write, String jsonField, String rowMetaXml) {
+ this.transformName = transformName;
+ this.write = write;
+ this.jsonField = jsonField;
+ this.rowMetaXml = rowMetaXml;
+ }
+
+ private static class CountWritesFn extends DoFn<ElasticsearchIO.Document, Void> {
+ private final String transformName;
+
+ CountWritesFn(String transformName) {
+ this.transformName = transformName;
+ }
+
+ @ProcessElement
+ public void processElement() {
+ Metrics.counter(org.apache.hop.pipeline.Pipeline.METRIC_NAME_WRITTEN, transformName).inc();
+ }
+ }
+
+ @Override
+ public PDone expand(PCollection<HopRow> input) {
+ PCollectionTuple results =
+ input
+ .apply(
+ "Extract JSON",
+ ParDo.of(new ElasticsearchJsonFn(transformName, jsonField, rowMetaXml)))
+ .setCoder(StringUtf8Coder.of())
+ .apply("Write documents", write);
+ results
+ .get(ElasticsearchIO.Write.SUCCESSFUL_WRITES)
+ .apply("Count written documents", ParDo.of(new CountWritesFn(transformName)));
+ return PDone.in(input.getPipeline());
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamMqttConnection.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamMqttConnection.java
new file mode 100644
index 0000000000..242784cb2b
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamMqttConnection.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.transform;
+
+import java.io.Serializable;
+import java.net.URI;
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.io.mqtt.MqttIO;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hop.core.encryption.Encr;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.variables.IVariables;
+
+/** Resolved worker connection settings. Do not add a toString that could expose credentials. */
+@RequiredArgsConstructor
+public class BeamMqttConnection implements Serializable {
+ private final String serverUri;
+ private final String topic;
+ private final String clientId;
+ private final String username;
+ private final String encryptedPassword;
+
+ public static BeamMqttConnection resolve(
+ IVariables variables,
+ String serverUri,
+ String topic,
+ String clientId,
+ String username,
+ String password,
+ boolean source)
+ throws HopException {
+ String uri = required(variables, serverUri, "server URI");
+ try {
+ URI parsed = new URI(uri);
+ if (!("tcp".equals(parsed.getScheme()) || "ssl".equals(parsed.getScheme()))
+ || StringUtils.isBlank(parsed.getHost())
+ || parsed.getUserInfo() != null
+ || parsed.getQuery() != null
+ || parsed.getFragment() != null
+ || !StringUtils.isEmpty(parsed.getPath())
+ || parsed.getPort() == 0
+ || parsed.getPort() > 65535) throw new IllegalArgumentException();
+ } catch (Exception e) {
+ throw new HopException(
+ "MQTT server URI must be tcp://host:port or ssl://host:port, without embedded credentials");
+ }
+ String resolvedTopic = required(variables, topic, "topic");
+ if (resolvedTopic.indexOf(0) >= 0
+ || (!source && (resolvedTopic.contains("#") || resolvedTopic.contains("+"))))
+ throw new HopException("MQTT output topic must not contain wildcards or NUL");
+ String user = variables.resolve(username);
+ String secret =
+ Encr.decryptPasswordOptionallyEncrypted(
+ variables.resolve(Encr.decryptPasswordOptionallyEncrypted(password)));
+ if (StringUtils.isNotEmpty(secret) && StringUtils.isBlank(user))
+ throw new HopException("MQTT password requires a username");
+ return new BeamMqttConnection(
+ uri,
+ resolvedTopic,
+ variables.resolve(clientId),
+ user,
+ Encr.encryptPasswordIfNotUsingVariables(secret));
+ }
+
+ public MqttIO.ConnectionConfiguration createConfiguration() {
+ var configuration = MqttIO.ConnectionConfiguration.create(serverUri, topic);
+ if (StringUtils.isNotEmpty(clientId)) configuration = configuration.withClientId(clientId);
+ if (StringUtils.isNotEmpty(username)) configuration = configuration.withUsername(username);
+ if (StringUtils.isNotEmpty(encryptedPassword))
+ configuration =
+ configuration.withPassword(Encr.decryptPasswordOptionallyEncrypted(encryptedPassword));
+ return configuration;
+ }
+
+ public static String required(IVariables variables, String value, String option)
+ throws HopException {
+ String resolved = variables.resolve(value);
+ if (StringUtils.isBlank(resolved) || resolved.contains("${"))
+ throw new HopException("Please specify a resolved MQTT " + option);
+ return resolved;
+ }
+
+ public static String payloadType(IVariables variables, String value) throws HopException {
+ String type = required(variables, value, "payload type");
+ if (!"String".equalsIgnoreCase(type) && !"Binary".equalsIgnoreCase(type))
+ throw new HopException("MQTT payload type must be String or Binary");
+ return type;
+ }
+
+ public static long limit(IVariables variables, String value, String option) throws HopException {
+ String resolved = variables.resolve(value);
+ if (StringUtils.isBlank(resolved)) return 0;
+ try {
+ long limit = Long.parseLong(resolved);
+ if (limit < 0) throw new NumberFormatException();
+ return limit;
+ } catch (NumberFormatException e) {
+ throw new HopException("MQTT " + option + " must be a non-negative integer");
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamMqttInputTransform.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamMqttInputTransform.java
new file mode 100644
index 0000000000..7ac060db88
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamMqttInputTransform.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.transform;
+
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.io.mqtt.MqttIO;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.coder.HopRowCoder;
+import org.apache.hop.beam.core.fn.MqttToHopFn;
+import org.joda.time.Duration;
+
+@RequiredArgsConstructor
+public class BeamMqttInputTransform extends PTransform<PBegin, PCollection<HopRow>> {
+ private final String transformName;
+ private final BeamMqttConnection connection;
+ private final String payloadType;
+ private final long maxNumRecords;
+ private final long maxReadTime;
+
+ public MqttIO.Read<byte[]> getRead() {
+ var read = MqttIO.read().withConnectionConfiguration(connection.createConfiguration());
+ if (maxNumRecords > 0) read = read.withMaxNumRecords(maxNumRecords);
+ if (maxReadTime > 0) read = read.withMaxReadTime(Duration.standardSeconds(maxReadTime));
+ return read;
+ }
+
+ @Override
+ public PCollection<HopRow> expand(PBegin input) {
+ return input
+ .apply("Read MQTT", getRead())
+ .apply("Convert MQTT payload", ParDo.of(new MqttToHopFn(transformName, payloadType)))
+ .setCoder(new HopRowCoder());
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamMqttOutputTransform.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamMqttOutputTransform.java
new file mode 100644
index 0000000000..161e2c45cd
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamMqttOutputTransform.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.transform;
+
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.io.mqtt.MqttIO;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.fn.HopToMqttFn;
+
+@RequiredArgsConstructor
+public class BeamMqttOutputTransform extends PTransform<PCollection<HopRow>, PDone> {
+ private final String transformName;
+ private final BeamMqttConnection connection;
+ private final String payloadField;
+ private final String payloadType;
+ private final String rowMetaXml;
+ private final boolean retained;
+
+ public MqttIO.Write<byte[]> getWrite() {
+ return MqttIO.write()
+ .withConnectionConfiguration(connection.createConfiguration())
+ .withRetained(retained);
+ }
+
+ @Override
+ public PDone expand(PCollection<HopRow> input) {
+ return input
+ .apply(
+ "Convert Hop payload",
+ ParDo.of(new HopToMqttFn(transformName, payloadField, payloadType, rowMetaXml)))
+ .setCoder(ByteArrayCoder.of())
+ .apply("Write MQTT", getWrite());
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamOutputTransform.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamOutputTransform.java
index b1298d23cc..acf5d46abe 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamOutputTransform.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamOutputTransform.java
@@ -18,6 +18,7 @@
package org.apache.hop.beam.core.transform;
import java.io.File;
+import org.apache.beam.sdk.io.Compression;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.metrics.Counter;
import org.apache.beam.sdk.metrics.Metrics;
@@ -47,6 +48,7 @@ public class BeamOutputTransform extends PTransform<PCollection<HopRow>, PDone>
private String enclosure;
private String rowMetaJson;
private boolean windowed;
+ private String compression;
// Log and count errors.
private static final Logger LOG = LoggerFactory.getLogger(BeamOutputTransform.class);
@@ -62,6 +64,7 @@ public class BeamOutputTransform extends PTransform<PCollection<HopRow>, PDone>
String separator,
String enclosure,
boolean windowed,
+ String compression,
String rowMetaJson) {
this.transformName = transformName;
this.outputLocation = outputLocation;
@@ -70,6 +73,7 @@ public class BeamOutputTransform extends PTransform<PCollection<HopRow>, PDone>
this.separator = separator;
this.enclosure = enclosure;
this.windowed = windowed;
+ this.compression = compression;
this.rowMetaJson = rowMetaJson;
}
@@ -108,8 +112,35 @@ public class BeamOutputTransform extends PTransform<PCollection<HopRow>, PDone>
}
write = write.to(outputPrefix);
}
- if (StringUtils.isNotEmpty(fileSuffix)) {
- write = write.withSuffix(fileSuffix);
+ // #2337: TextIO can write compressed files. Beam's AUTO is read-only ("AUTO is not supported
+ // for writing"), so for a write we resolve it to the codec the file suffix implies instead
+ // of passing it through. With no suffix there is nothing to infer, so AUTO degrades to no
+ // compression rather than failing the job.
+ //
+ // DefaultFilenamePolicy appends the codec suffix again after withSuffix, so a suffix that
+ // already ends in that extension (.csv.gz with GZIP) would be written as .csv.gz.gz.
+ //
+ Compression codec = null;
+ if (StringUtils.isNotEmpty(compression)) {
+ try {
+ codec = resolveCompression(compression, fileSuffix);
+ } catch (IllegalArgumentException e) {
+ throw new HopRuntimeException(
+ "Unknown compression '"
+ + compression
+ + "' in Beam output transform '"
+ + transformName
+ + "'. Use one of AUTO, UNCOMPRESSED, GZIP, BZIP2, ZIP, ZSTD, LZO, LZOP, "
+ + "DEFLATE or SNAPPY.",
+ e);
+ }
+ }
+ String suffix = stripSuggestedSuffix(fileSuffix, codec);
+ if (StringUtils.isNotEmpty(suffix)) {
+ write = write.withSuffix(suffix);
+ }
+ if (codec != null) {
+ write = write.withCompression(codec);
}
// For streaming data sources...
@@ -131,6 +162,50 @@ public class BeamOutputTransform extends PTransform<PCollection<HopRow>, PDone>
}
}
+ /**
+ * Resolve the configured compression name into a codec TextIO can actually write with.
+ *
+ * <p>Returns null to mean "write it uncompressed". Throws {@link IllegalArgumentException} for a
+ * name that is not a Beam {@link Compression} constant.
+ *
+ * @param compression the configured name, may be AUTO
+ * @param fileSuffix the configured file suffix, may be blank
+ */
+ private static Compression resolveCompression(String compression, String fileSuffix) {
+ if (!Compression.AUTO.name().equalsIgnoreCase(compression)) {
+ return Compression.valueOf(compression);
+ }
+
+ // AUTO: infer from the suffix. Compression.matches() knows the conventional suffixes, so
+ // ".gz" resolves to GZIP, ".bz2" to BZIP2 and so on.
+ //
+ if (StringUtils.isNotEmpty(fileSuffix)) {
+ for (Compression candidate : Compression.values()) {
+ if (candidate != Compression.AUTO
+ && candidate != Compression.UNCOMPRESSED
+ && candidate.matches(fileSuffix)) {
+ return candidate;
+ }
+ }
+ }
+ return null;
+ }
+
+ /**
+ * Drop the codec suffix from the configured file suffix. Beam appends {@link
+ * Compression#getSuggestedSuffix()} itself when the write is compressed.
+ */
+ private static String stripSuggestedSuffix(String fileSuffix, Compression codec) {
+ if (codec == null || StringUtils.isEmpty(fileSuffix)) {
+ return fileSuffix;
+ }
+ String suggested = codec.getSuggestedSuffix();
+ if (StringUtils.isNotEmpty(suggested) && fileSuffix.endsWith(suggested)) {
+ return fileSuffix.substring(0, fileSuffix.length() - suggested.length());
+ }
+ return fileSuffix;
+ }
+
/**
* Gets transformName
*
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamSnowflakeInputTransform.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamSnowflakeInputTransform.java
new file mode 100644
index 0000000000..c937ab77e0
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamSnowflakeInputTransform.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.transform;
+
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.io.snowflake.SnowflakeIO;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hop.beam.core.HopRow;
+
+/** Bounded Snowflake COPY into a GCS stage. Streaming read is not offered by SnowflakeIO. */
+@RequiredArgsConstructor
+public class BeamSnowflakeInputTransform extends PTransform<PBegin, PCollection<HopRow>> {
+ private final SnowflakeIO.Read<HopRow> read;
+
+ public SnowflakeIO.Read<HopRow> snowflakeRead() {
+ return read;
+ }
+
+ @Override
+ public PCollection<HopRow> expand(PBegin input) {
+ return input.apply(read);
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamSnowflakeOutputTransform.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamSnowflakeOutputTransform.java
new file mode 100644
index 0000000000..fb8c03a091
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamSnowflakeOutputTransform.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.transform;
+
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.io.snowflake.SnowflakeIO;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.hop.beam.core.HopRow;
+
+/** Bounded COPY from a GCS stage. Snowpipe is not used, so the result is {@link PDone}. */
+@RequiredArgsConstructor
+public class BeamSnowflakeOutputTransform extends PTransform<PCollection<HopRow>, PDone> {
+ private final SnowflakeIO.Write<HopRow> write;
+
+ public SnowflakeIO.Write<HopRow> snowflakeWrite() {
+ return write;
+ }
+
+ @Override
+ public PDone expand(PCollection<HopRow> input) {
+ input.apply(write);
+ return PDone.in(input.getPipeline());
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamSplunkOutputTransform.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamSplunkOutputTransform.java
new file mode 100644
index 0000000000..51e7aa821e
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/transform/BeamSplunkOutputTransform.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.transform;
+
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.io.splunk.SplunkEventCoder;
+import org.apache.beam.sdk.io.splunk.SplunkIO;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.coder.SplunkWriteErrorCoder;
+import org.apache.hop.beam.core.fn.HopToSplunkFn;
+import org.apache.hop.beam.core.fn.SplunkWriteFailureFn;
+
+/** Terminal HEC sink. Rejected events fail the pipeline and are not Hop output rows. */
+@RequiredArgsConstructor
+public class BeamSplunkOutputTransform extends PTransform<PCollection<HopRow>, PDone> {
+ private final String transformName;
+ private final String url;
+ private final String token;
+ private final Integer batchCount;
+ private final boolean disableCertificateValidation;
+ private final boolean enableGzip;
+ private final String rootCaCertificatePath;
+ private final String eventField;
+ private final String rowMetaXml;
+ private final String host;
+ private final String source;
+ private final String sourceType;
+ private final String index;
+
+ public SplunkIO.Write splunkWrite() {
+ SplunkIO.Write write =
+ SplunkIO.write(url, token)
+ .withEnableGzipHttpCompression(enableGzip)
+ .withDisableCertificateValidation(disableCertificateValidation);
+ if (batchCount != null) write = write.withBatchCount(batchCount);
+ if (StringUtils.isNotEmpty(rootCaCertificatePath))
+ write = write.withRootCaCertificatePath(rootCaCertificatePath);
+ return write;
+ }
+
+ @Override
+ public PDone expand(PCollection<HopRow> input) {
+ input
+ .apply(
+ "Convert Hop event",
+ ParDo.of(
+ new HopToSplunkFn(
+ transformName, eventField, rowMetaXml, host, source, sourceType, index)))
+ .setCoder(SplunkEventCoder.of())
+ .apply("Write Splunk", splunkWrite())
+ .setCoder(SplunkWriteErrorCoder.of())
+ .apply("Fail rejected Splunk events", ParDo.of(new SplunkWriteFailureFn(transformName)));
+ return PDone.in(input.getPipeline());
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/BeamPipelineEngine.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/BeamPipelineEngine.java
index 0ef0856791..f288a6fb31 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/BeamPipelineEngine.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/BeamPipelineEngine.java
@@ -68,6 +68,7 @@ import org.apache.hop.core.parameters.NamedParameters;
import org.apache.hop.core.parameters.UnknownParamException;
import org.apache.hop.core.plugins.EngineCompatibility;
import org.apache.hop.core.plugins.IPlugin;
+import org.apache.hop.core.plugins.PluginRegistry;
import org.apache.hop.core.util.ExecutorUtil;
import org.apache.hop.core.variables.IVariables;
import org.apache.hop.core.variables.Variables;
@@ -1788,6 +1789,17 @@ public abstract class BeamPipelineEngine extends Variables
}
}
Class<?> mainType = transformPlugin.getMainType();
+ // Registered plugins expose ITransformMeta as the main role, not their implementation.
+ // Resolve the implementation without constructing metadata or initializing its runtime.
+ if (mainType != null
+ && transformPlugin.getClassMap() != null
+ && transformPlugin.getClassMap().containsKey(mainType)) {
+ try {
+ mainType = PluginRegistry.getInstance().getClass(transformPlugin, mainType);
+ } catch (HopException e) {
+ return EngineCompatibility.unknown();
+ }
+ }
if (mainType != null) {
String banReason =
HopPipelineMetaToBeamPipelineConverter.HARD_BANNED_META_TYPES.get(mainType);
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/dataflow/BeamDataFlowPipelineRunConfiguration.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/dataflow/BeamDataFlowPipelineRunConfiguration.java
index 292b314e8e..970311718b 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/dataflow/BeamDataFlowPipelineRunConfiguration.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/dataflow/BeamDataFlowPipelineRunConfiguration.java
@@ -23,6 +23,7 @@ import java.util.Collections;
import java.util.List;
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
import org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.commons.lang3.StringUtils;
@@ -80,6 +81,15 @@ public class BeamDataFlowPipelineRunConfiguration extends BeamPipelineRunConfigu
@HopMetadataProperty
private String gcpStagingLocation;
+ @GuiWidgetElement(
+ order = "20025-dataflow-options",
+ parentId = PipelineRunConfiguration.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ type = GuiElementType.TEXT,
+ toolTip = "i18n::BeamEnginesDataflow.OptionsGcpTempLocation.ToolTip",
+ label = "i18n::BeamEnginesDataflow.OptionsGcpTempLocation.Label")
+ @HopMetadataProperty
+ private String gcpTempLocation;
+
@GuiWidgetElement(
order = "20030-dataflow-options",
parentId = PipelineRunConfiguration.GUI_PLUGIN_ELEMENT_PARENT_ID,
@@ -201,6 +211,7 @@ public class BeamDataFlowPipelineRunConfiguration extends BeamPipelineRunConfigu
this.gcpServiceAccount = config.gcpServiceAccount;
this.gcpAppName = config.gcpAppName;
this.gcpStagingLocation = config.gcpStagingLocation;
+ this.gcpTempLocation = config.gcpTempLocation;
this.gcpInitialNumberOfWorkers = config.gcpInitialNumberOfWorkers;
this.gcpMaximumNumberOfWorkers = config.gcpMaximumNumberOfWorkers;
this.gcpAutoScalingAlgorithm = config.gcpAutoScalingAlgorithm;
@@ -233,6 +244,12 @@ public class BeamDataFlowPipelineRunConfiguration extends BeamPipelineRunConfigu
options.setAppName(resolve(getGcpAppName()));
options.setStagingLocation(resolve(getGcpStagingLocation()));
+ if (StringUtils.isNotEmpty(getGcpTempLocation())) {
+ // #2355: BigQuery load jobs and the other GCP IOs need a GCS path in gcpTempLocation, which
+ // is a separate option from the general tempLocation.
+ options.as(GcpOptions.class).setGcpTempLocation(resolve(getGcpTempLocation()));
+ }
+
if (StringUtils.isNotEmpty(getGcpServiceAccount())) {
options.setServiceAccount(resolve(getGcpServiceAccount()));
}
@@ -393,6 +410,22 @@ public class BeamDataFlowPipelineRunConfiguration extends BeamPipelineRunConfigu
this.gcpStagingLocation = gcpStagingLocation;
}
+ /**
+ * Gets the GCP temp location
+ *
+ * @return value of gcpTempLocation
+ */
+ public String getGcpTempLocation() {
+ return gcpTempLocation;
+ }
+
+ /**
+ * @param gcpTempLocation The gcpTempLocation to set
+ */
+ public void setGcpTempLocation(String gcpTempLocation) {
+ this.gcpTempLocation = gcpTempLocation;
+ }
+
/**
* Gets gcpInitialNumberOfWorkers
*
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/direct/BeamDirectPipelineRunConfiguration.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/direct/BeamDirectPipelineRunConfiguration.java
index 0d01aba54b..fd480c29de 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/direct/BeamDirectPipelineRunConfiguration.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/engines/direct/BeamDirectPipelineRunConfiguration.java
@@ -14,27 +14,10 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
package org.apache.hop.beam.engines.direct;
import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.commons.lang3.StringUtils;
@@ -65,6 +48,15 @@ public class BeamDirectPipelineRunConfiguration extends BeamPipelineRunConfigura
@HopMetadataProperty(key = "number_of_workers")
private String numberOfWorkers;
+ @GuiWidgetElement(
+ order = "20010-direct-options",
+ parentId = PipelineRunConfiguration.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamEnginesDirect.OptionsGcpTempLocation.Label",
+ toolTip = "i18n::BeamEnginesDirect.OptionsGcpTempLocation.ToolTip")
+ @HopMetadataProperty(key = "gcp_temp_location")
+ private String gcpTempLocation;
+
public BeamDirectPipelineRunConfiguration(String numberOfWorkers) {
super("BeamDirectPipelineEngine", "direct runner");
this.numberOfWorkers = numberOfWorkers;
@@ -79,6 +71,7 @@ public class BeamDirectPipelineRunConfiguration extends BeamPipelineRunConfigura
public BeamDirectPipelineRunConfiguration(BeamDirectPipelineRunConfiguration config) {
super(config);
this.numberOfWorkers = config.numberOfWorkers;
+ this.gcpTempLocation = config.gcpTempLocation;
}
@Override
@@ -99,6 +92,10 @@ public class BeamDirectPipelineRunConfiguration extends BeamPipelineRunConfigura
int targetParallelism = Const.toInt(resolve(numberOfWorkers), 1);
options.setTargetParallelism(targetParallelism);
}
+ if (StringUtils.isNotEmpty(gcpTempLocation)) {
+ // #2355: GCP IOs need a gs:// path. This is separate from the general temp location.
+ options.as(GcpOptions.class).setGcpTempLocation(resolve(gcpTempLocation));
+ }
return options;
}
@@ -123,4 +120,20 @@ public class BeamDirectPipelineRunConfiguration extends BeamPipelineRunConfigura
public void setNumberOfWorkers(String numberOfWorkers) {
this.numberOfWorkers = numberOfWorkers;
}
+
+ /**
+ * Gets gcpTempLocation
+ *
+ * @return value of gcpTempLocation
+ */
+ public String getGcpTempLocation() {
+ return gcpTempLocation;
+ }
+
+ /**
+ * @param gcpTempLocation The gcpTempLocation to set
+ */
+ public void setGcpTempLocation(String gcpTempLocation) {
+ this.gcpTempLocation = gcpTempLocation;
+ }
}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/BeamSourceTopology.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/BeamSourceTopology.java
new file mode 100644
index 0000000000..a830791a21
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/BeamSourceTopology.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.pipeline;
+
+import java.util.List;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+/**
+ * Beam source handlers are called with a null predecessor list and a null input collection. A
+ * non-null list is the caller's explicit topology. A null list is resolved from enabled row hops,
+ * including error hops and excluding informational hops.
+ */
+public final class BeamSourceTopology {
+ private BeamSourceTopology() {}
+
+ public static void rejectIncomingRows(
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ List<TransformMeta> previousTransforms,
+ PCollection<?> input,
+ String message)
+ throws HopException {
+ if (input != null) {
+ throw new HopException(message);
+ }
+ List<TransformMeta> predecessors = previousTransforms;
+ if (predecessors == null && pipelineMeta != null && transformMeta != null) {
+ predecessors = pipelineMeta.findPreviousTransforms(transformMeta, false);
+ }
+ if (predecessors != null && !predecessors.isEmpty()) {
+ throw new HopException(message);
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/HopPipelineMetaToBeamPipelineConverter.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/HopPipelineMetaToBeamPipelineConverter.java
index e85dd60e68..b4c97c46b9 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/HopPipelineMetaToBeamPipelineConverter.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/HopPipelineMetaToBeamPipelineConverter.java
@@ -42,6 +42,7 @@ import org.apache.hop.beam.engines.HopPipelineExecutionOptions;
import org.apache.hop.beam.engines.IBeamPipelineEngineRunConfiguration;
import org.apache.hop.beam.engines.dataflow.BeamDataFlowPipelineRunConfiguration;
import org.apache.hop.beam.metadata.RunnerType;
+import org.apache.hop.beam.pipeline.handler.BeamAddSequenceTransformHandler;
import org.apache.hop.beam.pipeline.handler.BeamGenericTransformHandler;
import org.apache.hop.beam.pipeline.handler.BeamMemoryGroupByTransformHandler;
import org.apache.hop.beam.pipeline.handler.BeamMergeJoinTransformHandler;
@@ -88,7 +89,8 @@ public class HopPipelineMetaToBeamPipelineConverter {
Set.of(
BeamConst.STRING_MERGE_JOIN_PLUGIN_ID,
BeamConst.STRING_BEAM_ROW_GENERATOR_PLUGIN_ID,
- BeamConst.STRING_MEMORY_GROUP_BY_PLUGIN_ID);
+ BeamConst.STRING_MEMORY_GROUP_BY_PLUGIN_ID,
+ BeamConst.STRING_ADD_SEQUENCE_PLUGIN_ID);
/**
* Transform meta classes that Beam refuses to run at all, mapped to the user-facing reason. The
@@ -263,6 +265,8 @@ public class HopPipelineMetaToBeamPipelineConverter {
BeamConst.STRING_BEAM_ROW_GENERATOR_PLUGIN_ID, new BeamRowGeneratorTransformHandler());
transformHandlers.put(
BeamConst.STRING_MEMORY_GROUP_BY_PLUGIN_ID, new BeamMemoryGroupByTransformHandler());
+ transformHandlers.put(
+ BeamConst.STRING_ADD_SEQUENCE_PLUGIN_ID, new BeamAddSequenceTransformHandler());
genericTransformHandler = new BeamGenericTransformHandler();
}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/handler/BeamAddSequenceTransformHandler.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/handler/BeamAddSequenceTransformHandler.java
new file mode 100644
index 0000000000..373f8baa8b
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/handler/BeamAddSequenceTransformHandler.java
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.pipeline.handler;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Values;
+import org.apache.beam.sdk.transforms.WithKeys;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.fn.AddSequenceFn;
+import org.apache.hop.beam.engines.IBeamPipelineEngineRunConfiguration;
+import org.apache.hop.beam.pipeline.IBeamPipelineTransformHandler;
+import org.apache.hop.core.Const;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.JsonRowMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaFactory;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.pipeline.transforms.addsequence.AddSequenceMeta;
+
+/**
+ * Issue #2379: a Beam handler for the Add Sequence transform.
+ *
+ * <p>A bounded counter is one {@code GroupByKey} and one stateful {@code DoFn}, so every worker
+ * shares a single sequence. Database mode and unbounded input cannot use that {@code GroupByKey}.
+ * They go through {@link BeamGenericTransformHandler}, which is what ran this transform before.
+ */
+public class BeamAddSequenceTransformHandler extends BeamBaseTransformHandler
+ implements IBeamPipelineTransformHandler {
+
+ @Override
+ public boolean isInput() {
+ return false;
+ }
+
+ @Override
+ public boolean isOutput() {
+ return false;
+ }
+
+ @Override
+ public void handleTransform(
+ ILogChannel log,
+ IVariables variables,
+ String runConfigurationName,
+ IBeamPipelineEngineRunConfiguration runConfiguration,
+ String dataSamplersJson,
+ IHopMetadataProvider metadataProvider,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ Map<String, PCollection<HopRow>> transformCollectionMap,
+ Pipeline pipeline,
+ IRowMeta rowMeta,
+ List<TransformMeta> previousTransforms,
+ PCollection<HopRow> input,
+ String parentLogChannelId)
+ throws HopException {
+
+ if (input == null) {
+ throw new HopException(
+ "The Add Sequence transform '"
+ + transformMeta.getName()
+ + "' needs an input: it numbers the rows which arrive from a previous transform.");
+ }
+
+ // Don't simply cast but serialize/de-serialize the metadata to prevent classloader exceptions
+ //
+ AddSequenceMeta meta = new AddSequenceMeta();
+ loadTransformMetadata(meta, transformMeta, metadataProvider, pipelineMeta);
+
+ // GroupByKey cannot run on an unbounded global window, a database sequence is the Hop
+ // transform's own connection, and a configuration transform is an info stream that only the
+ // generic handler passes on. All three keep the generic handler.
+ //
+ if (meta.isDatabaseUsed()
+ || meta.isConfigurationFromTransform()
+ || input.isBounded() == PCollection.IsBounded.UNBOUNDED) {
+ new BeamGenericTransformHandler()
+ .handleTransform(
+ log,
+ variables,
+ runConfigurationName,
+ runConfiguration,
+ dataSamplersJson,
+ metadataProvider,
+ pipelineMeta,
+ transformMeta,
+ transformCollectionMap,
+ pipeline,
+ rowMeta,
+ previousTransforms,
+ input,
+ parentLogChannelId);
+ return;
+ }
+
+ long startAt = Const.toLong(variables.resolve(meta.getStartAt()), 1L);
+ long incrementBy = Const.toLong(variables.resolve(meta.getIncrementBy()), 1L);
+ if (incrementBy <= 0) {
+ throw new HopException(
+ "The Add Sequence transform '"
+ + transformMeta.getName()
+ + "' needs a positive increment, not '"
+ + meta.getIncrementBy()
+ + "'");
+ }
+ long maxValue = Const.toLong(variables.resolve(meta.getMaxValue()), -1L);
+
+ String valueName = variables.resolve(meta.getValueName());
+ if (valueName == null || valueName.isEmpty()) {
+ throw new HopException(
+ "The Add Sequence transform '" + transformMeta.getName() + "' has no value name");
+ }
+
+ // The row layout: the incoming fields plus the sequence field. The converter has already
+ // called AddSequenceMeta.getFields(), so rowMeta already carries the sequence field; adding
+ // another one here would emit a duplicate column and shift the value out of the last slot.
+ //
+ IRowMeta outputRowMeta = rowMeta == null ? new RowMeta() : rowMeta.clone();
+ if (outputRowMeta.searchValueMeta(valueName) == null) {
+ IValueMeta valueMeta = ValueMetaFactory.createValueMeta(valueName, IValueMeta.TYPE_INTEGER);
+ valueMeta.setOrigin(transformMeta.getName());
+ outputRowMeta.addValueMeta(valueMeta);
+ }
+
+ // Beam state is per key, so funnel every row onto one key: that is what makes a single
+ // counter produce a single, gap-free, increasing sequence.
+ //
+ PCollection<HopRow> grouped =
+ input
+ .apply(WithKeys.of((Void) null))
+ .setCoder(KvCoder.of(VoidCoder.of(), input.getCoder()))
+ .apply(GroupByKey.create())
+ .apply(Values.create())
+ .apply(Flatten.iterables());
+
+ // A stateful DoFn needs a KvCoder input, and Flatten leaves a PCollection<Iterable<HopRow>>.
+ // Key every row again on the same null key so the whole dataset is one key, which is what
+ // gives a single counter cell.
+ PCollection<KV<Void, HopRow>> keyedRows =
+ grouped
+ .apply(WithKeys.of((Void) null))
+ .setCoder(KvCoder.of(VoidCoder.of(), grouped.getCoder()));
+
+ PCollection<HopRow> afterSequence =
+ keyedRows.apply(
+ ParDo.of(
+ new AddSequenceFn(
+ transformMeta.getName(),
+ JsonRowMeta.toJson(outputRowMeta),
+ valueName,
+ startAt,
+ incrementBy,
+ maxValue)));
+
+ transformCollectionMap.put(transformMeta.getName(), afterSequence);
+ log.logBasic(
+ "Handled transform (ADD SEQUENCE) : "
+ + transformMeta.getName()
+ + ", gets data from "
+ + previousTransforms.size()
+ + " previous transform(s)");
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/bq/BeamBQInputDialog.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/bq/BeamBQInputDialog.java
index 45a658e66d..b553b7d6f9 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/bq/BeamBQInputDialog.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/bq/BeamBQInputDialog.java
@@ -51,6 +51,7 @@ import org.eclipse.swt.layout.FillLayout;
import org.eclipse.swt.layout.FormAttachment;
import org.eclipse.swt.layout.FormData;
import org.eclipse.swt.layout.FormLayout;
+import org.eclipse.swt.widgets.Button;
import org.eclipse.swt.widgets.Composite;
import org.eclipse.swt.widgets.Control;
import org.eclipse.swt.widgets.Label;
@@ -65,6 +66,8 @@ public class BeamBQInputDialog extends BaseTransformDialog {
private TextVar wDatasetId;
private TextVar wTableId;
private TextVar wQuery;
+ private TextVar wQueryLocation;
+ private Button wSkipQueryValidation;
private TableView wFields;
public BeamBQInputDialog(
@@ -169,6 +172,40 @@ public class BeamBQInputDialog extends BaseTransformDialog {
wQuery.setLayoutData(fdQuery);
lastControl = wQuery;
+ // #2416: query location and skipping validation. These only apply to a query, so they sit
+ // directly under the query box.
+ //
+ Label wlQueryLocation = new Label(wContent, SWT.RIGHT);
+ wlQueryLocation.setText(BaseMessages.getString(PKG, "BeamBQInputDialog.QueryLocation"));
+ PropsUi.setLook(wlQueryLocation);
+ FormData fdlQueryLocation = new FormData();
+ fdlQueryLocation.left = new FormAttachment(0, 0);
+ fdlQueryLocation.top = new FormAttachment(lastControl, margin);
+ fdlQueryLocation.right = new FormAttachment(middle, -margin);
+ wlQueryLocation.setLayoutData(fdlQueryLocation);
+ wQueryLocation = new TextVar(variables, wContent, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wQueryLocation);
+ wQueryLocation.setToolTipText(
+ BaseMessages.getString(PKG, "BeamBQInputDialog.QueryLocation.ToolTip"));
+ FormData fdQueryLocation = new FormData();
+ fdQueryLocation.left = new FormAttachment(middle, 0);
+ fdQueryLocation.top = new FormAttachment(wlQueryLocation, 0, SWT.CENTER);
+ fdQueryLocation.right = new FormAttachment(100, 0);
+ wQueryLocation.setLayoutData(fdQueryLocation);
+ lastControl = wQueryLocation;
+
+ wSkipQueryValidation = new Button(wContent, SWT.CHECK);
+ wSkipQueryValidation.setText(
+ BaseMessages.getString(PKG, "BeamBQInputDialog.SkipQueryValidation"));
+ PropsUi.setLook(wSkipQueryValidation);
+ wSkipQueryValidation.setToolTipText(
+ BaseMessages.getString(PKG, "BeamBQInputDialog.SkipQueryValidation.ToolTip"));
+ FormData fdSkipQueryValidation = new FormData();
+ fdSkipQueryValidation.left = new FormAttachment(0, 0);
+ fdSkipQueryValidation.top = new FormAttachment(lastControl, margin);
+ wSkipQueryValidation.setLayoutData(fdSkipQueryValidation);
+ lastControl = wSkipQueryValidation;
+
Label wlFields = new Label(wContent, SWT.LEFT);
wlFields.setText(BaseMessages.getString(PKG, "BeamBQInputDialog.Fields"));
PropsUi.setLook(wlFields);
@@ -287,6 +324,8 @@ public class BeamBQInputDialog extends BaseTransformDialog {
wDatasetId.setText(Const.NVL(input.getDatasetId(), ""));
wTableId.setText(Const.NVL(input.getTableId(), ""));
wQuery.setText(Const.NVL(input.getQuery(), ""));
+ wQueryLocation.setText(Const.NVL(input.getQueryLocation(), ""));
+ wSkipQueryValidation.setSelection(input.isSkippingQueryValidation());
for (int i = 0; i < input.getFields().size(); i++) {
BQField field = input.getFields().get(i);
@@ -323,6 +362,8 @@ public class BeamBQInputDialog extends BaseTransformDialog {
in.setDatasetId(wDatasetId.getText());
in.setTableId(wTableId.getText());
in.setQuery(wQuery.getText());
+ in.setQueryLocation(wQueryLocation.getText());
+ in.setSkippingQueryValidation(wSkipQueryValidation.getSelection());
in.getFields().clear();
for (int i = 0; i < wFields.nrNonEmpty(); i++) {
TableItem item = wFields.getNonEmpty(i);
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/bq/BeamBQInputMeta.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/bq/BeamBQInputMeta.java
index 67f2979dc8..7c0d24df1e 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/bq/BeamBQInputMeta.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/bq/BeamBQInputMeta.java
@@ -64,6 +64,20 @@ public class BeamBQInputMeta extends BaseTransformMeta<BeamBQInput, BeamBQInputD
@HopMetadataProperty(key = "query")
private String query;
+ /**
+ * #2416: the BigQuery dataset location that runs the query, for example {@code EU} or {@code US}.
+ * Leave blank to use the job's own location.
+ */
+ @HopMetadataProperty(key = "query_location")
+ private String queryLocation;
+
+ /**
+ * #2416: do not validate the query before the job starts. Useful for a query that is created or
+ * changed by an earlier step and is therefore not registered when the job is submitted.
+ */
+ @HopMetadataProperty(key = "skip_query_validation")
+ private boolean skippingQueryValidation;
+
@HopMetadataProperty(groupKey = "fields", key = "field")
private List<BQField> fields;
@@ -151,6 +165,8 @@ public class BeamBQInputMeta extends BaseTransformMeta<BeamBQInput, BeamBQInputD
variables.resolve(datasetId),
variables.resolve(tableId),
variables.resolve(query),
+ variables.resolve(queryLocation),
+ skippingQueryValidation,
JsonRowMeta.toJson(outputRowMeta));
PCollection<HopRow> afterInput = pipeline.apply(beamInputTransform);
transformCollectionMap.put(transformMeta.getName(), afterInput);
@@ -221,6 +237,38 @@ public class BeamBQInputMeta extends BaseTransformMeta<BeamBQInput, BeamBQInputD
this.query = query;
}
+ /**
+ * Gets queryLocation
+ *
+ * @return value of queryLocation
+ */
+ public String getQueryLocation() {
+ return queryLocation;
+ }
+
+ /**
+ * @param queryLocation The queryLocation to set
+ */
+ public void setQueryLocation(String queryLocation) {
+ this.queryLocation = queryLocation;
+ }
+
+ /**
+ * Is the query validation skipped
+ *
+ * @return value of skippingQueryValidation
+ */
+ public boolean isSkippingQueryValidation() {
+ return skippingQueryValidation;
+ }
+
+ /**
+ * @param skippingQueryValidation The skippingQueryValidation to set
+ */
+ public void setSkippingQueryValidation(boolean skippingQueryValidation) {
+ this.skippingQueryValidation = skippingQueryValidation;
+ }
+
/**
* Gets fields
*
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInput.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInput.java
new file mode 100644
index 0000000000..25f663fa2f
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInput.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.debezium;
+
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransform;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+public class BeamDebeziumInput extends BaseTransform<BeamDebeziumInputMeta, BeamDebeziumInputData> {
+ public BeamDebeziumInput(
+ TransformMeta transformMeta,
+ BeamDebeziumInputMeta meta,
+ BeamDebeziumInputData data,
+ int copyNr,
+ PipelineMeta pipelineMeta,
+ Pipeline pipeline) {
+ super(transformMeta, meta, data, copyNr, pipelineMeta, pipeline);
+ }
+
+ @Override
+ public boolean processRow() throws HopException {
+ throw new HopException("Beam Debezium input requires a Beam pipeline engine");
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInputData.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInputData.java
new file mode 100644
index 0000000000..ebe66be2c3
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInputData.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.debezium;
+
+import org.apache.hop.pipeline.transform.BaseTransformData;
+
+public class BeamDebeziumInputData extends BaseTransformData {}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInputDialog.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInputDialog.java
new file mode 100644
index 0000000000..94504a5202
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInputDialog.java
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.debezium;
+
+import org.apache.hop.core.util.Utils;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.ui.core.dialog.BaseDialog;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
+import org.apache.hop.ui.core.gui.IGuiPluginCompositeWidgetsListener;
+import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
+import org.eclipse.swt.widgets.Control;
+import org.eclipse.swt.widgets.Shell;
+
+/** Grouped, scrollable annotation-driven editor; OK/Cancel remain outside the content. */
+public class BeamDebeziumInputDialog extends BaseTransformDialog {
+ private final BeamDebeziumInputMeta input;
+ private GuiCompositeWidgets widgets;
+
+ public BeamDebeziumInputDialog(
+ Shell parent, IVariables variables, BeamDebeziumInputMeta meta, PipelineMeta pipelineMeta) {
+ super(parent, variables, meta, pipelineMeta);
+ input = meta;
+ }
+
+ @Override
+ public String open() {
+ createShell(BaseMessages.getString(BeamDebeziumInputMeta.class, "BeamDebeziumInput.Name"));
+ buildButtonBar().ok(e -> ok()).cancel(e -> cancel()).build();
+ changed = input.hasChanged();
+ widgets =
+ GuiCompositeWidgets.addScrolledComposite(
+ shell,
+ variables,
+ wTransformName,
+ wOk,
+ BeamDebeziumInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ input);
+ widgets.setCompositeWidgetsListener(
+ new IGuiPluginCompositeWidgetsListener() {
+ @Override
+ public void widgetsCreated(GuiCompositeWidgets widgets) {}
+
+ @Override
+ public void widgetsPopulated(GuiCompositeWidgets widgets) {}
+
+ @Override
+ public void widgetModified(GuiCompositeWidgets widgets, Control control, String id) {
+ input.setChanged();
+ }
+
+ @Override
+ public void persistContents(GuiCompositeWidgets widgets) {}
+ });
+ focusTransformName();
+ BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
+ return transformName;
+ }
+
+ private void ok() {
+ if (Utils.isEmpty(wTransformName.getText())) {
+ return;
+ }
+ widgets.getWidgetsContents(input, BeamDebeziumInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ transformName = wTransformName.getText();
+ input.setChanged();
+ dispose();
+ }
+
+ private void cancel() {
+ transformName = null;
+ input.setChanged(changed);
+ dispose();
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInputMeta.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInputMeta.java
new file mode 100644
index 0000000000..66df8bfbdd
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/debezium/BeamDebeziumInputMeta.java
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.debezium;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.beam.io.debezium.DebeziumIO;
+import org.apache.beam.sdk.options.StreamingOptions;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.transform.BeamDebeziumInputTransform;
+import org.apache.hop.beam.engines.IBeamPipelineEngineRunConfiguration;
+import org.apache.hop.beam.metadata.RunnerType;
+import org.apache.hop.beam.pipeline.BeamSourceTopology;
+import org.apache.hop.beam.pipeline.IBeamPipelineTransformHandler;
+import org.apache.hop.core.CheckResult;
+import org.apache.hop.core.ICheckResult;
+import org.apache.hop.core.annotations.Transform;
+import org.apache.hop.core.encryption.Encr;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.gui.plugin.GuiElementType;
+import org.apache.hop.core.gui.plugin.GuiPlugin;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.value.ValueMetaString;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.metadata.api.HopMetadataProperty;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransformMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.kafka.connect.source.SourceConnector;
+
+@Getter
+@Setter
+@GuiPlugin
+@Transform(
+ id = "BeamDebeziumInput",
+ name = "i18n::BeamDebeziumInput.Name",
+ description = "i18n::BeamDebeziumInput.Description",
+ image = "beam-debezium-input.svg",
+ categoryDescription = "i18n:org.apache.hop.pipeline.transform:BaseTransform.Category.BigData",
+ keywords = "i18n::BeamDebeziumInput.Keywords",
+ documentationUrl = "/pipeline/transforms/beamdebeziuminput.html",
+ supportedEngines = {
+ "BeamDirectPipelineEngine",
+ "BeamFlinkPipelineEngine",
+ "BeamDataFlowPipelineEngine"
+ })
+public class BeamDebeziumInputMeta
+ extends BaseTransformMeta<BeamDebeziumInput, BeamDebeziumInputData>
+ implements IBeamPipelineTransformHandler {
+ public static final String GUI_PLUGIN_ELEMENT_PARENT_ID = "BEAM_DEBEZIUM_INPUT_OPTIONS";
+
+ @HopMetadataProperty(key = "connector")
+ @GuiWidgetElement(
+ id = "connector",
+ order = "0100",
+ type = GuiElementType.COMBO,
+ comboValuesMethod = "getConnectorNames",
+ label = "i18n::BeamDebeziumInput.Connector.Label",
+ toolTip = "i18n::BeamDebeziumInput.Connector.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamDebeziumInput.Connection.Group",
+ groupOrder = "0100")
+ private String connector = "PostgreSQL";
+
+ @HopMetadataProperty(key = "connector_class")
+ @GuiWidgetElement(
+ id = "connectorClass",
+ order = "0200",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamDebeziumInput.ConnectorClass.Label",
+ toolTip = "i18n::BeamDebeziumInput.ConnectorClass.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamDebeziumInput.Connection.Group",
+ groupOrder = "0100")
+ private String connectorClass = "";
+
+ @HopMetadataProperty(key = "hostname")
+ @GuiWidgetElement(
+ id = "hostname",
+ order = "0300",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamDebeziumInput.Hostname.Label",
+ toolTip = "i18n::BeamDebeziumInput.Hostname.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamDebeziumInput.Connection.Group",
+ groupOrder = "0100")
+ private String hostname = "localhost";
+
+ @HopMetadataProperty(key = "port")
+ @GuiWidgetElement(
+ id = "port",
+ order = "0400",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamDebeziumInput.Port.Label",
+ toolTip = "i18n::BeamDebeziumInput.Port.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamDebeziumInput.Connection.Group",
+ groupOrder = "0100")
+ private String port = "5432";
+
+ @HopMetadataProperty(key = "username")
+ @GuiWidgetElement(
+ id = "username",
+ order = "0500",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamDebeziumInput.Username.Label",
+ toolTip = "i18n::BeamDebeziumInput.Username.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamDebeziumInput.Connection.Group",
+ groupOrder = "0100")
+ private String username = "";
+
+ @HopMetadataProperty(key = "password", password = true)
+ @GuiWidgetElement(
+ id = "password",
+ order = "0600",
+ type = GuiElementType.TEXT,
+ password = true,
+ label = "i18n::BeamDebeziumInput.Password.Label",
+ toolTip = "i18n::BeamDebeziumInput.Password.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamDebeziumInput.Connection.Group",
+ groupOrder = "0100")
+ private String password = "";
+
+ @HopMetadataProperty(key = "connector_properties")
+ @GuiWidgetElement(
+ id = "connectorProperties",
+ order = "0100",
+ type = GuiElementType.MULTI_LINE_TEXT,
+ multiLineTextHeight = 10,
+ label = "i18n::BeamDebeziumInput.Properties.Label",
+ toolTip = "i18n::BeamDebeziumInput.Properties.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamDebeziumInput.Properties.Group",
+ groupOrder = "0200")
+ private String connectorProperties =
+ "{\"database.dbname\":\"inventory\",\"plugin.name\":\"pgoutput\",\"topic.prefix\":\"hop-cdc\"}";
+
+ @HopMetadataProperty(key = "max_records")
+ @GuiWidgetElement(
+ id = "maxRecords",
+ order = "0200",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamDebeziumInput.MaxRecords.Label",
+ toolTip = "i18n::BeamDebeziumInput.MaxRecords.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamDebeziumInput.Output.Group",
+ groupOrder = "0300")
+ private String maxRecords = "";
+
+ @HopMetadataProperty(key = "max_time_ms")
+ @GuiWidgetElement(
+ id = "maxTimeMs",
+ order = "0300",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamDebeziumInput.MaxTime.Label",
+ toolTip = "i18n::BeamDebeziumInput.MaxTime.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamDebeziumInput.Output.Group",
+ groupOrder = "0300")
+ private String maxTimeMs = "";
+
+ @HopMetadataProperty(key = "polling_timeout_ms")
+ @GuiWidgetElement(
+ id = "pollingTimeoutMs",
+ order = "0400",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamDebeziumInput.PollingTimeout.Label",
+ toolTip = "i18n::BeamDebeziumInput.PollingTimeout.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamDebeziumInput.Output.Group",
+ groupOrder = "0300")
+ private String pollingTimeoutMs = "1000";
+
+ @HopMetadataProperty(key = "json_field")
+ @GuiWidgetElement(
+ id = "jsonField",
+ order = "0100",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamDebeziumInput.JsonField.Label",
+ toolTip = "i18n::BeamDebeziumInput.JsonField.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamDebeziumInput.Output.Group",
+ groupOrder = "0300")
+ private String jsonField = "event";
+
+ @Override
+ public void setDefault() {
+ jsonField = "event";
+ }
+
+ @Override
+ public String getDialogClassName() {
+ return BeamDebeziumInputDialog.class.getName();
+ }
+
+ public List<String> getConnectorNames(ILogChannel log, IHopMetadataProvider metadataProvider) {
+ return List.of("PostgreSQL", "Custom");
+ }
+
+ public DebeziumIO.ConnectorConfiguration buildConnectorConfiguration(IVariables variables)
+ throws HopException {
+ require(variables, hostname, "Hostname");
+ require(variables, username, "Username");
+ require(variables, jsonField, "JSON field");
+ long resolvedPort = positive(variables, port, "Port", false, 65535);
+ positive(variables, maxRecords, "Maximum records", true, Integer.MAX_VALUE);
+ positive(variables, maxTimeMs, "Maximum time", true, Long.MAX_VALUE);
+ positive(variables, pollingTimeoutMs, "Polling timeout", true, Long.MAX_VALUE);
+ Map<String, String> properties = resolveProperties(variables);
+ String selected = require(variables, connector, "Connector");
+ if (!"PostgreSQL".equals(selected) && !"Custom".equals(selected)) {
+ throw new HopException("Select PostgreSQL or Custom as the Debezium connector");
+ }
+ if ("PostgreSQL".equals(selected) && !properties.containsKey("database.dbname")) {
+ throw new HopException("PostgreSQL connector properties require database.dbname");
+ }
+ String className =
+ "PostgreSQL".equals(selected)
+ ? "io.debezium.connector.postgresql.PostgresConnector"
+ : require(variables, connectorClass, "Custom connector class");
+ try {
+ Class<?> type =
+ Class.forName(className, true, Thread.currentThread().getContextClassLoader());
+ type.asSubclass(SourceConnector.class);
+ return DebeziumIO.ConnectorConfiguration.create()
+ .withConnectorClass(type)
+ .withHostName(variables.resolve(hostname))
+ .withPort(Long.toString(resolvedPort))
+ .withUsername(variables.resolve(username))
+ .withPassword(
+ Encr.decryptPasswordOptionallyEncrypted(
+ variables.resolve(password == null ? "" : password)))
+ .withConnectionProperties(properties);
+ } catch (ReflectiveOperationException | LinkageError | ClassCastException e) {
+ throw new HopException(
+ "Unable to load Debezium connector "
+ + className
+ + "; add the compatible connector and driver jars to Hop and the Beam worker classpath",
+ e);
+ }
+ }
+
+ private Map<String, String> resolveProperties(IVariables variables) throws HopException {
+ JsonNode properties;
+ try {
+ properties =
+ new ObjectMapper()
+ .readTree(
+ variables.resolve(connectorProperties == null ? "{}" : connectorProperties));
+ } catch (Exception e) {
+ // Do not put the JSON or parser's source excerpt (which may contain secrets) into the log.
+ throw new HopException("Connector properties must be a JSON object with string values");
+ }
+ if (properties == null || !properties.isObject()) {
+ throw new HopException("Connector properties must be a JSON object with string values");
+ }
+ Map<String, String> config = new LinkedHashMap<>();
+ var entries = properties.fields();
+ while (entries.hasNext()) {
+ var entry = entries.next();
+ if (entry.getKey().isBlank() || !entry.getValue().isTextual()) {
+ throw new HopException("Connector properties must have non-empty keys and string values");
+ }
+ if (Set.of(
+ "connector.class",
+ "database.hostname",
+ "database.port",
+ "database.user",
+ "database.password")
+ .contains(entry.getKey())) {
+ throw new HopException(
+ "Use the Connection tab instead of connector property " + entry.getKey());
+ }
+ config.put(entry.getKey(), entry.getValue().textValue());
+ }
+ return config;
+ }
+
+ private static String require(IVariables variables, String value, String label)
+ throws HopException {
+ String resolved = variables.resolve(value);
+ if (resolved == null || resolved.isBlank()) {
+ throw new HopException(label + " is required");
+ }
+ return resolved;
+ }
+
+ private static long positive(
+ IVariables variables, String value, String label, boolean optional, long maximum)
+ throws HopException {
+ String resolved = variables.resolve(value);
+ if (optional && (resolved == null || resolved.isBlank())) {
+ return 0;
+ }
+ try {
+ long number = Long.parseLong(resolved);
+ if (number > 0 && number <= maximum) {
+ return number;
+ }
+ } catch (NumberFormatException e) {
+ // Report the setting name, not the potentially secret resolved value.
+ }
+ throw new HopException(label + " must be a positive integer no greater than " + maximum);
+ }
+
+ @Override
+ public boolean consumesMainInput() {
+ return false;
+ }
+
+ @Override
+ public boolean canStartWithoutInput() {
+ return true;
+ }
+
+ @Override
+ public boolean isInput() {
+ return true;
+ }
+
+ @Override
+ public boolean isOutput() {
+ return false;
+ }
+
+ @Override
+ public void getFields(
+ IRowMeta row,
+ String name,
+ IRowMeta[] info,
+ TransformMeta next,
+ IVariables variables,
+ IHopMetadataProvider metadataProvider) {
+ row.clear();
+ ValueMetaString value = new ValueMetaString(variables.resolve(jsonField));
+ value.setOrigin(name);
+ row.addValueMeta(value);
+ }
+
+ @Override
+ public void check(
+ List<ICheckResult> remarks,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ IRowMeta prev,
+ String[] input,
+ String[] output,
+ IRowMeta info,
+ IVariables variables,
+ IHopMetadataProvider metadataProvider) {
+ try {
+ buildConnectorConfiguration(variables);
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_OK,
+ BaseMessages.getString(BeamDebeziumInputMeta.class, "BeamDebeziumInput.Check.Valid"),
+ transformMeta));
+ } catch (HopException e) {
+ remarks.add(new CheckResult(ICheckResult.TYPE_RESULT_ERROR, e.getMessage(), transformMeta));
+ }
+ if (input != null && input.length > 0) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(
+ BeamDebeziumInputMeta.class, "BeamDebeziumInput.Check.NoIncoming"),
+ transformMeta));
+ }
+ }
+
+ @Override
+ public void handleTransform(
+ ILogChannel log,
+ IVariables variables,
+ String runConfigurationName,
+ IBeamPipelineEngineRunConfiguration runConfiguration,
+ String dataSamplersJson,
+ IHopMetadataProvider metadataProvider,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ Map<String, PCollection<HopRow>> transformCollectionMap,
+ org.apache.beam.sdk.Pipeline pipeline,
+ IRowMeta rowMeta,
+ List<TransformMeta> previousTransforms,
+ PCollection<HopRow> input,
+ String parentLogChannelId)
+ throws HopException {
+ BeamSourceTopology.rejectIncomingRows(
+ pipelineMeta,
+ transformMeta,
+ previousTransforms,
+ input,
+ "Beam Debezium input is a source and does not accept incoming rows");
+ Class<?> runner = pipeline.getOptions().getRunner();
+ if ((runConfiguration != null && runConfiguration.getRunnerType() == RunnerType.Spark)
+ || (runner != null && runner.getName().startsWith("org.apache.beam.runners.spark."))) {
+ throw new HopException(
+ "Beam Spark does not support the unbounded Splittable DoFn used by DebeziumIO; use Direct, Flink or Dataflow");
+ }
+ pipeline.getOptions().as(StreamingOptions.class).setStreaming(true);
+ DebeziumIO.Read<String> read =
+ DebeziumIO.readAsJson().withConnectorConfiguration(buildConnectorConfiguration(variables));
+ String records = variables.resolve(maxRecords);
+ if (records != null && !records.isBlank()) {
+ read = read.withMaxNumberOfRecords(Integer.valueOf(records));
+ }
+ String time = variables.resolve(maxTimeMs);
+ if (time != null && !time.isBlank()) {
+ read = read.withMaxTimeToRun(Long.valueOf(time));
+ }
+ String polling = variables.resolve(pollingTimeoutMs);
+ if (polling != null && !polling.isBlank()) {
+ read = read.withPollingTimeout(Long.valueOf(polling));
+ }
+ PCollection<HopRow> output =
+ pipeline.apply(new BeamDebeziumInputTransform(transformMeta.getName(), read));
+ transformCollectionMap.put(transformMeta.getName(), output);
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchConfig.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchConfig.java
new file mode 100644
index 0000000000..2bf4838dd5
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchConfig.java
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.transforms.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO;
+import org.apache.hop.core.encryption.Encr;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.variables.IVariables;
+
+/** Shared configuration; HTTPS keeps the client's verified default trust configuration. */
+public final class BeamElasticsearchConfig {
+ /**
+ * Jackson 2.21.5 parses JSON numbers as {@code double} unless told otherwise, and {@code double}
+ * turns values past about 1e308 into {@code Infinity} and drops digits past the 53-bit mantissa.
+ * BigDecimal and BigInteger keep the token's value. {@code writeValueAsString} on this mapper
+ * writes those nodes back; {@code JsonNode.toString()} uses a different mapper.
+ */
+ private static final ObjectMapper JSON_MAPPER =
+ new ObjectMapper()
+ .enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS)
+ .enable(DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS)
+ .enable(DeserializationFeature.USE_BIG_INTEGER_FOR_INTS)
+ .enable(JsonGenerator.Feature.WRITE_BIGDECIMAL_AS_PLAIN);
+
+ private BeamElasticsearchConfig() {}
+
+ public static ElasticsearchIO.ConnectionConfiguration connection(
+ IVariables variables,
+ String hosts,
+ String index,
+ String type,
+ String username,
+ String password)
+ throws HopException {
+ String[] addresses = required(variables, hosts, "Hosts").split(",", -1);
+ for (int i = 0; i < addresses.length; i++) {
+ addresses[i] = addresses[i].trim();
+ try {
+ URI uri = new URI(addresses[i]);
+ if (!("https".equalsIgnoreCase(uri.getScheme()) || "http".equalsIgnoreCase(uri.getScheme()))
+ || uri.getHost() == null
+ || uri.getUserInfo() != null
+ || uri.getQuery() != null
+ || uri.getFragment() != null
+ || (uri.getPath() != null && !uri.getPath().isEmpty() && !"/".equals(uri.getPath()))
+ || uri.getPort() > 65535
+ || uri.getPort() == 0) {
+ throw new HopException(
+ "Hosts must be HTTP(S) origins without credentials, paths, queries or fragments");
+ }
+ } catch (java.net.URISyntaxException e) {
+ // Do not include a URL: it might contain credentials.
+ throw new HopException("Invalid Elasticsearch host URI");
+ }
+ }
+ String resolvedIndex = required(variables, index, "Index");
+ String resolvedType = optional(variables, type, "Document type");
+ if (resolvedIndex.matches(".*[/\\?#\\s].*") || resolvedType.matches(".*[/\\?#\\s].*")) {
+ throw new HopException("Index and document type must not contain paths or whitespace");
+ }
+ String user = optional(variables, username, "Username");
+ String secret = password == null ? "" : variables.resolve(password);
+ if (secret == null) {
+ secret = "";
+ }
+ checkResolved(secret, "Password");
+ if (user.isBlank() != secret.isEmpty()) {
+ throw new HopException("Basic authentication requires both username and password");
+ }
+ ElasticsearchIO.ConnectionConfiguration connection =
+ ElasticsearchIO.ConnectionConfiguration.create(addresses, resolvedIndex, resolvedType);
+ if (!user.isBlank()) {
+ connection =
+ connection
+ .withUsername(user)
+ .withPassword(Encr.decryptPasswordOptionallyEncrypted(secret));
+ }
+ return connection;
+ }
+
+ public static long positiveLong(IVariables variables, String value, String label)
+ throws HopException {
+ try {
+ long result = Long.parseLong(required(variables, value, label));
+ if (result > 0) {
+ return result;
+ }
+ } catch (NumberFormatException e) {
+ /* Fall through to the configuration error. */
+ }
+ throw new HopException(label + " must be a positive integer");
+ }
+
+ public static String required(IVariables variables, String value, String label)
+ throws HopException {
+ String resolved = optional(variables, value, label);
+ if (resolved.isBlank()) {
+ throw new HopException(label + " is required");
+ }
+ return resolved;
+ }
+
+ public static String optional(IVariables variables, String value, String label)
+ throws HopException {
+ String resolved = value == null ? "" : variables.resolve(value);
+ if (resolved == null) {
+ resolved = "";
+ }
+ checkResolved(resolved, label);
+ return resolved.trim();
+ }
+
+ public static JsonNode jsonObject(String value, String label) throws HopException {
+ try {
+ JsonNode json = JSON_MAPPER.readTree(value);
+ if (json == null || !json.isObject()) {
+ throw new HopException(label + " must be a JSON object");
+ }
+ return json;
+ } catch (IOException e) {
+ // Parser exceptions include document contents; do not disclose them in errors.
+ throw new HopException(label + " must be a valid JSON object");
+ }
+ }
+
+ /** One compact JSON object. Numeric tokens keep their value; only whitespace is normalized. */
+ public static String compactJsonObject(String value, String label) throws HopException {
+ try {
+ return JSON_MAPPER.writeValueAsString(jsonObject(value, label));
+ } catch (IOException e) {
+ throw new HopException(label + " must be a valid JSON object");
+ }
+ }
+
+ private static void checkResolved(String value, String label) throws HopException {
+ if (value.contains("${") || value.matches(".*%%[^%]+%%.*")) {
+ throw new HopException(label + " contains an unresolved variable");
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInput.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInput.java
new file mode 100644
index 0000000000..ab2bf5fed1
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInput.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.transforms.elasticsearch;
+
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransform;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+public class BeamElasticsearchInput
+ extends BaseTransform<BeamElasticsearchInputMeta, BeamElasticsearchInputData> {
+ public BeamElasticsearchInput(
+ TransformMeta transformMeta,
+ BeamElasticsearchInputMeta meta,
+ BeamElasticsearchInputData data,
+ int copyNr,
+ PipelineMeta pipelineMeta,
+ Pipeline pipeline) {
+ super(transformMeta, meta, data, copyNr, pipelineMeta, pipeline);
+ }
+
+ @Override
+ public boolean processRow() throws HopException {
+ throw new HopException("Beam Elasticsearch input can only run in a Beam pipeline");
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInputData.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInputData.java
new file mode 100644
index 0000000000..f2481f7789
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInputData.java
@@ -0,0 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.transforms.elasticsearch;
+
+import org.apache.hop.pipeline.transform.BaseTransformData;
+
+public class BeamElasticsearchInputData extends BaseTransformData {}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInputDialog.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInputDialog.java
new file mode 100644
index 0000000000..cb320c2164
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInputDialog.java
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.transforms.elasticsearch;
+
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.ui.core.dialog.BaseDialog;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
+import org.apache.hop.ui.core.gui.IGuiPluginCompositeWidgetsListener;
+import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
+import org.eclipse.swt.widgets.Control;
+import org.eclipse.swt.widgets.Shell;
+
+public class BeamElasticsearchInputDialog extends BaseTransformDialog {
+ private final BeamElasticsearchInputMeta input;
+ private GuiCompositeWidgets widgets;
+
+ public BeamElasticsearchInputDialog(
+ Shell parent,
+ IVariables variables,
+ BeamElasticsearchInputMeta meta,
+ PipelineMeta pipelineMeta) {
+ super(parent, variables, meta, pipelineMeta);
+ input = meta;
+ }
+
+ @Override
+ public String open() {
+ createShell(
+ BaseMessages.getString(BeamElasticsearchInputMeta.class, "BeamElasticsearchInput.Name"));
+ buildButtonBar().ok(e -> ok()).cancel(e -> cancel()).build();
+ changed = input.hasChanged();
+ widgets =
+ GuiCompositeWidgets.addScrolledComposite(
+ shell,
+ variables,
+ wTransformName,
+ wOk,
+ BeamElasticsearchInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ input);
+ widgets.setCompositeWidgetsListener(
+ new IGuiPluginCompositeWidgetsListener() {
+ @Override
+ public void widgetsCreated(GuiCompositeWidgets compositeWidgets) {}
+
+ @Override
+ public void widgetsPopulated(GuiCompositeWidgets compositeWidgets) {}
+
+ @Override
+ public void widgetModified(
+ GuiCompositeWidgets compositeWidgets, Control control, String widgetId) {
+ input.setChanged();
+ }
+
+ @Override
+ public void persistContents(GuiCompositeWidgets compositeWidgets) {}
+ });
+ focusTransformName();
+ BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
+ return transformName;
+ }
+
+ private void cancel() {
+ transformName = null;
+ input.setChanged(changed);
+ dispose();
+ }
+
+ private void ok() {
+ if (wTransformName.getText().isBlank()) {
+ return;
+ }
+ widgets.getWidgetsContents(input, BeamElasticsearchInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ transformName = wTransformName.getText();
+ input.setChanged();
+ dispose();
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInputMeta.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInputMeta.java
new file mode 100644
index 0000000000..ab370c519d
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchInputMeta.java
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.transforms.elasticsearch;
+
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.transform.BeamElasticsearchInputTransform;
+import org.apache.hop.beam.engines.IBeamPipelineEngineRunConfiguration;
+import org.apache.hop.beam.pipeline.BeamSourceTopology;
+import org.apache.hop.beam.pipeline.IBeamPipelineTransformHandler;
+import org.apache.hop.core.annotations.Transform;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.exception.HopTransformException;
+import org.apache.hop.core.gui.plugin.GuiElementType;
+import org.apache.hop.core.gui.plugin.GuiPlugin;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.JsonRowMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaString;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.metadata.api.HopMetadataProperty;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransformMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+@Transform(
+ id = "BeamElasticsearchInput",
+ name = "i18n::BeamElasticsearchInput.Name",
+ description = "i18n::BeamElasticsearchInput.Description",
+ image = "beam-elasticsearch-input.svg",
+ categoryDescription = "i18n:org.apache.hop.pipeline.transform:BaseTransform.Category.BigData",
+ keywords = "i18n::BeamElasticsearch.keyword",
+ documentationUrl = "/pipeline/transforms/beamelasticsearchinput.html",
+ supportedEngines = {"Beam*"})
+@GuiPlugin
+@Getter
+@Setter
+public class BeamElasticsearchInputMeta
+ extends BaseTransformMeta<BeamElasticsearchInput, BeamElasticsearchInputData>
+ implements IBeamPipelineTransformHandler {
+ public static final String GUI_PLUGIN_ELEMENT_PARENT_ID = "BeamElasticsearchInput";
+ public static final String WIDGET_SCROLL_KEEPALIVE = "scrollKeepalive";
+
+ @GuiWidgetElement(
+ id = WIDGET_SCROLL_KEEPALIVE,
+ order = "0900",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.scrollKeepalive.Label",
+ toolTip = "i18n::BeamElasticsearch.scrollKeepalive.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Documents.Group",
+ groupOrder = "0200")
+ @HopMetadataProperty(key = "scroll_keepalive")
+ private String scrollKeepalive = "5m";
+
+ public static final String WIDGET_HOSTS = "hosts";
+ public static final String WIDGET_INDEX = "index";
+ public static final String WIDGET_DOCUMENT_TYPE = "documentType";
+ public static final String WIDGET_USERNAME = "username";
+ public static final String WIDGET_PASSWORD = "password";
+ public static final String WIDGET_JSON_FIELD = "jsonField";
+ public static final String WIDGET_QUERY = "query";
+
+ @GuiWidgetElement(
+ id = WIDGET_HOSTS,
+ order = "0000",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.hosts.Label",
+ toolTip = "i18n::BeamElasticsearch.hosts.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Connection.Group",
+ groupOrder = "0100")
+ @HopMetadataProperty
+ private String hosts;
+
+ @GuiWidgetElement(
+ id = WIDGET_INDEX,
+ order = "0001",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.index.Label",
+ toolTip = "i18n::BeamElasticsearch.index.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Connection.Group",
+ groupOrder = "0100")
+ @HopMetadataProperty
+ private String index;
+
+ @GuiWidgetElement(
+ id = WIDGET_DOCUMENT_TYPE,
+ order = "0002",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.documentType.Label",
+ toolTip = "i18n::BeamElasticsearch.documentType.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Connection.Group",
+ groupOrder = "0100")
+ @HopMetadataProperty(key = "document_type")
+ private String documentType;
+
+ @GuiWidgetElement(
+ id = WIDGET_USERNAME,
+ order = "0003",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.username.Label",
+ toolTip = "i18n::BeamElasticsearch.username.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Connection.Group",
+ groupOrder = "0100")
+ @HopMetadataProperty
+ private String username;
+
+ @GuiWidgetElement(
+ id = WIDGET_PASSWORD,
+ order = "0004",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.password.Label",
+ toolTip = "i18n::BeamElasticsearch.password.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Connection.Group",
+ groupOrder = "0100",
+ password = true)
+ @HopMetadataProperty(password = true)
+ private String password;
+
+ @GuiWidgetElement(
+ id = WIDGET_QUERY,
+ order = "0006",
+ type = GuiElementType.MULTI_LINE_TEXT,
+ label = "i18n::BeamElasticsearch.query.Label",
+ toolTip = "i18n::BeamElasticsearch.query.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Documents.Group",
+ groupOrder = "0200",
+ multiLineTextHeight = 5)
+ @HopMetadataProperty
+ private String query;
+
+ @GuiWidgetElement(
+ id = WIDGET_JSON_FIELD,
+ order = "0005",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.jsonField.Label",
+ toolTip = "i18n::BeamElasticsearch.jsonField.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Documents.Group",
+ groupOrder = "0200")
+ @HopMetadataProperty(key = "json_field")
+ private String jsonField = "json";
+
+ @Override
+ public boolean consumesMainInput() {
+ return false;
+ }
+
+ @Override
+ public boolean canStartWithoutInput() {
+ return true;
+ }
+
+ public ElasticsearchIO.ConnectionConfiguration connectionConfiguration(IVariables variables)
+ throws HopException {
+ return BeamElasticsearchConfig.connection(
+ variables, hosts, index, documentType, username, password);
+ }
+
+ public ElasticsearchIO.Read createRead(IVariables variables) throws HopException {
+ String resolvedQuery = BeamElasticsearchConfig.optional(variables, query, "Query");
+ if (!resolvedQuery.isBlank()) {
+ BeamElasticsearchConfig.jsonObject(resolvedQuery, "Query");
+ }
+ String keepalive =
+ BeamElasticsearchConfig.required(variables, scrollKeepalive, "Scroll keepalive");
+ if (!keepalive.matches("[1-9][0-9]*(ms|s|m|h|d)")) {
+ throw new HopException("Scroll keepalive must be a positive duration (ms, s, m, h or d)");
+ }
+ ElasticsearchIO.Read read =
+ ElasticsearchIO.read()
+ .withConnectionConfiguration(connectionConfiguration(variables))
+ .withScrollKeepalive(keepalive);
+ return resolvedQuery.isBlank() ? read : read.withQuery(resolvedQuery);
+ }
+
+ @Override
+ public String getDialogClassName() {
+ return "org.apache.hop.beam.transforms.elasticsearch.BeamElasticsearchInputDialog";
+ }
+
+ @Override
+ public boolean isInput() {
+ return true;
+ }
+
+ @Override
+ public boolean isOutput() {
+ return false;
+ }
+
+ @Override
+ public void getFields(
+ IRowMeta rowMeta,
+ String name,
+ IRowMeta[] info,
+ TransformMeta nextTransform,
+ IVariables variables,
+ IHopMetadataProvider metadataProvider)
+ throws HopTransformException {
+ String resolvedField;
+ try {
+ resolvedField = BeamElasticsearchConfig.required(variables, jsonField, "JSON field");
+ } catch (HopException e) {
+ throw new HopTransformException(e.getMessage(), e);
+ }
+ rowMeta.clear();
+ ValueMetaString field = new ValueMetaString(resolvedField);
+ field.setOrigin(name);
+ rowMeta.addValueMeta(field);
+ }
+
+ @Override
+ public void handleTransform(
+ ILogChannel log,
+ IVariables variables,
+ String runConfigurationName,
+ IBeamPipelineEngineRunConfiguration runConfiguration,
+ String dataSamplersJson,
+ IHopMetadataProvider metadataProvider,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ Map<String, PCollection<HopRow>> transformCollectionMap,
+ org.apache.beam.sdk.Pipeline pipeline,
+ IRowMeta rowMeta,
+ List<TransformMeta> previousTransforms,
+ PCollection<HopRow> input,
+ String parentLogChannelId)
+ throws HopException {
+ BeamSourceTopology.rejectIncomingRows(
+ pipelineMeta,
+ transformMeta,
+ previousTransforms,
+ input,
+ "Elasticsearch input is a source and does not accept incoming rows");
+ ElasticsearchIO.Read readIo = createRead(variables);
+ RowMeta outputMeta = new RowMeta();
+ getFields(outputMeta, transformMeta.getName(), null, null, variables, metadataProvider);
+ BeamElasticsearchInputTransform read =
+ new BeamElasticsearchInputTransform(
+ transformMeta.getName(), readIo, JsonRowMeta.toJson(outputMeta));
+ transformCollectionMap.put(
+ transformMeta.getName(), pipeline.apply(transformMeta.getName(), read));
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutput.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutput.java
new file mode 100644
index 0000000000..0eba2490a6
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutput.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.transforms.elasticsearch;
+
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransform;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+public class BeamElasticsearchOutput
+ extends BaseTransform<BeamElasticsearchOutputMeta, BeamElasticsearchOutputData> {
+ public BeamElasticsearchOutput(
+ TransformMeta transformMeta,
+ BeamElasticsearchOutputMeta meta,
+ BeamElasticsearchOutputData data,
+ int copyNr,
+ PipelineMeta pipelineMeta,
+ Pipeline pipeline) {
+ super(transformMeta, meta, data, copyNr, pipelineMeta, pipeline);
+ }
+
+ @Override
+ public boolean processRow() throws HopException {
+ throw new HopException("Beam Elasticsearch output can only run in a Beam pipeline");
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutputData.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutputData.java
new file mode 100644
index 0000000000..0aea411f93
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutputData.java
@@ -0,0 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.transforms.elasticsearch;
+
+import org.apache.hop.pipeline.transform.BaseTransformData;
+
+public class BeamElasticsearchOutputData extends BaseTransformData {}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutputDialog.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutputDialog.java
new file mode 100644
index 0000000000..2f81a640ed
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutputDialog.java
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.transforms.elasticsearch;
+
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.ui.core.dialog.BaseDialog;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
+import org.apache.hop.ui.core.gui.IGuiPluginCompositeWidgetsListener;
+import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
+import org.eclipse.swt.widgets.Control;
+import org.eclipse.swt.widgets.Shell;
+
+public class BeamElasticsearchOutputDialog extends BaseTransformDialog {
+ private final BeamElasticsearchOutputMeta input;
+ private GuiCompositeWidgets widgets;
+
+ public BeamElasticsearchOutputDialog(
+ Shell parent,
+ IVariables variables,
+ BeamElasticsearchOutputMeta meta,
+ PipelineMeta pipelineMeta) {
+ super(parent, variables, meta, pipelineMeta);
+ input = meta;
+ }
+
+ @Override
+ public String open() {
+ createShell(
+ BaseMessages.getString(BeamElasticsearchOutputMeta.class, "BeamElasticsearchOutput.Name"));
+ buildButtonBar().ok(e -> ok()).cancel(e -> cancel()).build();
+ changed = input.hasChanged();
+ widgets =
+ GuiCompositeWidgets.addScrolledComposite(
+ shell,
+ variables,
+ wTransformName,
+ wOk,
+ BeamElasticsearchOutputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ input);
+ widgets.setCompositeWidgetsListener(
+ new IGuiPluginCompositeWidgetsListener() {
+ @Override
+ public void widgetsCreated(GuiCompositeWidgets compositeWidgets) {}
+
+ @Override
+ public void widgetsPopulated(GuiCompositeWidgets compositeWidgets) {}
+
+ @Override
+ public void widgetModified(
+ GuiCompositeWidgets compositeWidgets, Control control, String widgetId) {
+ input.setChanged();
+ }
+
+ @Override
+ public void persistContents(GuiCompositeWidgets compositeWidgets) {}
+ });
+ focusTransformName();
+ BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
+ return transformName;
+ }
+
+ private void cancel() {
+ transformName = null;
+ input.setChanged(changed);
+ dispose();
+ }
+
+ private void ok() {
+ if (wTransformName.getText().isBlank()) {
+ return;
+ }
+ widgets.getWidgetsContents(input, BeamElasticsearchOutputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ transformName = wTransformName.getText();
+ input.setChanged();
+ dispose();
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutputMeta.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutputMeta.java
new file mode 100644
index 0000000000..3e61c39f6c
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/elasticsearch/BeamElasticsearchOutputMeta.java
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.beam.transforms.elasticsearch;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.transform.BeamElasticsearchOutputTransform;
+import org.apache.hop.beam.engines.IBeamPipelineEngineRunConfiguration;
+import org.apache.hop.beam.pipeline.IBeamPipelineTransformHandler;
+import org.apache.hop.core.annotations.Transform;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.gui.plugin.GuiElementType;
+import org.apache.hop.core.gui.plugin.GuiPlugin;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.metadata.api.HopMetadataProperty;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransformMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+@Transform(
+ id = "BeamElasticsearchOutput",
+ name = "i18n::BeamElasticsearchOutput.Name",
+ description = "i18n::BeamElasticsearchOutput.Description",
+ image = "beam-elasticsearch-output.svg",
+ categoryDescription = "i18n:org.apache.hop.pipeline.transform:BaseTransform.Category.BigData",
+ keywords = "i18n::BeamElasticsearch.keyword",
+ documentationUrl = "/pipeline/transforms/beamelasticsearchoutput.html",
+ supportedEngines = {"Beam*"})
+@GuiPlugin
+@Getter
+@Setter
+public class BeamElasticsearchOutputMeta
+ extends BaseTransformMeta<BeamElasticsearchOutput, BeamElasticsearchOutputData>
+ implements IBeamPipelineTransformHandler {
+ public static final String GUI_PLUGIN_ELEMENT_PARENT_ID = "BeamElasticsearchOutput";
+ public static final String WIDGET_MAX_BATCH_SIZE = "maxBatchSize";
+
+ @GuiWidgetElement(
+ id = WIDGET_MAX_BATCH_SIZE,
+ order = "0900",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.maxBatchSize.Label",
+ toolTip = "i18n::BeamElasticsearch.maxBatchSize.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Documents.Group",
+ groupOrder = "0200")
+ @HopMetadataProperty(key = "max_batch_size")
+ private String maxBatchSize = "1000";
+
+ public static final String WIDGET_MAX_BATCH_BYTES = "maxBatchBytes";
+
+ @GuiWidgetElement(
+ id = WIDGET_MAX_BATCH_BYTES,
+ order = "1000",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.maxBatchBytes.Label",
+ toolTip = "i18n::BeamElasticsearch.maxBatchBytes.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Documents.Group",
+ groupOrder = "0200")
+ @HopMetadataProperty(key = "max_batch_bytes")
+ private String maxBatchBytes = "5242880";
+
+ public static final String WIDGET_HOSTS = "hosts";
+ public static final String WIDGET_INDEX = "index";
+ public static final String WIDGET_DOCUMENT_TYPE = "documentType";
+ public static final String WIDGET_USERNAME = "username";
+ public static final String WIDGET_PASSWORD = "password";
+ public static final String WIDGET_JSON_FIELD = "jsonField";
+
+ @GuiWidgetElement(
+ id = WIDGET_HOSTS,
+ order = "0000",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.hosts.Label",
+ toolTip = "i18n::BeamElasticsearch.hosts.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Connection.Group",
+ groupOrder = "0100")
+ @HopMetadataProperty
+ private String hosts;
+
+ @GuiWidgetElement(
+ id = WIDGET_INDEX,
+ order = "0001",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.index.Label",
+ toolTip = "i18n::BeamElasticsearch.index.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Connection.Group",
+ groupOrder = "0100")
+ @HopMetadataProperty
+ private String index;
+
+ @GuiWidgetElement(
+ id = WIDGET_DOCUMENT_TYPE,
+ order = "0002",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.documentType.Label",
+ toolTip = "i18n::BeamElasticsearch.documentType.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Connection.Group",
+ groupOrder = "0100")
+ @HopMetadataProperty(key = "document_type")
+ private String documentType;
+
+ @GuiWidgetElement(
+ id = WIDGET_USERNAME,
+ order = "0003",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.username.Label",
+ toolTip = "i18n::BeamElasticsearch.username.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Connection.Group",
+ groupOrder = "0100")
+ @HopMetadataProperty
+ private String username;
+
+ @GuiWidgetElement(
+ id = WIDGET_PASSWORD,
+ order = "0004",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.password.Label",
+ toolTip = "i18n::BeamElasticsearch.password.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Connection.Group",
+ groupOrder = "0100",
+ password = true)
+ @HopMetadataProperty(password = true)
+ private String password;
+
+ @GuiWidgetElement(
+ id = WIDGET_JSON_FIELD,
+ order = "0005",
+ type = GuiElementType.TEXT,
+ label = "i18n::BeamElasticsearch.jsonField.Label",
+ toolTip = "i18n::BeamElasticsearch.jsonField.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamElasticsearch.Documents.Group",
+ groupOrder = "0200")
+ @HopMetadataProperty(key = "json_field")
+ private String jsonField = "json";
+
+ public ElasticsearchIO.ConnectionConfiguration connectionConfiguration(IVariables variables)
+ throws HopException {
+ return BeamElasticsearchConfig.connection(
+ variables, hosts, index, documentType, username, password);
+ }
+
+ public ElasticsearchIO.Write createWrite(IVariables variables) throws HopException {
+ return ElasticsearchIO.write()
+ .withConnectionConfiguration(connectionConfiguration(variables))
+ .withMaxBatchSize(
+ BeamElasticsearchConfig.positiveLong(variables, maxBatchSize, "Maximum batch size"))
+ .withMaxBatchSizeBytes(
+ BeamElasticsearchConfig.positiveLong(variables, maxBatchBytes, "Maximum batch bytes"));
+ }
+
+ @Override
+ public String getDialogClassName() {
+ return "org.apache.hop.beam.transforms.elasticsearch.BeamElasticsearchOutputDialog";
+ }
+
+ @Override
+ public boolean isInput() {
+ return false;
+ }
+
+ @Override
+ public boolean isOutput() {
+ return true;
+ }
+
+ @Override
+ public void getFields(
+ IRowMeta rowMeta,
+ String name,
+ IRowMeta[] info,
+ TransformMeta next,
+ IVariables variables,
+ IHopMetadataProvider provider) {
+ rowMeta.clear();
+ }
+
+ @Override
+ public void handleTransform(
+ ILogChannel log,
+ IVariables variables,
+ String runConfigurationName,
+ IBeamPipelineEngineRunConfiguration runConfiguration,
+ String dataSamplersJson,
+ IHopMetadataProvider metadataProvider,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ Map<String, PCollection<HopRow>> transformCollectionMap,
+ org.apache.beam.sdk.Pipeline pipeline,
+ IRowMeta rowMeta,
+ List<TransformMeta> previousTransforms,
+ PCollection<HopRow> input,
+ String parentLogChannelId)
+ throws HopException {
+ if (input == null) {
+ throw new HopException("Elasticsearch output requires incoming rows");
+ }
+ String resolvedField = BeamElasticsearchConfig.required(variables, jsonField, "JSON field");
+ int fieldIndex = rowMeta.indexOfValue(resolvedField);
+ if (fieldIndex < 0 || !rowMeta.getValueMeta(fieldIndex).isString()) {
+ throw new HopException(
+ "JSON field must exist in incoming rows and have type String: " + resolvedField);
+ }
+ ElasticsearchIO.Write writeIo = createWrite(variables);
+ String rowMetaXml;
+ try {
+ rowMetaXml = rowMeta.getMetaXml();
+ } catch (IOException e) {
+ throw new HopException("Unable to serialize Elasticsearch input fields", e);
+ }
+ input.apply(
+ transformMeta.getName(),
+ new BeamElasticsearchOutputTransform(
+ transformMeta.getName(), writeIo, resolvedField, rowMetaXml));
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/io/BeamOutputDialog.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/io/BeamOutputDialog.java
index fbc274e6c5..90b9bd251f 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/io/BeamOutputDialog.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/io/BeamOutputDialog.java
@@ -17,6 +17,7 @@
package org.apache.hop.beam.transforms.io;
+import org.apache.beam.sdk.io.Compression;
import org.apache.hop.beam.metadata.FileDefinition;
import org.apache.hop.core.Const;
import org.apache.hop.core.util.Utils;
@@ -25,6 +26,7 @@ import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.ui.core.PropsUi;
import org.apache.hop.ui.core.dialog.BaseDialog;
+import org.apache.hop.ui.core.widget.ComboVar;
import org.apache.hop.ui.core.widget.MetaSelectionLine;
import org.apache.hop.ui.core.widget.TextVar;
import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
@@ -52,6 +54,7 @@ public class BeamOutputDialog extends BaseTransformDialog {
private TextVar wFilePrefix;
private TextVar wFileSuffix;
private Button wWindowed;
+ private ComboVar wCompression;
public BeamOutputDialog(
Shell parent, IVariables variables, BeamOutputMeta transformMeta, PipelineMeta pipelineMeta) {
@@ -151,6 +154,31 @@ public class BeamOutputDialog extends BaseTransformDialog {
wWindowed.setLayoutData(fdWindowed);
lastControl = wWindowed;
+ // #2337: compression, a plain combo of the Beam Compression constants.
+ //
+ Label wlCompression = new Label(wContent, SWT.RIGHT);
+ wlCompression.setText(BaseMessages.getString(PKG, "BeamOutputDialog.Compression"));
+ PropsUi.setLook(wlCompression);
+ FormData fdlCompression = new FormData();
+ fdlCompression.left = new FormAttachment(0, 0);
+ fdlCompression.top = new FormAttachment(lastControl, margin);
+ fdlCompression.right = new FormAttachment(middle, -margin);
+ wlCompression.setLayoutData(fdlCompression);
+
+ wCompression = new ComboVar(variables, wContent, SWT.READ_ONLY);
+ PropsUi.setLook(wCompression);
+ wCompression.setToolTipText(
+ BaseMessages.getString(PKG, "BeamOutputDialog.Compression.ToolTip"));
+ for (Compression compression : Compression.values()) {
+ wCompression.add(compression.name());
+ }
+ FormData fdCompression = new FormData();
+ fdCompression.left = new FormAttachment(middle, 0);
+ fdCompression.top = new FormAttachment(wlCompression, 0, SWT.CENTER);
+ fdCompression.right = new FormAttachment(100, 0);
+ wCompression.setLayoutData(fdCompression);
+ lastControl = wCompression;
+
wFileDefinition =
new MetaSelectionLine<>(
variables,
@@ -195,6 +223,7 @@ public class BeamOutputDialog extends BaseTransformDialog {
wFilePrefix.setText(Const.NVL(input.getFilePrefix(), ""));
wFileSuffix.setText(Const.NVL(input.getFileSuffix(), ""));
wWindowed.setSelection(input.isWindowed());
+ wCompression.setText(Const.NVL(input.getCompression(), ""));
}
private void cancel() {
@@ -221,6 +250,7 @@ public class BeamOutputDialog extends BaseTransformDialog {
in.setFilePrefix(wFilePrefix.getText());
in.setFileSuffix(wFileSuffix.getText());
in.setWindowed(wWindowed.getSelection());
+ in.setCompression(wCompression.getText());
input.setChanged();
}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/io/BeamOutputMeta.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/io/BeamOutputMeta.java
index 96aadadee1..f7c8d03ec9 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/io/BeamOutputMeta.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/io/BeamOutputMeta.java
@@ -68,6 +68,13 @@ public class BeamOutputMeta extends BaseTransformMeta<BeamOutput, BeamOutputData
@HopMetadataProperty private boolean windowed;
+ /**
+ * #2337: how to compress the output files, as a {@link org.apache.beam.sdk.io.Compression}
+ * constant. Blank means no explicit compression; {@code AUTO} derives the codec from the file
+ * suffix.
+ */
+ @HopMetadataProperty private String compression;
+
@Override
public String getDialogClassName() {
return BeamOutputDialog.class.getName();
@@ -148,6 +155,7 @@ public class BeamOutputMeta extends BaseTransformMeta<BeamOutput, BeamOutputData
variables.resolve(outputFileDefinition.getSeparator()),
variables.resolve(outputFileDefinition.getEnclosure()),
windowed,
+ variables.resolve(compression),
JsonRowMeta.toJson(rowMeta));
// Which transform do we apply this transform to?
@@ -259,7 +267,23 @@ public class BeamOutputMeta extends BaseTransformMeta<BeamOutput, BeamOutputData
}
/**
- * Gets windowed
+ * Gets compression
+ *
+ * @return value of compression
+ */
+ public String getCompression() {
+ return compression;
+ }
+
+ /**
+ * @param compression The compression to set
+ */
+ public void setCompression(String compression) {
+ this.compression = compression;
+ }
+
+ /**
+ * Is windowed
*
* @return value of windowed
*/
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/kafka/BeamProduceMeta.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/kafka/BeamProduceMeta.java
index b157a14e68..4c79e85321 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/kafka/BeamProduceMeta.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/kafka/BeamProduceMeta.java
@@ -146,8 +146,10 @@ public class BeamProduceMeta extends BaseTransformMeta<BeamProduce, DummyData>
// Register a coder for the short time that KV<HopRow, GenericRecord> exists in Beam
//
- if (messageValueMeta.getType() == IValueMeta.TYPE_AVRO) {
- ValueMetaAvroRecord valueMetaAvroRecord = (ValueMetaAvroRecord) messageValueMeta;
+ checkAvroMessageSchema(rowMeta, variables);
+ if (rowMeta.searchValueMeta(messageFieldName).getType() == IValueMeta.TYPE_AVRO) {
+ ValueMetaAvroRecord valueMetaAvroRecord =
+ (ValueMetaAvroRecord) rowMeta.searchValueMeta(messageFieldName);
Schema schema = valueMetaAvroRecord.getSchema();
AvroCoder<GenericRecord> coder = AvroCoder.of(schema);
pipeline.getCoderRegistry().registerCoderForClass(GenericRecord.class, coder);
@@ -183,6 +185,36 @@ public class BeamProduceMeta extends BaseTransformMeta<BeamProduce, DummyData>
+ previousTransform.getName());
}
+ /**
+ * Verify the message field exists and, when it is an Avro Record, that it actually carries a
+ * schema.
+ *
+ * <p>Beam's {@code AvroCoder.of(null)} throws a bare NullPointerException from inside its
+ * determinism checker, which tells the person editing the pipeline nothing. This turns it into a
+ * message naming the field and what to do about it (#2675).
+ */
+ void checkAvroMessageSchema(IRowMeta rowMeta, IVariables variables) throws HopException {
+ String field = variables.resolve(getMessageField());
+
+ IValueMeta valueMeta = rowMeta == null ? null : rowMeta.searchValueMeta(field);
+ if (valueMeta == null) {
+ throw new HopException("Error finding message/value field " + field + " in the input rows");
+ }
+
+ if (valueMeta.getType() == IValueMeta.TYPE_AVRO
+ && ((ValueMetaAvroRecord) valueMeta).getSchema() == null) {
+ throw new HopException(
+ "The Avro field '"
+ + field
+ + "' in Kafka produce transform '"
+ + getName()
+ + "' has no schema attached. Beam cannot serialise an Avro value without one. "
+ + "Attach a schema to the field, for example with an Avro Encode transform or an "
+ + "Avro Decode transform which reads the schema from a registry, before this "
+ + "transform.");
+ }
+ }
+
/**
* Gets topic
*
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInput.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInput.java
new file mode 100644
index 0000000000..44bfbdefdd
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInput.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.mqtt;
+
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransform;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+public class BeamMqttInput extends BaseTransform<BeamMqttInputMeta, BeamMqttInputData> {
+ public BeamMqttInput(
+ TransformMeta transformMeta,
+ BeamMqttInputMeta meta,
+ BeamMqttInputData data,
+ int copyNr,
+ PipelineMeta pipelineMeta,
+ Pipeline pipeline) {
+ super(transformMeta, meta, data, copyNr, pipelineMeta, pipeline);
+ }
+
+ @Override
+ public boolean processRow() throws HopException {
+ throw new HopException("Beam MQTT input can only run in a Beam pipeline");
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInputData.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInputData.java
new file mode 100644
index 0000000000..0b0b4387cc
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInputData.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.mqtt;
+
+import org.apache.hop.pipeline.transform.BaseTransformData;
+
+public class BeamMqttInputData extends BaseTransformData {}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInputDialog.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInputDialog.java
new file mode 100644
index 0000000000..c3ba6ffe94
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInputDialog.java
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.mqtt;
+
+import org.apache.hop.core.util.Utils;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.ui.core.dialog.BaseDialog;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
+import org.apache.hop.ui.core.gui.IGuiPluginCompositeWidgetsListener;
+import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
+import org.eclipse.swt.widgets.Control;
+import org.eclipse.swt.widgets.Shell;
+
+public class BeamMqttInputDialog extends BaseTransformDialog {
+ private final BeamMqttInputMeta input;
+ private GuiCompositeWidgets widgets;
+
+ public BeamMqttInputDialog(
+ Shell parent, IVariables variables, BeamMqttInputMeta meta, PipelineMeta pipelineMeta) {
+ super(parent, variables, meta, pipelineMeta);
+ input = meta;
+ }
+
+ @Override
+ public String open() {
+ createShell(BaseMessages.getString(BeamMqttInputMeta.class, "BeamMqttInputDialog.Title"));
+ buildButtonBar().ok(e -> ok()).cancel(e -> cancel()).build();
+ changed = input.hasChanged();
+ widgets =
+ GuiCompositeWidgets.addScrolledComposite(
+ shell,
+ variables,
+ wTransformName,
+ wOk,
+ BeamMqttInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ input);
+ widgets.setCompositeWidgetsListener(
+ new IGuiPluginCompositeWidgetsListener() {
+ @Override
+ public void widgetsCreated(GuiCompositeWidgets w) {}
+
+ @Override
+ public void widgetsPopulated(GuiCompositeWidgets w) {}
+
+ @Override
+ public void widgetModified(GuiCompositeWidgets w, Control control, String id) {
+ input.setChanged();
+ }
+
+ @Override
+ public void persistContents(GuiCompositeWidgets w) {}
+ });
+ focusTransformName();
+ BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
+ return transformName;
+ }
+
+ private void cancel() {
+ transformName = null;
+ input.setChanged(changed);
+ dispose();
+ }
+
+ private void ok() {
+ if (Utils.isEmpty(wTransformName.getText())) return;
+ widgets.getWidgetsContents(input, BeamMqttInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ transformName = wTransformName.getText();
+ input.setChanged();
+ dispose();
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInputMeta.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInputMeta.java
new file mode 100644
index 0000000000..594c1bfa28
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttInputMeta.java
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.mqtt;
+
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.transform.BeamMqttConnection;
+import org.apache.hop.beam.core.transform.BeamMqttInputTransform;
+import org.apache.hop.beam.engines.IBeamPipelineEngineRunConfiguration;
+import org.apache.hop.beam.pipeline.BeamSourceTopology;
+import org.apache.hop.beam.pipeline.IBeamPipelineTransformHandler;
+import org.apache.hop.core.CheckResult;
+import org.apache.hop.core.ICheckResult;
+import org.apache.hop.core.annotations.Transform;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.exception.HopTransformException;
+import org.apache.hop.core.gui.plugin.GuiElementType;
+import org.apache.hop.core.gui.plugin.GuiPlugin;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.value.ValueMetaBinary;
+import org.apache.hop.core.row.value.ValueMetaString;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.metadata.api.HopMetadataProperty;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransformMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+@Getter
+@Setter
+@GuiPlugin
+@Transform(
+ id = "BeamMqttInput",
+ name = "i18n::BeamMqttInputDialog.Title",
+ description = "i18n::BeamMqttInputMeta.Description",
+ image = "beam-mqtt-input.svg",
+ categoryDescription = "i18n:org.apache.hop.pipeline.transform:BaseTransform.Category.BigData",
+ keywords = "i18n::BeamMqttMeta.keyword",
+ documentationUrl = "/pipeline/transforms/beammqttinput.html",
+ supportedEngines = {"Beam*"})
+public class BeamMqttInputMeta extends BaseTransformMeta<BeamMqttInput, BeamMqttInputData>
+ implements IBeamPipelineTransformHandler {
+ public static final String GUI_PLUGIN_ELEMENT_PARENT_ID = "BEAMMQTTINPUT_OPTIONS";
+
+ public static final String WIDGET_SERVER_URI = "serverUri";
+ public static final String WIDGET_TOPIC = "topic";
+ public static final String WIDGET_CLIENT_ID = "clientId";
+ public static final String WIDGET_USERNAME = "username";
+ public static final String WIDGET_PASSWORD = "password";
+ public static final String WIDGET_PAYLOAD_FIELD = "payloadField";
+ public static final String WIDGET_PAYLOAD_TYPE = "payloadType";
+ public static final String WIDGET_MAX_NUM_RECORDS = "maxNumRecords";
+ public static final String WIDGET_MAX_READ_TIME = "maxReadTime";
+
+ @HopMetadataProperty(key = "server_uri")
+ @GuiWidgetElement(
+ id = WIDGET_SERVER_URI,
+ order = "0100",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.serverUri.Label",
+ toolTip = "i18n::BeamMqttMeta.serverUri.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Connection",
+ groupOrder = "0100")
+ private String serverUri;
+
+ @HopMetadataProperty(key = "topic")
+ @GuiWidgetElement(
+ id = WIDGET_TOPIC,
+ order = "0200",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.topic.Label",
+ toolTip = "i18n::BeamMqttMeta.topic.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Connection",
+ groupOrder = "0100")
+ private String topic;
+
+ @HopMetadataProperty(key = "client_id")
+ @GuiWidgetElement(
+ id = WIDGET_CLIENT_ID,
+ order = "0300",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.clientId.Label",
+ toolTip = "i18n::BeamMqttMeta.clientId.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Connection",
+ groupOrder = "0100")
+ private String clientId;
+
+ @HopMetadataProperty(key = "username")
+ @GuiWidgetElement(
+ id = WIDGET_USERNAME,
+ order = "0400",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.username.Label",
+ toolTip = "i18n::BeamMqttMeta.username.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Connection",
+ groupOrder = "0100")
+ private String username;
+
+ @HopMetadataProperty(key = "password", password = true)
+ @GuiWidgetElement(
+ id = WIDGET_PASSWORD,
+ order = "0500",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.password.Label",
+ toolTip = "i18n::BeamMqttMeta.password.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Connection",
+ groupOrder = "0100",
+ password = true)
+ private String password;
+
+ @HopMetadataProperty(key = "payload_field")
+ @GuiWidgetElement(
+ id = WIDGET_PAYLOAD_FIELD,
+ order = "0600",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.payloadField.Label",
+ toolTip = "i18n::BeamMqttMeta.payloadField.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Payload",
+ groupOrder = "0200")
+ private String payloadField = "message";
+
+ @HopMetadataProperty(key = "payload_type")
+ @GuiWidgetElement(
+ id = WIDGET_PAYLOAD_TYPE,
+ order = "0700",
+ type = GuiElementType.COMBO,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.payloadType.Label",
+ toolTip = "i18n::BeamMqttMeta.payloadType.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Payload",
+ groupOrder = "0200",
+ comboValuesMethod = "getPayloadTypes")
+ private String payloadType = "String";
+
+ @HopMetadataProperty(key = "max_num_records")
+ @GuiWidgetElement(
+ id = WIDGET_MAX_NUM_RECORDS,
+ order = "0800",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.maxNumRecords.Label",
+ toolTip = "i18n::BeamMqttMeta.maxNumRecords.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Limits",
+ groupOrder = "0300")
+ private String maxNumRecords = "0";
+
+ @HopMetadataProperty(key = "max_read_time")
+ @GuiWidgetElement(
+ id = WIDGET_MAX_READ_TIME,
+ order = "0900",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.maxReadTime.Label",
+ toolTip = "i18n::BeamMqttMeta.maxReadTime.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Limits",
+ groupOrder = "0300")
+ private String maxReadTime = "0";
+
+ public BeamMqttInputTransform buildInputTransform(IVariables variables, String name)
+ throws HopException {
+ var connection =
+ BeamMqttConnection.resolve(variables, serverUri, topic, clientId, username, password, true);
+ BeamMqttConnection.required(variables, payloadField, "payload field");
+ String type = BeamMqttConnection.payloadType(variables, payloadType);
+ long records = BeamMqttConnection.limit(variables, maxNumRecords, "maximum records");
+ long seconds = BeamMqttConnection.limit(variables, maxReadTime, "maximum read time");
+ if (seconds > Long.MAX_VALUE / 1000)
+ throw new HopException("MQTT maximum read time is too large");
+ return new BeamMqttInputTransform(name, connection, type, records, seconds);
+ }
+
+ @Override
+ public void handleTransform(
+ ILogChannel log,
+ IVariables variables,
+ String runConfigurationName,
+ IBeamPipelineEngineRunConfiguration runConfiguration,
+ String dataSamplersJson,
+ IHopMetadataProvider metadataProvider,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ Map<String, PCollection<HopRow>> transformCollectionMap,
+ org.apache.beam.sdk.Pipeline pipeline,
+ IRowMeta rowMeta,
+ List<TransformMeta> previousTransforms,
+ PCollection<HopRow> input,
+ String parentLogChannelId)
+ throws HopException {
+ BeamSourceTopology.rejectIncomingRows(
+ pipelineMeta,
+ transformMeta,
+ previousTransforms,
+ input,
+ "Beam MQTT input does not accept incoming rows");
+ transformCollectionMap.put(
+ transformMeta.getName(),
+ pipeline.apply(
+ transformMeta.getName(), buildInputTransform(variables, transformMeta.getName())));
+ }
+
+ @Override
+ public void check(
+ List<ICheckResult> remarks,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ IRowMeta prev,
+ String[] input,
+ String[] output,
+ IRowMeta info,
+ IVariables variables,
+ IHopMetadataProvider metadataProvider) {
+ try {
+ if (input != null && input.length > 0)
+ throw new HopException("Beam MQTT input does not accept incoming rows");
+ buildInputTransform(variables, transformMeta.getName());
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_OK,
+ BaseMessages.getString(BeamMqttInputMeta.class, "BeamMqttMeta.ConfigurationValid"),
+ transformMeta));
+ } catch (HopException e) {
+ remarks.add(new CheckResult(ICheckResult.TYPE_RESULT_ERROR, e.getMessage(), transformMeta));
+ }
+ }
+
+ @Override
+ public String getDialogClassName() {
+ return BeamMqttInputDialog.class.getName();
+ }
+
+ public List<String> getPayloadTypes(ILogChannel log, IHopMetadataProvider metadataProvider) {
+ return List.of("String", "Binary");
+ }
+
+ @Override
+ public boolean isInput() {
+ return true;
+ }
+
+ @Override
+ public boolean isOutput() {
+ return false;
+ }
+
+ @Override
+ public boolean consumesMainInput() {
+ return false;
+ }
+
+ @Override
+ public boolean canStartWithoutInput() {
+ return true;
+ }
+
+ @Override
+ public void getFields(
+ IRowMeta row,
+ String name,
+ IRowMeta[] info,
+ TransformMeta next,
+ IVariables variables,
+ IHopMetadataProvider provider)
+ throws HopTransformException {
+ try {
+ String field = BeamMqttConnection.required(variables, payloadField, "payload field");
+ String type = BeamMqttConnection.payloadType(variables, payloadType);
+ IValueMeta value =
+ "Binary".equalsIgnoreCase(type) ? new ValueMetaBinary(field) : new ValueMetaString(field);
+ value.setOrigin(name);
+ row.clear();
+ row.addValueMeta(value);
+ } catch (HopException e) {
+ throw new HopTransformException(e.getMessage(), e);
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutput.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutput.java
new file mode 100644
index 0000000000..ca501421f9
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutput.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.mqtt;
+
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransform;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+public class BeamMqttOutput extends BaseTransform<BeamMqttOutputMeta, BeamMqttOutputData> {
+ public BeamMqttOutput(
+ TransformMeta transformMeta,
+ BeamMqttOutputMeta meta,
+ BeamMqttOutputData data,
+ int copyNr,
+ PipelineMeta pipelineMeta,
+ Pipeline pipeline) {
+ super(transformMeta, meta, data, copyNr, pipelineMeta, pipeline);
+ }
+
+ @Override
+ public boolean processRow() throws HopException {
+ throw new HopException("Beam MQTT output can only run in a Beam pipeline");
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutputData.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutputData.java
new file mode 100644
index 0000000000..7c801fb5f0
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutputData.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.mqtt;
+
+import org.apache.hop.pipeline.transform.BaseTransformData;
+
+public class BeamMqttOutputData extends BaseTransformData {}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutputDialog.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutputDialog.java
new file mode 100644
index 0000000000..db6c8a9cf2
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutputDialog.java
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.mqtt;
+
+import org.apache.hop.core.util.Utils;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.ui.core.dialog.BaseDialog;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
+import org.apache.hop.ui.core.gui.IGuiPluginCompositeWidgetsListener;
+import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
+import org.eclipse.swt.widgets.Control;
+import org.eclipse.swt.widgets.Shell;
+
+public class BeamMqttOutputDialog extends BaseTransformDialog {
+ private final BeamMqttOutputMeta input;
+ private GuiCompositeWidgets widgets;
+
+ public BeamMqttOutputDialog(
+ Shell parent, IVariables variables, BeamMqttOutputMeta meta, PipelineMeta pipelineMeta) {
+ super(parent, variables, meta, pipelineMeta);
+ input = meta;
+ }
+
+ @Override
+ public String open() {
+ createShell(BaseMessages.getString(BeamMqttOutputMeta.class, "BeamMqttOutputDialog.Title"));
+ buildButtonBar().ok(e -> ok()).cancel(e -> cancel()).build();
+ changed = input.hasChanged();
+ widgets =
+ GuiCompositeWidgets.addScrolledComposite(
+ shell,
+ variables,
+ wTransformName,
+ wOk,
+ BeamMqttOutputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ input);
+ widgets.setCompositeWidgetsListener(
+ new IGuiPluginCompositeWidgetsListener() {
+ @Override
+ public void widgetsCreated(GuiCompositeWidgets w) {}
+
+ @Override
+ public void widgetsPopulated(GuiCompositeWidgets w) {}
+
+ @Override
+ public void widgetModified(GuiCompositeWidgets w, Control control, String id) {
+ input.setChanged();
+ }
+
+ @Override
+ public void persistContents(GuiCompositeWidgets w) {}
+ });
+ focusTransformName();
+ BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
+ return transformName;
+ }
+
+ private void cancel() {
+ transformName = null;
+ input.setChanged(changed);
+ dispose();
+ }
+
+ private void ok() {
+ if (Utils.isEmpty(wTransformName.getText())) return;
+ widgets.getWidgetsContents(input, BeamMqttOutputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ transformName = wTransformName.getText();
+ input.setChanged();
+ dispose();
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutputMeta.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutputMeta.java
new file mode 100644
index 0000000000..4d4c261f1b
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/mqtt/BeamMqttOutputMeta.java
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.mqtt;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.transform.BeamMqttConnection;
+import org.apache.hop.beam.core.transform.BeamMqttOutputTransform;
+import org.apache.hop.beam.engines.IBeamPipelineEngineRunConfiguration;
+import org.apache.hop.beam.pipeline.IBeamPipelineTransformHandler;
+import org.apache.hop.core.CheckResult;
+import org.apache.hop.core.ICheckResult;
+import org.apache.hop.core.annotations.Transform;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.exception.HopTransformException;
+import org.apache.hop.core.gui.plugin.GuiElementType;
+import org.apache.hop.core.gui.plugin.GuiPlugin;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.metadata.api.HopMetadataProperty;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransformMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+@Getter
+@Setter
+@GuiPlugin
+@Transform(
+ id = "BeamMqttOutput",
+ name = "i18n::BeamMqttOutputDialog.Title",
+ description = "i18n::BeamMqttOutputMeta.Description",
+ image = "beam-mqtt-output.svg",
+ categoryDescription = "i18n:org.apache.hop.pipeline.transform:BaseTransform.Category.BigData",
+ keywords = "i18n::BeamMqttMeta.keyword",
+ documentationUrl = "/pipeline/transforms/beammqttoutput.html",
+ supportedEngines = {"Beam*"})
+public class BeamMqttOutputMeta extends BaseTransformMeta<BeamMqttOutput, BeamMqttOutputData>
+ implements IBeamPipelineTransformHandler {
+ public static final String GUI_PLUGIN_ELEMENT_PARENT_ID = "BEAMMQTTOUTPUT_OPTIONS";
+
+ public static final String WIDGET_SERVER_URI = "serverUri";
+ public static final String WIDGET_TOPIC = "topic";
+ public static final String WIDGET_CLIENT_ID = "clientId";
+ public static final String WIDGET_USERNAME = "username";
+ public static final String WIDGET_PASSWORD = "password";
+ public static final String WIDGET_PAYLOAD_FIELD = "payloadField";
+ public static final String WIDGET_PAYLOAD_TYPE = "payloadType";
+ public static final String WIDGET_RETAINED = "retained";
+
+ @HopMetadataProperty(key = "server_uri")
+ @GuiWidgetElement(
+ id = WIDGET_SERVER_URI,
+ order = "0100",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.serverUri.Label",
+ toolTip = "i18n::BeamMqttMeta.serverUri.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Connection",
+ groupOrder = "0100")
+ private String serverUri;
+
+ @HopMetadataProperty(key = "topic")
+ @GuiWidgetElement(
+ id = WIDGET_TOPIC,
+ order = "0200",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.topic.Label",
+ toolTip = "i18n::BeamMqttMeta.topic.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Connection",
+ groupOrder = "0100")
+ private String topic;
+
+ @HopMetadataProperty(key = "client_id")
+ @GuiWidgetElement(
+ id = WIDGET_CLIENT_ID,
+ order = "0300",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.clientId.Label",
+ toolTip = "i18n::BeamMqttMeta.clientId.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Connection",
+ groupOrder = "0100")
+ private String clientId;
+
+ @HopMetadataProperty(key = "username")
+ @GuiWidgetElement(
+ id = WIDGET_USERNAME,
+ order = "0400",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.username.Label",
+ toolTip = "i18n::BeamMqttMeta.username.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Connection",
+ groupOrder = "0100")
+ private String username;
+
+ @HopMetadataProperty(key = "password", password = true)
+ @GuiWidgetElement(
+ id = WIDGET_PASSWORD,
+ order = "0500",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.password.Label",
+ toolTip = "i18n::BeamMqttMeta.password.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Connection",
+ groupOrder = "0100",
+ password = true)
+ private String password;
+
+ @HopMetadataProperty(key = "payload_field")
+ @GuiWidgetElement(
+ id = WIDGET_PAYLOAD_FIELD,
+ order = "0600",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.payloadField.Label",
+ toolTip = "i18n::BeamMqttMeta.payloadField.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Payload",
+ groupOrder = "0200")
+ private String payloadField = "message";
+
+ @HopMetadataProperty(key = "payload_type")
+ @GuiWidgetElement(
+ id = WIDGET_PAYLOAD_TYPE,
+ order = "0700",
+ type = GuiElementType.COMBO,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.payloadType.Label",
+ toolTip = "i18n::BeamMqttMeta.payloadType.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Payload",
+ groupOrder = "0200",
+ comboValuesMethod = "getPayloadTypes")
+ private String payloadType = "String";
+
+ @HopMetadataProperty(key = "retained")
+ @GuiWidgetElement(
+ id = WIDGET_RETAINED,
+ order = "0800",
+ type = GuiElementType.CHECKBOX,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamMqttMeta.retained.Label",
+ toolTip = "i18n::BeamMqttMeta.retained.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = "i18n::BeamMqttMeta.Payload",
+ groupOrder = "0200")
+ private boolean retained = false;
+
+ public BeamMqttOutputTransform buildOutputTransform(
+ IVariables variables, String name, IRowMeta rowMeta) throws HopException {
+ var connection =
+ BeamMqttConnection.resolve(
+ variables, serverUri, topic, clientId, username, password, false);
+ String field = BeamMqttConnection.required(variables, payloadField, "payload field");
+ String type = BeamMqttConnection.payloadType(variables, payloadType);
+ int index = rowMeta.indexOfValue(field);
+ if (index < 0) throw new HopException("MQTT payload field not found: " + field);
+ if ("Binary".equalsIgnoreCase(type)
+ && rowMeta.getValueMeta(index).getType() != IValueMeta.TYPE_BINARY)
+ throw new HopException("MQTT binary payload requires a Binary field: " + field);
+ String rowMetaXml;
+ try {
+ rowMetaXml = rowMeta.getMetaXml();
+ } catch (IOException e) {
+ throw new HopException("Unable to serialize MQTT row metadata", e);
+ }
+ return new BeamMqttOutputTransform(name, connection, field, type, rowMetaXml, retained);
+ }
+
+ @Override
+ public void handleTransform(
+ ILogChannel log,
+ IVariables variables,
+ String runConfigurationName,
+ IBeamPipelineEngineRunConfiguration runConfiguration,
+ String dataSamplersJson,
+ IHopMetadataProvider metadataProvider,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ Map<String, PCollection<HopRow>> transformCollectionMap,
+ org.apache.beam.sdk.Pipeline pipeline,
+ IRowMeta rowMeta,
+ List<TransformMeta> previousTransforms,
+ PCollection<HopRow> input,
+ String parentLogChannelId)
+ throws HopException {
+ if (input == null || previousTransforms.size() != 1)
+ throw new HopException("Beam MQTT output requires exactly one incoming transform");
+ input.apply(
+ transformMeta.getName(), buildOutputTransform(variables, transformMeta.getName(), rowMeta));
+ }
+
+ @Override
+ public void check(
+ List<ICheckResult> remarks,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ IRowMeta prev,
+ String[] input,
+ String[] output,
+ IRowMeta info,
+ IVariables variables,
+ IHopMetadataProvider metadataProvider) {
+ try {
+ if (input == null || input.length != 1 || prev == null)
+ throw new HopException("Beam MQTT output requires exactly one incoming transform");
+ buildOutputTransform(variables, transformMeta.getName(), prev);
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_OK,
+ BaseMessages.getString(BeamMqttOutputMeta.class, "BeamMqttMeta.ConfigurationValid"),
+ transformMeta));
+ } catch (HopException e) {
+ remarks.add(new CheckResult(ICheckResult.TYPE_RESULT_ERROR, e.getMessage(), transformMeta));
+ }
+ }
+
+ @Override
+ public String getDialogClassName() {
+ return BeamMqttOutputDialog.class.getName();
+ }
+
+ public List<String> getPayloadTypes(ILogChannel log, IHopMetadataProvider metadataProvider) {
+ return List.of("String", "Binary");
+ }
+
+ @Override
+ public boolean isInput() {
+ return false;
+ }
+
+ @Override
+ public boolean isOutput() {
+ return true;
+ }
+
+ @Override
+ public void getFields(
+ IRowMeta row,
+ String name,
+ IRowMeta[] info,
+ TransformMeta next,
+ IVariables variables,
+ IHopMetadataProvider provider)
+ throws HopTransformException {
+ row.clear();
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartition.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartition.java
new file mode 100644
index 0000000000..291c20adc6
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartition.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.partition;
+
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransform;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+/**
+ * Issue #2040: partitioning support for Beam pipelines.
+ *
+ * <p>The work is all done when the Beam graph is built, by {@link BeamPartitionMeta}; there is
+ * nothing left to do per row at runtime.
+ */
+public class BeamPartition extends BaseTransform<BeamPartitionMeta, BeamPartitionData> {
+
+ public BeamPartition(
+ TransformMeta transformMeta,
+ BeamPartitionMeta meta,
+ BeamPartitionData data,
+ int copyNr,
+ PipelineMeta pipelineMeta,
+ Pipeline pipeline) {
+ super(transformMeta, meta, data, copyNr, pipelineMeta, pipeline);
+ }
+
+ @Override
+ public boolean processRow() throws HopException {
+ // Nothing to do: partitioning only shapes the Beam graph.
+ return true;
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartitionData.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartitionData.java
new file mode 100644
index 0000000000..2169af5ca7
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartitionData.java
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.partition;
+
+import org.apache.hop.pipeline.transform.BaseTransformData;
+import org.apache.hop.pipeline.transform.ITransformData;
+
+public class BeamPartitionData extends BaseTransformData implements ITransformData {}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartitionDialog.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartitionDialog.java
new file mode 100644
index 0000000000..e727096077
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartitionDialog.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.partition;
+
+import org.apache.hop.core.util.Utils;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.ui.core.dialog.BaseDialog;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
+import org.apache.hop.ui.core.gui.IGuiPluginCompositeWidgetsListener;
+import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
+import org.eclipse.swt.widgets.Control;
+import org.eclipse.swt.widgets.Shell;
+
+/** Issue #2040: dialog for the Beam partition transform. */
+public class BeamPartitionDialog extends BaseTransformDialog {
+ private static final Class<?> PKG = BeamPartitionMeta.class;
+ private final BeamPartitionMeta input;
+ private GuiCompositeWidgets widgets;
+
+ public BeamPartitionDialog(
+ Shell parent,
+ IVariables variables,
+ BeamPartitionMeta transformMeta,
+ PipelineMeta pipelineMeta) {
+ super(parent, variables, transformMeta, pipelineMeta);
+ input = transformMeta;
+ }
+
+ @Override
+ public String open() {
+ createShell(BaseMessages.getString(PKG, "BeamPartitionDialog.DialogTitle"));
+ buildButtonBar().ok(e -> ok()).cancel(e -> cancel()).build();
+ changed = input.hasChanged();
+
+ widgets =
+ GuiCompositeWidgets.addScrolledComposite(
+ shell,
+ variables,
+ wTransformName,
+ wOk,
+ BeamPartitionMeta.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ input);
+ widgets.setCompositeWidgetsListener(
+ new IGuiPluginCompositeWidgetsListener() {
+ @Override
+ public void widgetsCreated(GuiCompositeWidgets compositeWidgets) {}
+
+ @Override
+ public void widgetsPopulated(GuiCompositeWidgets compositeWidgets) {}
+
+ @Override
+ public void widgetModified(
+ GuiCompositeWidgets compositeWidgets, Control control, String widgetId) {
+ input.setChanged();
+ }
+
+ @Override
+ public void persistContents(GuiCompositeWidgets compositeWidgets) {}
+ });
+
+ focusTransformName();
+ BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
+
+ return transformName;
+ }
+
+ private void cancel() {
+ transformName = null;
+ input.setChanged(changed);
+ dispose();
+ }
+
+ private void ok() {
+ if (Utils.isEmpty(wTransformName.getText())) {
+ return;
+ }
+ widgets.getWidgetsContents(input, BeamPartitionMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ transformName = wTransformName.getText();
+ input.setChanged();
+ dispose();
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartitionMeta.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartitionMeta.java
new file mode 100644
index 0000000000..5a75a8b5ed
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/partition/BeamPartitionMeta.java
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.partition;
+
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.Partition;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.hop.beam.core.BeamDefaults;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.partition.KeyedPartitionFn;
+import org.apache.hop.beam.core.partition.SinglePartitionFn;
+import org.apache.hop.beam.engines.IBeamPipelineEngineRunConfiguration;
+import org.apache.hop.beam.pipeline.IBeamPipelineTransformHandler;
+import org.apache.hop.core.Const;
+import org.apache.hop.core.annotations.Transform;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.gui.plugin.GuiElementType;
+import org.apache.hop.core.gui.plugin.GuiPlugin;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.JsonRowMeta;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.metadata.api.HopMetadataProperty;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransformMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+/**
+ * Issue #2040: partitioning support for Beam pipelines.
+ *
+ * <p>Mirrors what the local Hop engine does with its partitioning option: decide up front how many
+ * partitions exist and which partition each row belongs to, so that the steps that follow can be
+ * grouped per key instead of shuffled.
+ *
+ * <p>Two modes. {@code Single} sends everything to one partition, which serialises everything
+ * downstream. {@code Key} hashes a field so rows sharing a value land together.
+ */
+@Transform(
+ id = "BeamPartition",
+ image = "beam-partition.svg",
+ name = "i18n::BeamPartitionDialog.DialogTitle",
+ description = "i18n::BeamPartitionDialog.Description",
+ categoryDescription = "i18n:org.apache.hop.pipeline.transform:BaseTransform.Category.BigData",
+ keywords = "i18n::BeamPartitionMeta.keyword",
+ documentationUrl = "/pipeline/transforms/beampartition.html",
+ supportedEngines = {"Beam*"})
+@GuiPlugin
+@Getter
+@Setter
+public class BeamPartitionMeta extends BaseTransformMeta<BeamPartition, BeamPartitionData>
+ implements IBeamPipelineTransformHandler {
+
+ public static final String GUI_PLUGIN_ELEMENT_PARENT_ID = "BEAM_PARTITION_OPTIONS";
+ public static final String WIDGET_PARTITION_TYPE = "partitionType";
+ public static final String WIDGET_KEY_FIELD = "keyField";
+ public static final String WIDGET_NUM_PARTITIONS = "numPartitions";
+
+ @GuiWidgetElement(
+ id = WIDGET_PARTITION_TYPE,
+ order = "0100",
+ type = GuiElementType.COMBO,
+ comboValuesMethod = "getPartitionTypes",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamPartitionDialog.PartitionType",
+ toolTip = "i18n::BeamPartitionDialog.PartitionType.ToolTip",
+ groupType = GuiWidgetGroupType.BOXES,
+ group = "i18n::BeamPartitionDialog.Group")
+ @HopMetadataProperty(key = "partition_type")
+ private String partitionType;
+
+ /** The field to partition on. Required for the Key mode, ignored for Single. */
+ @GuiWidgetElement(
+ id = WIDGET_KEY_FIELD,
+ order = "0200",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamPartitionDialog.KeyField",
+ toolTip = "i18n::BeamPartitionDialog.KeyField.ToolTip",
+ groupType = GuiWidgetGroupType.BOXES,
+ group = "i18n::BeamPartitionDialog.Group")
+ @HopMetadataProperty(key = "key_field")
+ private String keyField;
+
+ /** How many partitions to create. */
+ @GuiWidgetElement(
+ id = WIDGET_NUM_PARTITIONS,
+ order = "0300",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamPartitionDialog.NumPartitions",
+ toolTip = "i18n::BeamPartitionDialog.NumPartitions.ToolTip",
+ groupType = GuiWidgetGroupType.BOXES,
+ group = "i18n::BeamPartitionDialog.Group")
+ @HopMetadataProperty(key = "num_partitions")
+ private String numPartitions;
+
+ public List<String> getPartitionTypes(ILogChannel log, IHopMetadataProvider provider) {
+ return List.of(BeamDefaults.PARTITION_TYPE_SINGLE, BeamDefaults.PARTITION_TYPE_KEY);
+ }
+
+ @Override
+ public String getDialogClassName() {
+ return BeamPartitionDialog.class.getName();
+ }
+
+ public BeamPartitionMeta() {
+ super();
+ setDefault();
+ }
+
+ @Override
+ public void setDefault() {
+ partitionType = BeamDefaults.PARTITION_TYPE_SINGLE;
+ keyField = "";
+ numPartitions = "1";
+ }
+
+ @Override
+ public boolean isInput() {
+ return false;
+ }
+
+ @Override
+ public boolean isOutput() {
+ return false;
+ }
+
+ @Override
+ public void handleTransform(
+ ILogChannel log,
+ IVariables variables,
+ String runConfigurationName,
+ IBeamPipelineEngineRunConfiguration runConfiguration,
+ String dataSamplersJson,
+ IHopMetadataProvider metadataProvider,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ Map<String, PCollection<HopRow>> transformCollectionMap,
+ Pipeline pipeline,
+ IRowMeta rowMeta,
+ List<TransformMeta> previousTransforms,
+ PCollection<HopRow> input,
+ String parentLogChannelId)
+ throws HopException {
+
+ int partitions = 1;
+ if (org.apache.commons.lang3.StringUtils.isNotEmpty(numPartitions)) {
+ partitions = Const.toInt(variables.resolve(numPartitions), 1);
+ }
+ if (partitions < 1) {
+ throw new HopException(
+ "The number of partitions of Beam Partition transform '"
+ + transformMeta.getName()
+ + "' has to be at least 1, but it is "
+ + partitions);
+ }
+
+ Partition.PartitionFn<HopRow> partitionFn;
+ if (BeamDefaults.PARTITION_TYPE_KEY.equals(partitionType)) {
+ if (org.apache.commons.lang3.StringUtils.isEmpty(keyField)) {
+ throw new HopException(
+ "Please specify a key field for Beam Partition transform '"
+ + transformMeta.getName()
+ + "'");
+ }
+ String realKeyField = variables.resolve(keyField);
+ int keyIndex = rowMeta == null ? -1 : rowMeta.indexOfValue(realKeyField);
+ if (keyIndex < 0) {
+ throw new HopException(
+ "Unable to find the key field '"
+ + realKeyField
+ + "' in the input of Beam Partition transform '"
+ + transformMeta.getName()
+ + "'");
+ }
+ partitionFn =
+ new KeyedPartitionFn(transformMeta.getName(), JsonRowMeta.toJson(rowMeta), keyIndex);
+ } else {
+ partitionFn = new SinglePartitionFn();
+ }
+
+ // Beam's Partition transform returns a PCollectionList, one collection per partition. The rows
+ // have to be recombined, because Beam picks the partition per row at runtime and a pipeline
+ // cannot branch on that. Taking a single partition would silently drop the rows that landed in
+ // the others, so flatten them all back into one stream. What partitioning buys is that the
+ // decision was made deterministically, before the following transforms ran.
+ //
+ PCollectionList<HopRow> partitionList = input.apply(Partition.of(partitions, partitionFn));
+
+ PCollection<HopRow> result =
+ partitions == 1 ? partitionList.get(0) : partitionList.apply(Flatten.pCollections());
+
+ transformCollectionMap.put(transformMeta.getName(), result);
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInput.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInput.java
new file mode 100644
index 0000000000..db441ca16e
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInput.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.snowflake;
+
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransform;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+public class BeamSnowflakeInput
+ extends BaseTransform<BeamSnowflakeInputMeta, BeamSnowflakeInputData> {
+ public BeamSnowflakeInput(
+ TransformMeta transformMeta,
+ BeamSnowflakeInputMeta meta,
+ BeamSnowflakeInputData data,
+ int copyNr,
+ PipelineMeta pipelineMeta,
+ Pipeline pipeline) {
+ super(transformMeta, meta, data, copyNr, pipelineMeta, pipeline);
+ }
+
+ @Override
+ public boolean processRow() throws HopException {
+ throw new HopException("Beam Snowflake input can only run in a Beam pipeline");
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInputData.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInputData.java
new file mode 100644
index 0000000000..db3469f1b2
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInputData.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.snowflake;
+
+import org.apache.hop.pipeline.transform.BaseTransformData;
+
+public class BeamSnowflakeInputData extends BaseTransformData {}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInputDialog.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInputDialog.java
new file mode 100644
index 0000000000..cc51c0d42c
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInputDialog.java
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.snowflake;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hop.core.Const;
+import org.apache.hop.core.util.Utils;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.ui.core.dialog.BaseDialog;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
+import org.apache.hop.ui.core.gui.IGuiPluginCompositeWidgetsListener;
+import org.apache.hop.ui.core.widget.ColumnInfo;
+import org.apache.hop.ui.core.widget.TableView;
+import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
+import org.eclipse.swt.SWT;
+import org.eclipse.swt.layout.FormAttachment;
+import org.eclipse.swt.layout.FormData;
+import org.eclipse.swt.widgets.Composite;
+import org.eclipse.swt.widgets.Control;
+import org.eclipse.swt.widgets.Shell;
+import org.eclipse.swt.widgets.TableItem;
+
+public class BeamSnowflakeInputDialog extends BaseTransformDialog {
+ private static final Class<?> PKG = BeamSnowflakeInputMeta.class;
+ private final BeamSnowflakeInputMeta input;
+ private GuiCompositeWidgets widgets;
+ private TableView wFields;
+
+ public BeamSnowflakeInputDialog(
+ Shell parent, IVariables variables, BeamSnowflakeInputMeta meta, PipelineMeta pipelineMeta) {
+ super(parent, variables, meta, pipelineMeta);
+ input = meta;
+ }
+
+ @Override
+ public String open() {
+ createShell(BaseMessages.getString(PKG, "BeamSnowflakeInputDialog.Title"));
+ buildButtonBar().ok(e -> ok()).cancel(e -> cancel()).build();
+ changed = input.hasChanged();
+ widgets =
+ GuiCompositeWidgets.addScrolledComposite(
+ shell,
+ variables,
+ wTransformName,
+ wOk,
+ BeamSnowflakeInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ input,
+ w -> {
+ widgets = w;
+ w.registerExtraGroup(
+ BaseMessages.getString(PKG, "BeamSnowflakeInput.Fields.Group"),
+ "0400",
+ null,
+ this::addFieldsTable);
+ });
+ widgets.setCompositeWidgetsListener(
+ new IGuiPluginCompositeWidgetsListener() {
+ @Override
+ public void widgetsCreated(GuiCompositeWidgets w) {}
+
+ @Override
+ public void widgetsPopulated(GuiCompositeWidgets w) {}
+
+ @Override
+ public void widgetModified(GuiCompositeWidgets w, Control control, String id) {
+ input.setChanged();
+ }
+
+ @Override
+ public void persistContents(GuiCompositeWidgets w) {
+ input.setFields(readFields());
+ }
+ });
+ populateFields();
+ input.setChanged(changed);
+ focusTransformName();
+ BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
+ return transformName;
+ }
+
+ private void addFieldsTable(Composite parent) {
+ ColumnInfo[] columns =
+ new ColumnInfo[] {
+ new ColumnInfo(
+ BaseMessages.getString(PKG, "BeamSnowflakeInput.Fields.Column.Name"),
+ ColumnInfo.COLUMN_TYPE_TEXT,
+ false),
+ new ColumnInfo(
+ BaseMessages.getString(PKG, "BeamSnowflakeInput.Fields.Column.Type"),
+ ColumnInfo.COLUMN_TYPE_CCOMBO,
+ new String[] {
+ "String", "Integer", "Number", "BigNumber", "Boolean", "Date", "Timestamp", "Binary"
+ },
+ true)
+ };
+ int rows = input.getFields() == null ? 0 : input.getFields().size();
+ wFields =
+ new TableView(
+ variables,
+ parent,
+ SWT.BORDER | SWT.FULL_SELECTION | SWT.MULTI,
+ columns,
+ rows,
+ e -> input.setChanged(),
+ props);
+ FormData formData = new FormData();
+ formData.left = new FormAttachment(0, 0);
+ formData.right = new FormAttachment(100, 0);
+ formData.top = new FormAttachment(0, 0);
+ formData.bottom = new FormAttachment(100, 0);
+ wFields.setLayoutData(formData);
+ }
+
+ private void populateFields() {
+ if (wFields == null || input.getFields() == null) return;
+ for (SnowflakeField field : input.getFields()) {
+ TableItem item = new TableItem(wFields.table, SWT.NONE);
+ item.setText(1, Const.NVL(field.getName(), ""));
+ item.setText(2, Const.NVL(field.getType(), ""));
+ }
+ wFields.removeEmptyRows();
+ wFields.setRowNums();
+ }
+
+ private List<SnowflakeField> readFields() {
+ List<SnowflakeField> fields = new ArrayList<>();
+ if (wFields == null || wFields.isDisposed()) return fields;
+ for (TableItem item : wFields.getNonEmptyItems()) {
+ if (!Utils.isEmpty(item.getText(1)))
+ fields.add(new SnowflakeField(item.getText(1), item.getText(2)));
+ }
+ return fields;
+ }
+
+ private void cancel() {
+ transformName = null;
+ input.setChanged(changed);
+ dispose();
+ }
+
+ private void ok() {
+ if (Utils.isEmpty(wTransformName.getText())) return;
+ widgets.getWidgetsContents(input, BeamSnowflakeInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ input.setFields(readFields());
+ transformName = wTransformName.getText();
+ input.setChanged();
+ dispose();
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInputMeta.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInputMeta.java
new file mode 100644
index 0000000000..84a27d14c7
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeInputMeta.java
@@ -0,0 +1,442 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.snowflake;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.snowflake.SnowflakeIO;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.coder.HopRowCoder;
+import org.apache.hop.beam.core.fn.SnowflakeCsvToHop;
+import org.apache.hop.beam.core.fn.SnowflakeValues;
+import org.apache.hop.beam.core.transform.BeamSnowflakeInputTransform;
+import org.apache.hop.beam.engines.IBeamPipelineEngineRunConfiguration;
+import org.apache.hop.beam.pipeline.BeamSourceTopology;
+import org.apache.hop.beam.pipeline.IBeamPipelineTransformHandler;
+import org.apache.hop.core.CheckResult;
+import org.apache.hop.core.ICheckResult;
+import org.apache.hop.core.annotations.Transform;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.exception.HopTransformException;
+import org.apache.hop.core.gui.plugin.GuiElementType;
+import org.apache.hop.core.gui.plugin.GuiPlugin;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaFactory;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.metadata.api.HopMetadataProperty;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransformMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+@Getter
+@Setter
+@GuiPlugin
+@Transform(
+ id = "BeamSnowflakeInput",
+ name = "i18n::BeamSnowflakeInputDialog.Title",
+ description = "i18n::BeamSnowflakeInputMeta.Description",
+ image = "beam-snowflake-input.svg",
+ categoryDescription = "i18n:org.apache.hop.pipeline.transform:BaseTransform.Category.BigData",
+ keywords = "i18n::BeamSnowflakeInput.keyword",
+ documentationUrl = "/pipeline/transforms/beamsnowflakeinput.html",
+ supportedEngines = {"Beam*"})
+public class BeamSnowflakeInputMeta
+ extends BaseTransformMeta<BeamSnowflakeInput, BeamSnowflakeInputData>
+ implements IBeamPipelineTransformHandler {
+ public static final String GUI_PLUGIN_ELEMENT_PARENT_ID = "BEAMSNOWFLAKEINPUT_OPTIONS";
+ private static final String CONNECTION = "i18n::BeamSnowflake.Connection.Group";
+ private static final String STAGING = "i18n::BeamSnowflake.Staging.Group";
+ private static final String READ = "i18n::BeamSnowflakeInput.Read.Group";
+
+ @HopMetadataProperty(key = "server_name")
+ @GuiWidgetElement(
+ id = "serverName",
+ order = "0100",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.serverName.Label",
+ toolTip = "i18n::BeamSnowflake.serverName.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = CONNECTION,
+ groupOrder = "0100")
+ private String serverName;
+
+ @HopMetadataProperty(key = "username")
+ @GuiWidgetElement(
+ id = "username",
+ order = "0200",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.username.Label",
+ toolTip = "i18n::BeamSnowflake.username.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = CONNECTION,
+ groupOrder = "0100")
+ private String username;
+
+ @HopMetadataProperty(key = "password", password = true)
+ @GuiWidgetElement(
+ id = "password",
+ order = "0300",
+ type = GuiElementType.TEXT,
+ password = true,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.password.Label",
+ toolTip = "i18n::BeamSnowflake.password.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = CONNECTION,
+ groupOrder = "0100")
+ private String password;
+
+ @HopMetadataProperty(key = "private_key", password = true)
+ @GuiWidgetElement(
+ id = "privateKey",
+ order = "0400",
+ type = GuiElementType.TEXT,
+ password = true,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.privateKey.Label",
+ toolTip = "i18n::BeamSnowflake.privateKey.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = CONNECTION,
+ groupOrder = "0100")
+ private String privateKey;
+
+ @HopMetadataProperty(key = "private_key_passphrase", password = true)
+ @GuiWidgetElement(
+ id = "privateKeyPassphrase",
+ order = "0500",
+ type = GuiElementType.TEXT,
+ password = true,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.privateKeyPassphrase.Label",
+ toolTip = "i18n::BeamSnowflake.privateKeyPassphrase.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = CONNECTION,
+ groupOrder = "0100")
+ private String privateKeyPassphrase;
+
+ @HopMetadataProperty(key = "database")
+ @GuiWidgetElement(
+ id = "database",
+ order = "0600",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.database.Label",
+ toolTip = "i18n::BeamSnowflake.database.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = CONNECTION,
+ groupOrder = "0100")
+ private String database;
+
+ @HopMetadataProperty(key = "warehouse")
+ @GuiWidgetElement(
+ id = "warehouse",
+ order = "0700",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.warehouse.Label",
+ toolTip = "i18n::BeamSnowflake.warehouse.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = CONNECTION,
+ groupOrder = "0100")
+ private String warehouse;
+
+ @HopMetadataProperty(key = "schema")
+ @GuiWidgetElement(
+ id = "schema",
+ order = "0800",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.schema.Label",
+ toolTip = "i18n::BeamSnowflake.schema.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = CONNECTION,
+ groupOrder = "0100")
+ private String schema;
+
+ @HopMetadataProperty(key = "role")
+ @GuiWidgetElement(
+ id = "role",
+ order = "0900",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.role.Label",
+ toolTip = "i18n::BeamSnowflake.role.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = CONNECTION,
+ groupOrder = "0100")
+ private String role;
+
+ @HopMetadataProperty(key = "port")
+ @GuiWidgetElement(
+ id = "port",
+ order = "0950",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.port.Label",
+ toolTip = "i18n::BeamSnowflake.port.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = CONNECTION,
+ groupOrder = "0100")
+ private String port;
+
+ @HopMetadataProperty(key = "staging_bucket")
+ @GuiWidgetElement(
+ id = "stagingBucket",
+ order = "1000",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.stagingBucket.Label",
+ toolTip = "i18n::BeamSnowflake.stagingBucket.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = STAGING,
+ groupOrder = "0200")
+ private String stagingBucket;
+
+ @HopMetadataProperty(key = "storage_integration")
+ @GuiWidgetElement(
+ id = "storageIntegration",
+ order = "1100",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.storageIntegration.Label",
+ toolTip = "i18n::BeamSnowflake.storageIntegration.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = STAGING,
+ groupOrder = "0200")
+ private String storageIntegration;
+
+ @HopMetadataProperty(key = "quotation_mark")
+ @GuiWidgetElement(
+ id = "quotationMark",
+ order = "1200",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflake.quotationMark.Label",
+ toolTip = "i18n::BeamSnowflake.quotationMark.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = STAGING,
+ groupOrder = "0200")
+ private String quotationMark;
+
+ @HopMetadataProperty(key = "table_name")
+ @GuiWidgetElement(
+ id = "tableName",
+ order = "1300",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflakeInput.tableName.Label",
+ toolTip = "i18n::BeamSnowflakeInput.tableName.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = READ,
+ groupOrder = "0300")
+ private String tableName;
+
+ @HopMetadataProperty(key = "query")
+ @GuiWidgetElement(
+ id = "query",
+ order = "1400",
+ type = GuiElementType.MULTI_LINE_TEXT,
+ multiLineTextHeight = 4,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ label = "i18n::BeamSnowflakeInput.query.Label",
+ toolTip = "i18n::BeamSnowflakeInput.query.Tooltip",
+ groupType = GuiWidgetGroupType.TABS,
+ group = READ,
+ groupOrder = "0300")
+ private String query;
+
+ @HopMetadataProperty(groupKey = "fields", key = "field")
+ private List<SnowflakeField> fields = new ArrayList<>();
+
+ public SnowflakeSpec spec(IVariables variables) throws HopException {
+ return SnowflakeSpec.build(
+ variables,
+ serverName,
+ username,
+ password,
+ privateKey,
+ privateKeyPassphrase,
+ database,
+ warehouse,
+ schema,
+ role,
+ port,
+ stagingBucket,
+ storageIntegration,
+ quotationMark,
+ tableName,
+ query,
+ true);
+ }
+
+ public RowMeta outputRowMeta(IVariables variables, String origin) throws HopException {
+ RowMeta rowMeta = new RowMeta();
+ if (fields == null || fields.isEmpty())
+ throw new HopException("Snowflake input requires at least one field");
+ for (SnowflakeField field : fields) {
+ if (field == null || StringUtils.isBlank(field.getName()))
+ throw new HopException("Snowflake field name must be an identifier");
+ String name = SnowflakeSpec.columnName(variables.resolve(field.getName()).trim());
+ if (rowMeta.indexOfValue(name) >= 0)
+ throw new HopException("Snowflake field name is duplicated");
+ int type = SnowflakeValues.hopType(field.getType());
+ IValueMeta value = ValueMetaFactory.createValueMeta(name, type);
+ value.setOrigin(origin);
+ rowMeta.addValueMeta(value);
+ }
+ return rowMeta;
+ }
+
+ public BeamSnowflakeInputTransform buildRead(IVariables variables, String name)
+ throws HopException {
+ SnowflakeSpec spec = spec(variables);
+ RowMeta rowMeta = outputRowMeta(variables, name);
+ String rowMetaXml;
+ try {
+ rowMetaXml = rowMeta.getMetaXml();
+ } catch (IOException e) {
+ throw new HopException("Unable to serialize Snowflake input fields", e);
+ }
+ Coder<HopRow> coder = new HopRowCoder();
+ SnowflakeIO.Read<HopRow> read =
+ SnowflakeIO.<HopRow>read()
+ .withDataSourceConfiguration(spec.getDataSource())
+ .withStagingBucketName(spec.getStagingBucket())
+ .withStorageIntegrationName(spec.getStorageIntegration())
+ .withCsvMapper(new SnowflakeCsvToHop(name, rowMetaXml))
+ .withCoder(coder);
+ read =
+ spec.getQuery() != null ? read.fromQuery(spec.getQuery()) : read.fromTable(spec.getTable());
+ if (spec.getQuotationMark() != null) read = read.withQuotationMark(spec.getQuotationMark());
+ return new BeamSnowflakeInputTransform(read);
+ }
+
+ @Override
+ public void handleTransform(
+ ILogChannel log,
+ IVariables variables,
+ String runConfigurationName,
+ IBeamPipelineEngineRunConfiguration runConfiguration,
+ String dataSamplersJson,
+ IHopMetadataProvider metadataProvider,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ Map<String, PCollection<HopRow>> transformCollectionMap,
+ org.apache.beam.sdk.Pipeline pipeline,
+ IRowMeta rowMeta,
+ List<TransformMeta> previousTransforms,
+ PCollection<HopRow> input,
+ String parentLogChannelId)
+ throws HopException {
+ BeamSourceTopology.rejectIncomingRows(
+ pipelineMeta,
+ transformMeta,
+ previousTransforms,
+ input,
+ "Beam Snowflake input is a source and does not accept incoming rows");
+ transformCollectionMap.put(
+ transformMeta.getName(),
+ pipeline.apply(transformMeta.getName(), buildRead(variables, transformMeta.getName())));
+ }
+
+ @Override
+ public void check(
+ List<ICheckResult> remarks,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ IRowMeta prev,
+ String[] input,
+ String[] output,
+ IRowMeta info,
+ IVariables variables,
+ IHopMetadataProvider metadataProvider) {
+ try {
+ if (input != null && input.length > 0)
+ throw new HopException(
+ "Beam Snowflake input is a source and does not accept incoming rows");
+ buildRead(variables, transformMeta.getName());
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_OK,
+ BaseMessages.getString(
+ BeamSnowflakeInputMeta.class, "BeamSnowflakeInput.ConfigurationValid"),
+ transformMeta));
+ } catch (HopException e) {
+ remarks.add(new CheckResult(ICheckResult.TYPE_RESULT_ERROR, e.getMessage(), transformMeta));
+ }
+ }
+
+ @Override
+ public String getDialogClassName() {
+ return BeamSnowflakeInputDialog.class.getName();
+ }
+
+ @Override
+ public boolean isInput() {
+ return true;
+ }
+
+ @Override
+ public boolean isOutput() {
+ return false;
+ }
+
+ @Override
+ public boolean consumesMainInput() {
+ return false;
+ }
+
+ @Override
+ public boolean canStartWithoutInput() {
+ return true;
+ }
+
+ @Override
+ public void getFields(
+ IRowMeta row,
+ String name,
+ IRowMeta[] info,
+ TransformMeta next,
+ IVariables variables,
+ IHopMetadataProvider provider)
+ throws HopTransformException {
+ try {
+ RowMeta output = outputRowMeta(variables, name);
+ row.clear();
+ for (int i = 0; i < output.size(); i++) row.addValueMeta(output.getValueMeta(i));
+ } catch (HopException e) {
+ throw new HopTransformException(e.getMessage(), e);
+ }
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeOutput.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeOutput.java
new file mode 100644
index 0000000000..90df4fe60a
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeOutput.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.transforms.snowflake;
+
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.BaseTransform;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+public class BeamSnowflakeOutput
+ extends BaseTransform<BeamSnowflakeOutputMeta, BeamSnowflakeOutputData> {
+ public BeamSnowflakeOutput(
+ TransformMeta transformMeta,
+ BeamSnowflakeOutputMeta meta,
+ BeamSnowflakeOutputData data,
+ int copyNr,
+ PipelineMeta pipelineMeta,
+ Pipeline pipeline) {
+ super(transformMeta, meta, data, copyNr, pipelineMeta, pipeline);
+ }
+
+ @Override
+ public boolean processRow() throws HopException {
+ throw new HopException("Beam Snowflake output can only run in a Beam pipeline");
+ }
+}
diff --git a/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeOutputData.java b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeOutputData.java
new file mode 100644
index 0000000000..90650ad487
--- /dev/null
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/transforms/snowflake/BeamSnowflakeOutputData.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
... 12729 lines suppressed ...