You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hop.apache.org by mc...@apache.org on 2026/07/19 07:31:31 UTC
(hop) branch main updated: issue #7486 : A dedicated Spark pipeline execution engine (#7539)
This is an automated email from the ASF dual-hosted git repository.
mattcasters pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new c84f65c4f6 issue #7486 : A dedicated Spark pipeline execution engine (#7539)
c84f65c4f6 is described below
commit c84f65c4f69cb453beac789128ad4c27aedfac5b
Author: Matt Casters <ma...@gmail.com>
AuthorDate: Sun Jul 19 09:31:26 2026 +0200
issue #7486 : A dedicated Spark pipeline execution engine (#7539)
* issue #7486 : A dedicated Spark pipeline execution engine
* issue #7486 : A dedicated Spark pipeline execution engine (.gitignore)
* issue #7486 : A dedicated Spark pipeline execution engine (execution information location)
* issue #7486 : A dedicated Spark pipeline execution engine (extra integration tests)
* issue #7486 : A dedicated Spark pipeline execution engine (info streams via side loading)
* issue #7486 : A dedicated Spark pipeline execution engine (target streams, multi-input, docs)
Add target-stream fan-out for Filter Rows and Switch/Case, union of multiple
previous transforms with the same row layout, no-header CSV positional mapping,
metrics fix for leaf mapPartitions transforms, integration tests, and user-manual
getting-started documentation for the native Spark engine.
* issue #7486 : A dedicated Spark pipeline execution engine (classic I/O, ITs, docs)
Support classic file I/O on mapPartitions (accept-filenames, partition-scoped
Internal.Transform.*), active-graph collection for disabled hops, integration
tests 0007/0008, and a full native Spark getting started guide with screenshots.
* issue #7486 : A dedicated Spark pipeline execution engine (lakehouse Delta/Iceberg)
Add optional open table format support on the native Spark engine: Lake Table
Input/Output/Merge/Maintenance, Spark Catalog metadata, connector probe and
session plan, Maven -Plakehouse tests, user docs, and sample walkthroughs.
Connectors remain operator-installed (not in default assembly).
* issue #7486 : A dedicated Spark pipeline execution engine (ship lakehouse connectors)
Embed Delta Lake 4.3.1 and Apache Iceberg 1.11.0 in the default engines-spark
assembly under lib/delta and lib/iceberg. native-provided fat jars keep these
connectors; docs and probe messages updated for shipped packaging.
* issue #7486 : A dedicated Spark pipeline execution engine (lake table MDI)
Enable metadata injection on Spark Lake Table Input/Output/Merge/Maintenance
so multi-table landings, CDC merges, and fleet maintenance can use template
pipelines. Add injection unit tests and document inject keys.
* issue #7486 : A dedicated Spark pipeline execution engine (lakehouse/iceberg ITs)
Add integration-tests/lakehouse (Delta) and integration-tests/iceberg projects
with PATH Input/Output, overwrite + time travel (Delta VERSION 0), and MERGE
upsert workflows. Wire Docker compose and docs cross-links.
* issue #7486 : apache-rat : exclude empty .gitkeep files
* issue #7486 : fix lakehouse/iceberg IT cleanup for Docker permissions
Use /tmp/hop-it-{lakehouse,iceberg} for tables and extracts, and replace
DELETE_FILES with a Shell cleanup that always exits 0. Avoids failures
deleting host-owned Spark .crc files on the mounted project volume.
* issue #7486 : add GuiElementType.MULTI_LINE_TEXT for multi-line config fields
Introduce MULTI_LINE_TEXT with multiLineTextHeight (lines, default 1) on
@GuiWidgetElement, wire creation/layout/bind in GuiCompositeWidgets, and use
it for native Spark sparkConfigs (5 lines).
* issue #7486 : Load configuration template for Native Spark run config
Add a Load configuration template button with Local, capped local,
standalone, spark-submit, Databricks, YARN, and lakehouse presets.
Wire PipelineRunConfigurationEditor button listener and refresh widgets
after button actions so form fields update.
* issue #7486 : Load catalog template for Spark Catalog; rebind BUTTON widgets
Add Load catalog template presets (Iceberg Hadoop/REST, Hive, Glue, Nessie,
Unity, Delta) with # docs: links in conf extra. Wire SparkCatalogEditor
button flush/rebind, multi-line conf extra, and GuiCompositeWidgets
afterButtonPressed refresh so form fields update after modal template apply.
Document templates in spark-catalog and cross-link from lakehouse/native Spark.
* issue #7486 : Databricks Connection metadata and Job Run action
Add workspace connection (PAT) for the Jobs REST API, a RestDatabricksJobsClient
with WireMock tests, and a Big Data workflow action to run-now or one-time
submit with wait/fire-and-forget and result variables. Document connection
and action; nav and metadata index links.
* issue #7486 : Databricks Job Run deploy mode (DBFS upload + MainSpark JAR)
Add Deploy & run: upload fat jar, pipeline, and metadata JSON to DBFS,
create or update a JAR job (MainSpark) on an existing cluster, then
run-now. JobSpecFactory, HopSparkDeployHelper, client uploadToDbfs,
dialog fields, tests, and docs.
* issue #7486 : Databricks Job Wait action (poll runs/get)
Extract shared DatabricksRunWaiter used by Job Run and new Job Wait.
Job Wait polls by run id (default ${DatabricksRunId}) with timeout,
poll interval, cancel-on-stop, and the same result variables. Docs and
nav cross-links for fire-and-forget + wait orchestration.
* issue #7486 : Clarify Hop VFS vs Spark/Hadoop path dialects
Document that Spark File/Lake paths use Hadoop URIs (s3a://, hdfs://),
not Hop VFS (s3://, named MinIO). Relabel UI fields, fix cluster examples,
and append error hints via SparkPathDialect when known Hop-only schemes
appear in Dataset I/O failures.
* issue #7486 : Path scheme map for Spark File/Lake URIs
Add an optional from=to path scheme map on the Native Spark run
configuration so Dataset and Lake PATH I/O can rewrite schemes (e.g.
s3=s3a, minio=s3a) after variable resolution. Classic Hop VFS is
unchanged; FS credentials still come from Spark/Hadoop config.
* issue #7486 : Spark project package for nested pipelines on cluster
Add a Native Spark project package (export via Tools menu and hop-conf
--export-spark-project) so Simple Mapping and Pipeline Executor can resolve
${PROJECT_HOME} children on workers. MainSpark accepts --HopProjectPackage;
the engine distributes the zip with SparkContext.addFile and mini-pipelines
materialize PROJECT_HOME from SparkFiles on each executor.
* issue #7486 : Docker walk-through for Native Spark mapping package
Add spark-mapping-demo sample project, Spark 4.1 compose cluster with
shared data volume, prepare-dist and spark-submit-demo scripts, and a
user-manual walk-through so Simple Mapping under PROJECT_HOME can be
verified on multi-node spark-submit with a project package.
* issue #7486 : Fix Spark project package export without PROJECT_HOME
--project/-p only names a managed project and does not set PROJECT_HOME.
Export now accepts --export-spark-project-home and loads metadata from
that folder; prepare-dist.sh no longer requires registering the sample.
* issue #7486 : Fix package extract on Spark workers and demo defaults
Open SparkFiles-local package zips via java.io (HopVfs misses bare
absolute paths). Default DIST_DIR to /tmp/spark-mapping-demo-dist.
Spark 4 master uses --host instead of --ip.
* issue #7486 : Shared-volume package path for Spark workers
Stop stashing SparkFiles userFiles paths in HOP_SPARK_PROJECT_PACKAGE
(not valid on other JVMs). Resolve package from SparkFiles, then
$HOP_DATA/packages/<zip>, then the portable URI. Demo submit copies the
zip onto the shared volume and passes --files as backup.
* issue #7486 : apache-rat
* issue #7486 : Spark run mode, nested pipelines, and expanded spark-demo
Add DISTRIBUTED/DRIVER_ONLY generic transform run mode with per-transform
override and canvas badge, nested Native Spark session reuse for Pipeline
Executor, Hop VFS-safe execution-info cache writes, and rename/expand the
cluster sample to spark-demo (workflows, nested pipelines, host hop-data
bind mount).
* issue #7486 : bug fixing, hardening special cases
* issue #7486 : bug fixing execution information locations
* issue #7486 : transforms documentation updates and fixes
* issue #7486 : pipeline engine capabilities alignment
* issue #7486 : fix IT Excel/ODS write permissions and surefire summary paths
Write temp Excel/ODS outputs under project output/ (world-writable), align
container UID defaults with the ASF Jenkins agent identity, and fix relative
path bugs when printing passed/failed test summaries from the repo root.
* issue #7486 : fix PipelineExecutor sticky params, JsonOutput parent folders, hop_server rebuild
Clear empty field mappings on PipelineExecutor so child parameter defaults
apply (IT main-0003). Harden JsonOutput parent-folder creation and messages
for Beam 0005. Force docker compose --build for hop_server so remote export
ITs use a server image matching current assemblies.
* issue #7486 : spark-native output cleanup, skip Google Sheets without GCP key
Wipe residual Spark output under project output/ (including other-UID files)
before IT runs so delete-output steps succeed. Skip spreadsheet Google Sheets
workflows when GCP_KEY_FILE is missing or not a service-account JSON, and
surface a clear credentials error from GoogleSheetsCredentials.
* issue #7486 : honor SKIP_GOOGLE_SHEETS under single-JVM suite runner
The Google Sheets skip only ran on the classic per-workflow path. Spreadsheet
ITs default to run-project-tests.hpl, so main-0009/0010 still executed with the
dummy GCP key. Force the classic runner for the spreadsheet project when
SKIP_GOOGLE_SHEETS=true so those workflows are skipped.
* issue #7486 : preserve .gitkeep when cleaning IT output dirs
Do not delete tracked .gitkeep/.gitignore under project output/ when wiping
residual Spark/UID-mismatched files before integration tests.
* issue #7486 : do not wipe tracked files under IT output/
LDAP stores setup pipelines in ldap/output/*.hpl. The pre-test output cleanup
was deleting everything except .gitkeep/.gitignore, which broke main-ldap-tests.
Only remove untracked residuals (git clean) and Spark part-*/.crc artifacts.
* issue #7486 : .gitignore
* issue #7486 : getting Databricks to work
* issue #7486 : getting Databricks to work Deploy & Run fixes, mappings
* issue #7486 : Databricks package materialize, content-addressed fat jar, library-cache lesson
Stage UC Volume project packages to local disk for SparkFiles addFile so
executors rematerialize PROJECT_HOME (mappings and nested pipelines).
Deploy fat jars as hop-native-<sha12>.jar so Dedicated clusters pick up new
library URIs after rebuilds; document overwrite-vs-classpath caching and
cluster restart. MainSpark prints pkg-dist fingerprint for verification.
---
.gitignore | 6 +-
NOTICE | 10 +
assemblies/client/pom.xml | 6 +
assemblies/debug/pom.xml | 6 +
assemblies/plugins/pom.xml | 6 +
.../apache/hop/core/gui/plugin/GuiElementType.java | 5 +
.../apache/hop/core/gui/plugin/GuiElements.java | 21 +
.../hop/core/gui/plugin/GuiWidgetElement.java | 8 +
dist/spark-native-demo/README.md | 33 +
.../integration-tests/integration-tests-base.yaml | 2 +
.../integration-tests-beam-base.yaml | 1 +
...ts-base.yaml => integration-tests-iceberg.yaml} | 23 +-
...-base.yaml => integration-tests-lakehouse.yaml} | 23 +-
.../integration-tests-spark-native-cluster.yaml | 105 ++
...se.yaml => integration-tests-spark-native.yaml} | 25 +-
docker/integration-tests/spark/scripts/master.sh | 3 +-
...obs-pipelines-hello-databricks-initializing.png | Bin 0 -> 122449 bytes
...pipelines-hello-databricks-memory-usage-4gb.png | Bin 0 -> 53146 bytes
...cks-jobs-pipelines-hello-databricks-running.png | Bin 0 -> 52257 bytes
...s-jobs-pipelines-hello-databricks-succeeded.png | Bin 0 -> 46502 bytes
.../spark/databricks-pipeline-hello-databricks.png | Bin 0 -> 121805 bytes
.../databricks-workflow-deploy-to-databricks.png | Bin 0 -> 253662 bytes
.../databricks-workspace-catalog-jars-volume.png | Bin 0 -> 125586 bytes
...park-logs-to-execution-information-location.png | Bin 0 -> 238771 bytes
...ark-pipeline-engine-run-configurtion-editor.png | Bin 0 -> 142110 bytes
...rovides-runtime-transform-metrics-and-state.png | Bin 0 -> 182751 bytes
.../spark/native-spark-run-mode-force-driver.png | Bin 0 -> 59878 bytes
.../spark/spark-native-0001-input-output.png | Bin 0 -> 16991 bytes
.../pipeline/spark/spark-native-0002-group-by.png | Bin 0 -> 26542 bytes
.../spark/spark-native-0003-filter-rows.png | Bin 0 -> 27878 bytes
.../spark/spark-native-0004-stream-lookup.png | Bin 0 -> 49898 bytes
.../spark/spark-native-0005-filter-targets.png | Bin 0 -> 59714 bytes
.../spark/spark-native-0005-switch-case.png | Bin 0 -> 88023 bytes
.../spark-native-0008-excel-stream-lookup.png | Bin 0 -> 43200 bytes
...-native-0008-text-file-input-spark-readback.png | Bin 0 -> 27554 bytes
.../pipeline/spark/spark-native-0099-complex.png | Bin 0 -> 207282 bytes
.../assets/images/transforms/icons/database.svg | 11 +
.../images/transforms/icons/spark-catalog.svg | 5 +
.../transforms/icons/spark-lake-table-input.svg | 5 +
.../icons/spark-lake-table-maintenance.svg | 4 +
.../transforms/icons/spark-lake-table-merge.svg | 4 +
.../transforms/icons/spark-lake-table-output.svg | 4 +
docs/hop-user-manual/modules/ROOT/nav.adoc | 13 +
.../metadata-types/databricks-connection.adoc | 62 +
.../modules/ROOT/pages/metadata-types/index.adoc | 2 +
.../ROOT/pages/metadata-types/spark-catalog.adoc | 150 ++
.../native-spark-pipeline-engine.adoc | 218 +++
.../pipeline-run-configurations.adoc | 1 +
.../modules/ROOT/pages/pipeline/pipelines.adoc | 2 +
.../spark/cluster-mapping-walkthrough.adoc | 328 ++++
.../ROOT/pages/pipeline/spark/databricks.adoc | 544 ++++++
.../spark/getting-started-with-native-spark.adoc | 706 ++++++++
.../ROOT/pages/pipeline/spark/lakehouse.adoc | 314 ++++
.../pipeline/spark/paths-and-filesystems.adoc | 312 ++++
.../modules/ROOT/pages/pipeline/transforms.adoc | 4 +
.../ROOT/pages/pipeline/transforms/abort.adoc | 10 +-
.../pages/pipeline/transforms/accessoutput.adoc | 8 +-
.../pages/pipeline/transforms/addchecksum.adoc | 8 +-
.../pages/pipeline/transforms/addconstant.adoc | 8 +-
.../transforms/addfieldschangesequence.adoc | 8 +-
.../pages/pipeline/transforms/addsequence.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/addxml.adoc | 8 +-
.../pages/pipeline/transforms/analyticquery.adoc | 8 +-
.../pages/pipeline/transforms/apache-tika.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/append.adoc | 8 +-
.../pages/pipeline/transforms/avro-decode.adoc | 8 +-
.../pages/pipeline/transforms/avro-encode.adoc | 8 +-
.../pages/pipeline/transforms/avro-file-input.adoc | 8 +-
.../pipeline/transforms/avro-file-output.adoc | 8 +-
.../pages/pipeline/transforms/aws-sns-notify.adoc | 8 +-
.../pages/pipeline/transforms/aws-sqs-reader.adoc | 8 +-
.../transforms/azure-event-hubs-listener.adoc | 8 +-
.../transforms/azure-event-hubs-writer.adoc | 10 +-
.../pipeline/transforms/beambigqueryinput.adoc | 10 +-
.../pipeline/transforms/beambigqueryoutput.adoc | 10 +-
.../pipeline/transforms/beambigtableinput.adoc | 8 +-
.../pipeline/transforms/beambigtableoutput.adoc | 8 +-
.../pages/pipeline/transforms/beamfileinput.adoc | 8 +-
.../pages/pipeline/transforms/beamfileoutput.adoc | 8 +-
.../pipeline/transforms/beamgcppublisher.adoc | 8 +-
.../pipeline/transforms/beamgcpsubscriber.adoc | 8 +-
.../pipeline/transforms/beamhivecataloginput.adoc | 8 +-
.../pipeline/transforms/beamkafkaconsume.adoc | 8 +-
.../pipeline/transforms/beamkafkaproduce.adoc | 8 +-
.../pipeline/transforms/beamkinesisconsume.adoc | 8 +-
.../pipeline/transforms/beamkinesisproduce.adoc | 8 +-
.../pages/pipeline/transforms/beamtimestamp.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/beamwindow.adoc | 8 +-
.../pipeline/transforms/blockingtransform.adoc | 8 +-
.../transforms/blockuntiltransformsfinish.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/calculator.adoc | 12 +-
.../ROOT/pages/pipeline/transforms/calldbproc.adoc | 8 +-
.../pages/pipeline/transforms/cassandra-input.adoc | 8 +-
.../pipeline/transforms/cassandra-output.adoc | 8 +-
.../pipeline/transforms/changefileencoding.adoc | 8 +-
.../pages/pipeline/transforms/checkfilelocked.adoc | 8 +-
.../transforms/checkwebserviceavailable.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/clonerow.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/closure.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/coalesce.adoc | 8 +-
.../pages/pipeline/transforms/columnexists.adoc | 8 +-
.../pipeline/transforms/combinationlookup.adoc | 10 +-
.../pages/pipeline/transforms/concatfields.adoc | 8 +-
.../pipeline/transforms/copyrowstoresult.adoc | 10 +-
.../pipeline/transforms/cratedb-bulkloader.adoc | 8 +-
.../pipeline/transforms/creditcardvalidator.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/csvinput.adoc | 8 +-
.../pipeline/transforms/data-stream-input.adoc | 21 +-
.../pipeline/transforms/data-stream-output.adoc | 20 +-
.../pages/pipeline/transforms/databasejoin.adoc | 8 +-
.../pages/pipeline/transforms/databaselookup.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/datagrid.adoc | 10 +-
.../pages/pipeline/transforms/datasetinput.adoc | 8 +-
.../pages/pipeline/transforms/dbimpactinput.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/ddl.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/delay.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/delete.adoc | 10 +-
.../pipeline/transforms/detectemptystream.adoc | 10 +-
.../pages/pipeline/transforms/detectlanguage.adoc | 10 +-
.../pages/pipeline/transforms/dimensionlookup.adoc | 10 +-
.../pages/pipeline/transforms/dorisbulkloader.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/dummy.adoc | 10 +-
.../pages/pipeline/transforms/dynamicsqlrow.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/edi2xml.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/emailinput.adoc | 10 +-
.../pipeline/transforms/enhancedjsonoutput.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/excelinput.adoc | 8 +-
.../pages/pipeline/transforms/excelwriter.adoc | 27 +-
.../ROOT/pages/pipeline/transforms/execinfo.adoc | 13 +-
.../pages/pipeline/transforms/execprocess.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/execsql.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/execsqlrow.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/exectests.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/fake.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/fileexists.adoc | 10 +-
.../pages/pipeline/transforms/filemetadata.adoc | 10 +-
.../pages/pipeline/transforms/filesfromresult.adoc | 8 +-
.../pages/pipeline/transforms/filestoresult.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/filterrows.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/formula.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/fuzzymatch.adoc | 8 +-
.../pipeline/transforms/generaterandomvalue.adoc | 8 +-
.../pages/pipeline/transforms/getdatafromxml.adoc | 73 +-
.../pages/pipeline/transforms/getfilenames.adoc | 14 +-
.../pipeline/transforms/getfilesrowcount.adoc | 8 +-
.../pipeline/transforms/getrecordsfromstream.adoc | 10 +-
.../pipeline/transforms/getrowsfromresult.adoc | 10 +-
.../pages/pipeline/transforms/getsubfolders.adoc | 8 +-
.../pages/pipeline/transforms/getsystemdata.adoc | 8 +-
.../pages/pipeline/transforms/gettablenames.adoc | 8 +-
.../pages/pipeline/transforms/getvariable.adoc | 8 +-
.../pipeline/transforms/google-analytics.adoc | 8 +-
.../pipeline/transforms/google-sheets-input.adoc | 8 +-
.../pipeline/transforms/google-sheets-output.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/groupby.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/html2text.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/http.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/httppost.adoc | 8 +-
.../pages/pipeline/transforms/identifylastrow.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/ifnull.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/injector.adoc | 10 +-
.../pages/pipeline/transforms/insertupdate.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/javafilter.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/javascript.adoc | 8 +-
.../pages/pipeline/transforms/jdbcmetadata.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/joinrows.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/jsoninput.adoc | 29 +-
.../pipeline/transforms/jsonnormalizeinput.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/jsonoutput.adoc | 8 +-
.../pages/pipeline/transforms/kafkaconsumer.adoc | 10 +-
.../pages/pipeline/transforms/kafkaproducer.adoc | 8 +-
.../pipeline/transforms/languagemodelchat.adoc | 12 +-
.../ROOT/pages/pipeline/transforms/ldapinput.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/ldapoutput.adoc | 8 +-
.../pages/pipeline/transforms/loadfileinput.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/mail.adoc | 8 +-
.../pages/pipeline/transforms/mapping-input.adoc | 8 +-
.../pages/pipeline/transforms/mapping-output.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/memgroupby.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/mergejoin.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/mergerows.adoc | 8 +-
.../pages/pipeline/transforms/metadata-input.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/metainject.adoc | 8 +-
.../pages/pipeline/transforms/metastructure.adoc | 8 +-
.../pipeline/transforms/monetdbbulkloader.adoc | 8 +-
.../pages/pipeline/transforms/mongodbdelete.adoc | 8 +-
.../pages/pipeline/transforms/mongodbinput.adoc | 8 +-
.../pages/pipeline/transforms/mongodboutput.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/multimerge.adoc | 8 +-
.../pages/pipeline/transforms/mysqlbulkloader.adoc | 8 +-
.../pages/pipeline/transforms/neo4j-cypher.adoc | 8 +-
.../pages/pipeline/transforms/neo4j-gencsv.adoc | 8 +-
.../pipeline/transforms/neo4j-getloginfo.adoc | 8 +-
.../pipeline/transforms/neo4j-graphoutput.adoc | 8 +-
.../pages/pipeline/transforms/neo4j-import.adoc | 8 +-
.../pages/pipeline/transforms/neo4j-output.adoc | 8 +-
.../pipeline/transforms/neo4j-split-graph.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/nullif.adoc | 8 +-
.../pages/pipeline/transforms/numberrange.adoc | 8 +-
.../pages/pipeline/transforms/odata-input.adoc | 8 +-
.../pages/pipeline/transforms/orabulkloader.adoc | 10 +-
.../pipeline/transforms/parquet-file-input.adoc | 8 +-
.../pipeline/transforms/parquet-file-output.adoc | 8 +-
.../pipeline/transforms/pgpdecryptstream.adoc | 8 +-
.../pipeline/transforms/pgpencryptstream.adoc | 8 +-
.../pipeline/transforms/pipeline-data-probe.adoc | 8 +-
.../pipeline/transforms/pipeline-executor.adoc | 10 +-
.../pipeline/transforms/pipeline-logging.adoc | 8 +-
.../pipeline/transforms/postgresbulkloader.adoc | 8 +-
.../pages/pipeline/transforms/processfiles.adoc | 8 +-
.../pages/pipeline/transforms/propertyinput.adoc | 57 +-
.../pages/pipeline/transforms/propertyoutput.adoc | 8 +-
.../pipeline/transforms/redshift-bulkloader.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/regexeval.adoc | 8 +-
.../pages/pipeline/transforms/repeatfields.adoc | 8 +-
.../pages/pipeline/transforms/replacestring.adoc | 8 +-
.../pipeline/transforms/reservoirsampling.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/rest.adoc | 8 +-
.../pages/pipeline/transforms/rowdenormaliser.adoc | 8 +-
.../pages/pipeline/transforms/rowflattener.adoc | 8 +-
.../pages/pipeline/transforms/rowgenerator.adoc | 8 +-
.../pages/pipeline/transforms/rownormaliser.adoc | 8 +-
.../pipeline/transforms/rulesaccumulator.adoc | 8 +-
.../pages/pipeline/transforms/rulesexecutor.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/runssh.adoc | 8 +-
.../pipeline/transforms/salesforcedelete.adoc | 8 +-
.../pages/pipeline/transforms/salesforceinput.adoc | 8 +-
.../pipeline/transforms/salesforceinsert.adoc | 8 +-
.../pipeline/transforms/salesforceupdate.adoc | 8 +-
.../pipeline/transforms/salesforceupsert.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/samplerows.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/sasinput.adoc | 8 +-
.../pages/pipeline/transforms/schemamapping.adoc | 10 +-
.../ROOT/pages/pipeline/transforms/script.adoc | 8 +-
.../pages/pipeline/transforms/selectvalues.adoc | 8 +-
.../transforms/serialize-de-from-file.adoc | 8 +-
.../pipeline/transforms/serialize-to-file.adoc | 8 +-
.../pages/pipeline/transforms/serverstatus.adoc | 8 +-
.../pipeline/transforms/setvalueconstant.adoc | 8 +-
.../pages/pipeline/transforms/setvaluefield.adoc | 8 +-
.../pages/pipeline/transforms/setvariable.adoc | 8 +-
.../pages/pipeline/transforms/simple-mapping.adoc | 8 +-
.../pipeline/transforms/snowflakebulkloader.adoc | 10 +-
.../pages/pipeline/transforms/snowflakeid.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/sort.adoc | 8 +-
.../pages/pipeline/transforms/sortedmerge.adoc | 8 +-
.../pipeline/transforms/sortedschemamerge.adoc | 8 +-
.../transforms/spark-lake-table-input.adoc | 130 ++
.../transforms/spark-lake-table-maintenance.adoc | 140 ++
.../transforms/spark-lake-table-merge.adoc | 119 ++
.../transforms/spark-lake-table-output.adoc | 113 ++
.../pages/pipeline/transforms/splitfields.adoc | 8 +-
.../pipeline/transforms/splitfieldtorows.adoc | 8 +-
.../pages/pipeline/transforms/splunkinput.adoc | 8 +-
.../pages/pipeline/transforms/sqlfileoutput.adoc | 8 +-
.../pages/pipeline/transforms/sstable-output.adoc | 8 +-
.../transforms/standardizephonenumber.adoc | 8 +-
.../pages/pipeline/transforms/stanfordnlp.adoc | 8 +-
.../pages/pipeline/transforms/streamlookup.adoc | 8 +-
.../pipeline/transforms/streamschemamerge.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/stringcut.adoc | 8 +-
.../pipeline/transforms/stringoperations.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/switchcase.adoc | 8 +-
.../pipeline/transforms/synchronizeaftermerge.adoc | 8 +-
.../pages/pipeline/transforms/tablecompare.adoc | 8 +-
.../pages/pipeline/transforms/tableexists.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/tableinput.adoc | 8 +-
.../pages/pipeline/transforms/tableoutput.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/terafast.adoc | 8 +-
.../pages/pipeline/transforms/textfileinput.adoc | 8 +-
.../pages/pipeline/transforms/textfileoutput.adoc | 8 +-
.../pipeline/transforms/tokenreplacement.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/uniquerows.adoc | 8 +-
.../pipeline/transforms/uniquerowsbyhashset.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/update.adoc | 8 +-
.../pipeline/transforms/userdefinedjavaclass.adoc | 8 +-
.../transforms/userdefinedjavaexpression.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/validator.adoc | 8 +-
.../pages/pipeline/transforms/valuemapper.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/vcardinput.adoc | 8 +-
.../pages/pipeline/transforms/vcardoutput.adoc | 8 +-
.../pipeline/transforms/verticabulkloader.adoc | 8 +-
.../pages/pipeline/transforms/webservices.adoc | 8 +-
.../pipeline/transforms/workflow-executor.adoc | 8 +-
.../pipeline/transforms/workflow-logging.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/writetolog.adoc | 8 +-
.../pages/pipeline/transforms/xmlinputstream.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/xmljoin.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/xmloutput.adoc | 8 +-
.../pipeline/transforms/xmloutputadvanced.adoc | 8 +-
.../pages/pipeline/transforms/xsdvalidator.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/xslt.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/yamlinput.adoc | 8 +-
.../ROOT/pages/pipeline/transforms/zipfile.adoc | 8 +-
docs/hop-user-manual/modules/ROOT/pages/vfs.adoc | 7 +
.../pages/workflow/actions/databricks-job-run.adoc | 181 ++
.../workflow/actions/databricks-job-wait.adoc | 66 +
.../caching/BaseCachingExecutionInfoLocation.java | 118 +-
.../apache/hop/execution/caching/CacheEntry.java | 68 +-
.../caching/CachingFileExecutionInfoLocation.java | 70 +-
.../transforms/file/BaseFileInputMeta.java | 11 +
.../engines/local/LocalWorkflowEngine.java | 55 +-
.../CachingFileExecutionInfoLocationTest.java | 266 +++
.../beam_directrunner/main-0005-single-thread.hwf | 19 +
integration-tests/iceberg/.gitignore | 2 +
integration-tests/iceberg/0001-read-extract.hpl | 106 ++
integration-tests/iceberg/0001-validation.hpl | 167 ++
integration-tests/iceberg/0001-write.hpl | 120 ++
integration-tests/iceberg/0002-read-current.hpl | 106 ++
.../iceberg/0002-validation-current.hpl | 167 ++
integration-tests/iceberg/0002-write-v1.hpl | 115 ++
integration-tests/iceberg/0002-write-v2.hpl | 130 ++
integration-tests/iceberg/0003-merge.hpl | 113 ++
integration-tests/iceberg/0003-read-extract.hpl | 106 ++
integration-tests/iceberg/0003-seed.hpl | 112 ++
integration-tests/iceberg/0003-validation.hpl | 167 ++
integration-tests/iceberg/README.md | 56 +
.../iceberg/datasets/0001-roundtrip-golden.csv | 11 +
.../iceberg/datasets/0002-asof-golden.csv | 6 +
.../iceberg/datasets/0002-current-golden.csv | 21 +
.../iceberg/datasets/0003-merge-golden.csv | 4 +
integration-tests/iceberg/dev-env-config.json | 3 +
integration-tests/iceberg/hop-config.json | 290 +++
.../main-0001-input-output.hwf} | 121 +-
.../iceberg/main-0002-overwrite-timetravel.hwf | 215 +++
integration-tests/iceberg/main-0003-merge.hwf | 215 +++
.../metadata/dataset/0001-roundtrip-golden.json | 24 +
.../iceberg/metadata/dataset/0002-asof-golden.json | 24 +
.../metadata/dataset/0002-current-golden.json | 24 +
.../metadata/dataset/0003-merge-golden.json | 24 +
.../metadata/pipeline-run-configuration/local.json | 17 +
.../pipeline-run-configuration/spark-local.json | 18 +
.../metadata/unit-test/0001-validation UNIT.json | 32 +
.../unit-test/0002-validation-current UNIT.json | 32 +
.../metadata/unit-test/0003-validation UNIT.json | 32 +
.../metadata/workflow-run-configuration/local.json | 9 +
integration-tests/iceberg/output/.gitkeep | 0
integration-tests/iceberg/project-config.json | 15 +
integration-tests/lakehouse/.gitignore | 2 +
integration-tests/lakehouse/0001-read-extract.hpl | 106 ++
integration-tests/lakehouse/0001-validation.hpl | 167 ++
integration-tests/lakehouse/0001-write.hpl | 120 ++
integration-tests/lakehouse/0002-read-asof.hpl | 106 ++
integration-tests/lakehouse/0002-read-current.hpl | 106 ++
.../lakehouse/0002-validation-asof.hpl | 167 ++
.../lakehouse/0002-validation-current.hpl | 167 ++
integration-tests/lakehouse/0002-write-v1.hpl | 115 ++
integration-tests/lakehouse/0002-write-v2.hpl | 130 ++
integration-tests/lakehouse/0003-merge.hpl | 113 ++
integration-tests/lakehouse/0003-read-extract.hpl | 106 ++
integration-tests/lakehouse/0003-seed.hpl | 112 ++
integration-tests/lakehouse/0003-validation.hpl | 167 ++
integration-tests/lakehouse/README.md | 56 +
.../lakehouse/datasets/0001-roundtrip-golden.csv | 11 +
.../lakehouse/datasets/0002-asof-golden.csv | 6 +
.../lakehouse/datasets/0002-current-golden.csv | 21 +
.../lakehouse/datasets/0003-merge-golden.csv | 4 +
integration-tests/lakehouse/dev-env-config.json | 3 +
integration-tests/lakehouse/hop-config.json | 290 +++
.../main-0001-input-output.hwf} | 121 +-
.../lakehouse/main-0002-overwrite-timetravel.hwf | 253 +++
integration-tests/lakehouse/main-0003-merge.hwf | 215 +++
.../metadata/dataset/0001-roundtrip-golden.json | 24 +
.../metadata/dataset/0002-asof-golden.json | 24 +
.../metadata/dataset/0002-current-golden.json | 24 +
.../metadata/dataset/0003-merge-golden.json | 24 +
.../metadata/pipeline-run-configuration/local.json | 17 +
.../pipeline-run-configuration/spark-local.json | 18 +
.../metadata/unit-test/0001-validation UNIT.json | 32 +
.../unit-test/0002-validation-asof UNIT.json | 32 +
.../unit-test/0002-validation-current UNIT.json | 32 +
.../metadata/unit-test/0003-validation UNIT.json | 32 +
.../metadata/workflow-run-configuration/local.json | 9 +
integration-tests/lakehouse/output/.gitkeep | 0
integration-tests/lakehouse/project-config.json | 15 +
integration-tests/scripts/run-tests-docker.sh | 91 +-
integration-tests/scripts/run-tests.sh | 38 +-
.../spark-native/0001-input-output-validation.hpl | 289 +++
.../spark-native/0001-input-output.hpl | 152 ++
.../spark-native/0002-group-by-validation.hpl | 229 +++
integration-tests/spark-native/0002-group-by.hpl | 211 +++
.../spark-native/0003-filter-rows-validation.hpl | 289 +++
.../spark-native/0003-filter-rows.hpl | 193 ++
.../spark-native/0004-stream-lookup-validation.hpl | 304 ++++
.../spark-native/0004-stream-lookup.hpl | 230 +++
.../0005-filter-targets-validation.hpl | 328 ++++
.../spark-native/0005-filter-targets.hpl | 171 ++
.../spark-native/0006-switch-case-validation.hpl | 165 ++
.../spark-native/0006-switch-case.hpl | 251 +++
.../0007-text-file-input-create-test-files.hpl | 314 ++++
.../0007-text-file-input-spark-readback.hpl | 226 +++
.../spark-native/0007-text-file-input-validate.hpl | 307 ++++
.../0008-excel-stream-lookup-create-test-files.hpl | 317 ++++
.../0008-excel-stream-lookup-validation.hpl | 265 +++
.../spark-native/0008-excel-stream-lookup.hpl | 262 +++
integration-tests/spark-native/0099-complex.hpl | 662 +++++++
.../datasets/0001-customers-1k-golden.csv | 1001 +++++++++++
.../spark-native/datasets/0002-group-by-golden.csv | 61 +
.../datasets/0003-filter-rows-golden.csv | 48 +
.../datasets/0004-stream-lookup-golden.csv | 1001 +++++++++++
.../datasets/0005-filter-false-golden.csv | 954 ++++++++++
.../datasets/0005-filter-true-golden.csv | 48 +
.../datasets/0006-switch-ca-golden.csv | 20 +
.../datasets/0006-switch-default-golden.csv | 941 ++++++++++
.../datasets/0006-switch-fl-golden.csv | 21 +
.../datasets/0006-switch-ny-golden.csv | 22 +
.../datasets/0008-excel-stream-lookup-golden.csv | 7 +
.../spark-native/datasets/0099-complex-golden.csv | 2 +
.../spark-native/datasets/0099-customers-input.csv | 1002 +++++++++++
.../datasets/0099-state-data-input.csv | 52 +
integration-tests/spark-native/dev-env-config.json | 3 +
.../spark-native/files/customers-1k.txt | 1001 +++++++++++
.../spark-native/files/customers-noheader-1k.txt | 1000 +++++++++++
.../spark-native/files/state-population.txt | 51 +
integration-tests/spark-native/hop-config.json | 290 +++
integration-tests/spark-native/lakehouse/README.md | 34 +
.../main-0001-input-output.hwf} | 72 +-
.../main-0002-group-by.hwf} | 72 +-
.../main-0003-filter-rows.hwf} | 72 +-
.../main-0004-stream-lookup.hwf} | 72 +-
.../main-0005-filter-targets.hwf} | 76 +-
.../main-0006-switch-case.hwf} | 98 +-
.../spark-native/main-0007-text-file-input.hwf | 177 ++
.../main-0008-excel-stream-lookup.hwf} | 99 +-
.../metadata/dataset/0001-customers-1k-golden.json | 88 +
.../metadata/dataset/0002-group-by-golden.json | 56 +
.../metadata/dataset/0003-filter-rows-golden.json | 88 +
.../dataset/0004-stream-lookup-golden.json | 96 +
.../metadata/dataset/0005-filter-false-golden.json | 32 +
.../metadata/dataset/0005-filter-true-golden.json | 32 +
.../metadata/dataset/0006-switch-ca-golden.json | 24 +
.../dataset/0008-excel-stream-lookup-golden.json | 48 +
.../metadata/dataset/0099-complex-golden.json | 128 ++
.../metadata/dataset/0099-customers-input.json | 88 +
.../dataset/0099-state-population-input.json | 24 +
.../metadata/pipeline-run-configuration/local.json | 17 +
.../pipeline-run-configuration/spark-local.json | 18 +
.../0001-input-output-validation UNIT.json | 32 +
.../unit-test/0002-group-by-validation UNIT.json | 28 +
.../0003-filter-rows-validation UNIT.json | 32 +
.../0004-stream-lookup-validation UNIT.json | 33 +
.../0005-filter-targets-validation UNIT.json | 57 +
.../0006-switch-case-validation UNIT.json | 32 +
.../0008-excel-stream-lookup-validation UNIT.json | 27 +
.../metadata/unit-test/0099-complex UNIT.json | 152 ++
.../metadata/workflow-run-configuration/local.json | 9 +
integration-tests/spark-native/project-config.json | 13 +
.../transforms/0065-check-excel-file-exists.hpl | 2 +-
.../transforms/0065-create-excel-file.hpl | 2 +-
.../transforms/0101-check-ods-file-exists.hpl | 2 +-
.../transforms/0101-create-ods-file.hpl | 2 +-
integration-tests/transforms/output/.gitignore | 4 +
integration-tests/transforms/output/.gitkeep | 0
.../beam/config/GenerateFatJarConfigPlugin.java | 58 +-
.../org/apache/hop/beam/gui/HopBeamGuiPlugin.java | 216 ++-
.../beam/gui/HopBeamGuiPluginNativeFatJarTest.java | 128 ++
plugins/engines/pom.xml | 1 +
plugins/engines/spark/README.md | 455 +++++
plugins/engines/spark/pom.xml | 269 +++
plugins/engines/spark/src/assembly/assembly.xml | 90 +
.../config/ExportSparkProjectConfigPlugin.java | 188 ++
.../apache/hop/spark/core/HopMapPartitionsFn.java | 1110 ++++++++++++
.../hop/spark/core/HopSparkRowConverter.java | 225 +++
.../org/apache/hop/spark/core/HopSparkUtil.java | 81 +
.../spark/core/SparkExecutionDataAccumulator.java | 132 ++
.../java/org/apache/hop/spark/core/SparkHop.java | 32 +-
.../hop/spark/core/SparkInfoStreamSupport.java | 99 ++
.../apache/hop/spark/core/SparkNativeMetrics.java | 210 +++
.../org/apache/hop/spark/core/SparkRowHandler.java | 128 ++
.../hop/spark/core/SparkTransformMetricSlice.java | 232 +++
.../core/SparkTransformMetricsAccumulator.java | 106 ++
.../apache/hop/spark/core/SparkVariableValue.java | 30 +-
.../ISparkPipelineEngineRunConfiguration.java | 54 +
.../hop/spark/engines/SparkPipelineEngine.java | 1841 ++++++++++++++++++++
.../engines/SparkPipelineEngineCapabilities.java | 17 +-
.../engines/SparkPipelineRunConfiguration.java | 236 +++
.../engines/template/SparkRunConfigTemplate.java | 215 +++
.../execution/SparkTransformExecutionSampling.java | 466 +++++
.../DrawSparkRunModeOnTransformExtensionPoint.java | 100 ++
.../apache/hop/spark/gui/HopSparkGuiPlugin.java | 232 +++
.../hop/spark/gui/SparkPackageConfigDialog.java | 261 +++
.../apache/hop/spark/metadata/SparkCatalog.java | 192 ++
.../hop/spark/metadata/SparkCatalogEditor.java | 156 ++
.../metadata/template/SparkCatalogTemplate.java | 258 +++
.../pipeline/HopPipelineMetaToSparkConverter.java | 526 ++++++
.../pipeline/ISparkPipelineTransformHandler.java | 55 +
.../handler/SparkBaseTransformHandler.java | 88 +
.../pipeline/handler/SparkFileInputHandler.java | 380 ++++
.../spark/pipeline/handler/SparkFileIoSupport.java | 106 ++
.../pipeline/handler/SparkFileOutputHandler.java | 166 ++
.../handler/SparkGenericTransformHandler.java | 457 +++++
.../handler/SparkLakeTableInputHandler.java | 92 +
.../handler/SparkLakeTableMaintenanceHandler.java | 153 ++
.../handler/SparkLakeTableMergeHandler.java | 139 ++
.../handler/SparkLakeTableOutputHandler.java | 101 ++
.../handler/SparkMemoryGroupByHandler.java | 212 +++
.../pipeline/handler/SparkMergeJoinHandler.java | 202 +++
.../pipeline/handler/SparkSortRowsHandler.java | 116 ++
.../pipeline/handler/SparkUniqueRowsHandler.java | 177 ++
.../apache/hop/spark/pkg/PackageExportFilter.java | 334 ++++
.../apache/hop/spark/pkg/SparkProjectPackage.java | 1021 +++++++++++
.../java/org/apache/hop/spark/run/MainSpark.java | 333 ++++
.../org/apache/hop/spark/run/MainSparkArgs.java | 151 ++
.../apache/hop/spark/table/LakeSessionPlan.java | 427 +++++
.../hop/spark/table/SparkCatalogApplier.java | 189 ++
.../hop/spark/table/SparkLakeActionSupport.java | 49 +
.../hop/spark/table/SparkLakeConnectorProbe.java | 159 ++
.../apache/hop/spark/table/SparkLakeFormats.java | 62 +
.../hop/spark/table/SparkLakeTableSupport.java | 877 ++++++++++
.../spark/table/SparkMaintenanceSqlBuilder.java | 257 +++
.../hop/spark/table/SparkMergeSqlBuilder.java | 171 ++
.../apache/hop/spark/transforms/io/SparkField.java | 72 +
.../hop/spark/transforms/io/SparkFileInput.java | 48 +
.../spark/transforms/io/SparkFileInputData.java | 16 +-
.../spark/transforms/io/SparkFileInputDialog.java | 323 ++++
.../spark/transforms/io/SparkFileInputMeta.java | 127 ++
.../hop/spark/transforms/io/SparkFileOutput.java | 47 +
.../spark/transforms/io/SparkFileOutputData.java | 16 +-
.../spark/transforms/io/SparkFileOutputDialog.java | 272 +++
.../spark/transforms/io/SparkFileOutputMeta.java | 99 ++
.../transforms/table/SparkLakeTableInput.java | 47 +
.../transforms/table/SparkLakeTableInputData.java | 16 +-
.../table/SparkLakeTableInputDialog.java | 348 ++++
.../transforms/table/SparkLakeTableInputMeta.java | 138 ++
.../table/SparkLakeTableMaintenance.java | 48 +
.../table/SparkLakeTableMaintenanceData.java | 16 +-
.../table/SparkLakeTableMaintenanceDialog.java | 273 +++
.../table/SparkLakeTableMaintenanceMeta.java | 110 ++
.../transforms/table/SparkLakeTableMerge.java | 48 +
.../transforms/table/SparkLakeTableMergeData.java | 16 +-
.../table/SparkLakeTableMergeDialog.java | 279 +++
.../transforms/table/SparkLakeTableMergeMeta.java | 111 ++
.../transforms/table/SparkLakeTableOutput.java | 47 +
.../transforms/table/SparkLakeTableOutputData.java | 16 +-
.../table/SparkLakeTableOutputDialog.java | 259 +++
.../transforms/table/SparkLakeTableOutputMeta.java | 101 ++
.../java/org/apache/hop/spark/util/SparkConst.java | 54 +
.../apache/hop/spark/util/SparkPathDialect.java | 222 +++
.../org/apache/hop/spark/util/SparkRunMode.java | 182 ++
.../spark/src/main/resources/dependencies.xml | 38 +
.../engines/messages/messages_en_US.properties | 46 +
.../spark/gui/messages/messages_en_US.properties | 58 +
.../metadata/messages/messages_en_US.properties | 40 +
.../io/messages/messages_en_US.properties | 51 +
.../table/messages/messages_en_US.properties | 89 +
.../spark/src/main/resources/spark-catalog.svg | 5 +
.../spark/src/main/resources/spark-file-input.svg | 5 +
.../spark/src/main/resources/spark-file-output.svg | 4 +
.../src/main/resources/spark-lake-table-input.svg | 5 +
.../resources/spark-lake-table-maintenance.svg | 4 +
.../src/main/resources/spark-lake-table-merge.svg | 4 +
.../src/main/resources/spark-lake-table-output.svg | 4 +
.../src/main/resources/spark-run-distributed.svg | 8 +
.../spark/src/main/resources/spark-run-driver.svg | 6 +
.../spark/src/main/resources/version.xml} | 15 +-
.../engines/spark/src/samples/spark-demo/README.md | 167 ++
.../spark/src/samples/spark-demo/cluster-env.json | 14 +
.../src/samples/spark-demo/data/countries.csv | 4 +
.../samples/spark-demo/data/customers-sample.csv | 6 +
.../execution-data-profile/first-last.json | 16 +
.../execution-info-location/data-executions.json | 14 +
.../metadata/pipeline-run-configuration/local.json | 22 +
.../pipeline-run-configuration/spark-cluster.json | 23 +
.../metadata/workflow-run-configuration/local.json | 12 +
.../pipelines/01-enrich-with-mapping.hpl | 207 +++
.../spark-demo/pipelines/02-run-workflows.hpl | 165 ++
.../spark-demo/pipelines/03-run-pipelines.hpl | 175 ++
.../spark-demo/pipelines/mappings/upper-name.hpl | 128 ++
.../pipelines/nested/enrich-by-country.hpl | 116 ++
.../src/samples/spark-demo/project-config.json | 20 +
.../src/samples/spark-demo/scripts/prepare-dist.sh | 117 ++
.../spark-demo/scripts/spark-submit-demo.sh | 176 ++
.../scripts/spark-submit-pipelines-demo.sh | 29 +-
.../scripts/spark-submit-workflows-demo.sh | 29 +-
.../samples/spark-demo/spark-demo-gui-config.json | 14 +
.../spark-demo/workflows/create-country-folder.hwf | 141 +-
.../hop/spark/core/AcceptFilenamesBindingTest.java | 93 +
.../hop/spark/core/HopSparkRowConverterTest.java | 113 ++
.../hop/spark/core/SparkNativeMetricsTest.java | 152 ++
.../spark/core/SparkParallelFileContextTest.java | 86 +
.../core/SparkTransformMetricsAccumulatorTest.java | 86 +
.../SparkPipelineEngineExecutionInfoTest.java | 136 ++
.../engines/SparkPipelineEngineMetricsTest.java | 235 +++
.../engines/SparkPipelineRunConfigurationTest.java | 49 +
.../template/SparkRunConfigTemplateTest.java | 96 +
.../SparkTransformExecutionSamplingTest.java | 235 +++
.../template/SparkCatalogTemplateTest.java | 160 ++
.../HopPipelineMetaToSparkConverterTest.java | 183 ++
.../hop/spark/pipeline/MultiInputUnionTest.java | 237 +++
.../pipeline/handler/SparkFileIoHandlersTest.java | 358 ++++
.../pipeline/handler/SparkInfoStreamTest.java | 143 ++
.../pipeline/handler/SparkNativeHandlersTest.java | 354 ++++
.../pipeline/handler/SparkTargetStreamTest.java | 103 ++
.../hop/spark/pkg/PackageExportFilterTest.java | 125 ++
.../hop/spark/pkg/SparkProjectPackageTest.java | 379 ++++
.../apache/hop/spark/run/MainSparkArgsTest.java | 110 ++
.../hop/spark/run/MainSparkDatabricksEnvTest.java | 50 +
.../hop/spark/table/SparkCatalogApplierTest.java | 85 +
.../spark/table/SparkLakeConnectorProbeTest.java | 125 ++
.../spark/table/SparkLakeTableDeltaPathTest.java | 179 ++
.../spark/table/SparkLakeTableIcebergPathTest.java | 182 ++
.../spark/table/SparkLakeTableMaintenanceTest.java | 206 +++
.../hop/spark/table/SparkLakeTableMergeTest.java | 203 +++
.../hop/spark/table/SparkLakeTableSupportTest.java | 202 +++
.../spark/table/SparkLakeTableTableModeTest.java | 202 +++
.../spark/table/SparkLakeTableTimeTravelTest.java | 222 +++
.../spark/table/SparkLakehouseConnectorTest.java | 266 +++
.../table/SparkMaintenanceSqlBuilderTest.java | 171 ++
.../hop/spark/table/SparkMergeSqlBuilderTest.java | 107 ++
.../SparkLakeTableInputMetaInjectionTest.java | 61 +
...SparkLakeTableMaintenanceMetaInjectionTest.java | 51 +
.../SparkLakeTableMergeMetaInjectionTest.java | 50 +
.../SparkLakeTableOutputMetaInjectionTest.java | 49 +
.../hop/spark/util/SparkPathDialectTest.java | 130 ++
.../apache/hop/spark/util/SparkRunModeTest.java | 92 +
.../writer/Cql3SsTableWriterTest.java | 7 +
plugins/tech/databricks/pom.xml | 10 +
.../databricks/client/DatabricksJobsClient.java | 77 +
.../client/DatabricksRunLifeCycleState.java | 51 +
.../hop/databricks/client/DatabricksRunStatus.java | 97 ++
.../hop/databricks/client/DatabricksRunWaiter.java | 109 ++
.../client/RestDatabricksJobsClient.java | 697 ++++++++
.../databricks/client/WorkspaceFileMetadata.java | 19 +-
.../deploy/DatabricksJobSpecFactory.java | 603 +++++++
.../databricks/deploy/HopSparkDeployHelper.java | 665 +++++++
.../hop/databricks/deploy/MainSparkLaunchSpec.java | 71 +
.../databricks/metadata/DatabricksConnection.java | 91 +
.../metadata/DatabricksConnectionEditor.java | 169 ++
.../actions/databricks/ActionDatabricksJobRun.java | 569 ++++++
.../databricks/ActionDatabricksJobRunDialog.java | 871 +++++++++
.../databricks/ActionDatabricksJobWait.java | 271 +++
.../databricks/ActionDatabricksJobWaitDialog.java | 242 +++
.../src/main/resources/databricks-connection.svg | 23 +
.../src/main/resources/databricks-job-run.svg | 21 +
.../src/main/resources/databricks-job-wait.svg | 23 +
.../src/main/resources/databricks-logo.svg | 11 +
.../src/main/resources/databricks-run.svg | 15 +
.../src/main/resources/databricks-wait.svg | 34 +
.../metadata/messages/messages_en_US.properties | 33 +
.../databricks/messages/messages_en_US.properties | 115 ++
.../databricks/client/DatabricksRunWaiterTest.java | 107 ++
.../client/RestDatabricksJobsClientTest.java | 381 ++++
.../deploy/DatabricksJobSpecFactoryTest.java | 232 +++
.../deploy/HopSparkDeployHelperTest.java | 307 ++++
.../databricks/ActionDatabricksJobRunTest.java | 199 +++
.../databricks/ActionDatabricksJobWaitTest.java | 111 ++
.../googlesheets/GoogleSheetsCredentials.java | 12 +-
.../hop/vfs/gs/GoogleStorageFileProvider.java | 61 +-
.../pipeline/transforms/groupby/GroupByMeta.java | 4 +-
.../pipeline/transforms/jsonoutput/JsonOutput.java | 28 +-
.../transforms/jsonoutputenhanced/JsonEOutput.java | 32 +-
.../jsonoutput/messages/messages_en_US.properties | 4 +-
.../jsonoutput/messages/messages_ja_JP.properties | 4 +-
.../jsonoutput/messages/messages_zh_CN.properties | 2 +-
.../messages/messages_en_US.properties | 4 +-
.../pipeline/transforms/mapping/SimpleMapping.java | 55 +-
.../transforms/mapping/SimpleMappingTest.java | 22 +-
.../pipelineexecutor/PipelineExecutor.java | 24 +-
.../pipelineexecutor/PipelineExecutorMeta.java | 16 +
.../PipelineExecutorMetaCoverageTest.java | 16 +
.../pipelineexecutor/PipelineExecutorTest.java | 75 +
.../fileinput/text/TextFileInputDialog.java | 2 +-
.../workflowexecutor/WorkflowExecutor.java | 9 +-
.../workflowexecutor/WorkflowExecutorMeta.java | 13 +
pom.xml | 4 +-
tools/spark-client-pack/materialize-pack.sh | 6 +-
.../hop/ui/core/gui/GuiCompositeWidgets.java | 153 +-
.../gui/IGuiPluginCompositeButtonsListener.java | 15 +-
.../config/PipelineRunConfigurationEditor.java | 62 +-
669 files changed, 58247 insertions(+), 1608 deletions(-)
diff --git a/.gitignore b/.gitignore
index c4d839d039..f1160bc4c3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -57,6 +57,7 @@ integration-tests/spreadsheet/files/exel-multi-part_4.xlsx
integration-tests/json/output
integration-tests/mdi/files/excelwriter-mdi-test.xlsx
integration-tests/spreadsheet/files/sample-file-append.xlsx
+integration-tests/spark-native/output
assemblies/debug/audit
@@ -66,11 +67,8 @@ rap/src/main/resources/org/apache/hop/ui/hopgui/monaco/monaco-files.list
# Screenshots captured by SWTbot
screenshots/
-<<<<<<< Updated upstream
# Docker script leaving audit data
docker/local-data/
-=======
-/docker/local-data/
->>>>>>> Stashed changes
+/integration-tests/spreadsheet/files/exelwriter-testfile
diff --git a/NOTICE b/NOTICE
index dfff7c4211..3c525898df 100644
--- a/NOTICE
+++ b/NOTICE
@@ -9,3 +9,13 @@ Hitachi Vantara (https://www.hitachivantara.com//).
This product includes software developed by
JADAPTIVE (https://www.jadaptive.com/)
+
+This product bundles Delta Lake (io.delta) under the Apache License, Version 2.0.
+See: https://github.com/delta-io/delta
+
+This product bundles Apache Iceberg under the Apache License, Version 2.0.
+See: https://iceberg.apache.org/
+
+This product bundles Unity Catalog client libraries (transitive dependency of
+Delta Lake) under the Apache License, Version 2.0.
+See: https://github.com/unitycatalog/unitycatalog
diff --git a/assemblies/client/pom.xml b/assemblies/client/pom.xml
index 1adf6ccf45..c7e1dd1c21 100644
--- a/assemblies/client/pom.xml
+++ b/assemblies/client/pom.xml
@@ -67,6 +67,12 @@
<version>${project.version}</version>
<type>zip</type>
</dependency>
+ <dependency>
+ <groupId>org.apache.hop</groupId>
+ <artifactId>hop-engines-spark</artifactId>
+ <version>${project.version}</version>
+ <type>zip</type>
+ </dependency>
<dependency>
<groupId>org.apache.hop</groupId>
<artifactId>hop-ui</artifactId>
diff --git a/assemblies/debug/pom.xml b/assemblies/debug/pom.xml
index 8036f69593..f734be09fa 100644
--- a/assemblies/debug/pom.xml
+++ b/assemblies/debug/pom.xml
@@ -355,6 +355,12 @@
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hop</groupId>
+ <artifactId>hop-engines-spark</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
<dependency>
<groupId>org.apache.hop</groupId>
<artifactId>hop-misc-git</artifactId>
diff --git a/assemblies/plugins/pom.xml b/assemblies/plugins/pom.xml
index 6fc89979ce..30d0e6d0f5 100644
--- a/assemblies/plugins/pom.xml
+++ b/assemblies/plugins/pom.xml
@@ -618,6 +618,12 @@
<version>${project.version}</version>
<type>zip</type>
</dependency>
+ <dependency>
+ <groupId>org.apache.hop</groupId>
+ <artifactId>hop-engines-spark</artifactId>
+ <version>${project.version}</version>
+ <type>zip</type>
+ </dependency>
<dependency>
<groupId>org.apache.hop</groupId>
<artifactId>hop-misc-async</artifactId>
diff --git a/core/src/main/java/org/apache/hop/core/gui/plugin/GuiElementType.java b/core/src/main/java/org/apache/hop/core/gui/plugin/GuiElementType.java
index f7b5af4fbd..afe04a8c70 100644
--- a/core/src/main/java/org/apache/hop/core/gui/plugin/GuiElementType.java
+++ b/core/src/main/java/org/apache/hop/core/gui/plugin/GuiElementType.java
@@ -20,6 +20,11 @@ package org.apache.hop.core.gui.plugin;
public enum GuiElementType {
NONE, // To disable default options
TEXT,
+ /**
+ * Multi-line text widget ({@code SWT.MULTI}). Height in lines is set via {@link
+ * GuiWidgetElement#multiLineTextHeight()}.
+ */
+ MULTI_LINE_TEXT,
FILENAME, // Text widget with browse button
FOLDER, // Text widget with browse button
COMBO,
diff --git a/core/src/main/java/org/apache/hop/core/gui/plugin/GuiElements.java b/core/src/main/java/org/apache/hop/core/gui/plugin/GuiElements.java
index d49c67bf5a..0c773a3c6e 100644
--- a/core/src/main/java/org/apache/hop/core/gui/plugin/GuiElements.java
+++ b/core/src/main/java/org/apache/hop/core/gui/plugin/GuiElements.java
@@ -49,6 +49,9 @@ public class GuiElements extends BaseGuiElements implements Comparable<GuiElemen
private boolean password;
+ /** Height in text lines for MULTI_LINE_TEXT (minimum 1). */
+ private int multiLineTextHeight = 1;
+
private String fieldName;
private Class<?> fieldClass;
@@ -104,6 +107,7 @@ public class GuiElements extends BaseGuiElements implements Comparable<GuiElemen
this.disabledImage = null;
this.variablesEnabled = guiElement.variables();
this.password = guiElement.password();
+ this.multiLineTextHeight = Math.max(1, guiElement.multiLineTextHeight());
this.ignored = guiElement.ignored();
this.addingSeparator = guiElement.separator();
this.label = getTranslation(guiElement.label(), fieldPackageName, field.getDeclaringClass());
@@ -143,6 +147,7 @@ public class GuiElements extends BaseGuiElements implements Comparable<GuiElemen
this.disabledImage = null;
this.variablesEnabled = guiElement.variables();
this.password = guiElement.password();
+ this.multiLineTextHeight = Math.max(1, guiElement.multiLineTextHeight());
this.ignored = guiElement.ignored();
this.addingSeparator = guiElement.separator();
this.label = getTranslation(guiElement.label(), methodPackageName, method.getDeclaringClass());
@@ -370,6 +375,22 @@ public class GuiElements extends BaseGuiElements implements Comparable<GuiElemen
this.password = password;
}
+ /**
+ * Preferred height for {@link GuiElementType#MULTI_LINE_TEXT}, in text lines (at least 1).
+ *
+ * @return value of multiLineTextHeight
+ */
+ public int getMultiLineTextHeight() {
+ return multiLineTextHeight;
+ }
+
+ /**
+ * @param multiLineTextHeight The multiLineTextHeight to set (clamped to at least 1)
+ */
+ public void setMultiLineTextHeight(int multiLineTextHeight) {
+ this.multiLineTextHeight = Math.max(1, multiLineTextHeight);
+ }
+
/**
* Gets fieldName
*
diff --git a/core/src/main/java/org/apache/hop/core/gui/plugin/GuiWidgetElement.java b/core/src/main/java/org/apache/hop/core/gui/plugin/GuiWidgetElement.java
index d52cab091f..3d9adda3e9 100644
--- a/core/src/main/java/org/apache/hop/core/gui/plugin/GuiWidgetElement.java
+++ b/core/src/main/java/org/apache/hop/core/gui/plugin/GuiWidgetElement.java
@@ -80,6 +80,14 @@ public @interface GuiWidgetElement {
*/
boolean password() default false;
+ /**
+ * Preferred height of a {@link GuiElementType#MULTI_LINE_TEXT} widget, in text lines (not
+ * pixels). Default is 1. Values less than 1 are treated as 1. Ignored for other element types.
+ *
+ * @return height in lines
+ */
+ int multiLineTextHeight() default 1;
+
/**
* @return true if the widget supports variables
*/
diff --git a/dist/spark-native-demo/README.md b/dist/spark-native-demo/README.md
new file mode 100644
index 0000000000..0ec658ac4e
--- /dev/null
+++ b/dist/spark-native-demo/README.md
@@ -0,0 +1,33 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Staging location moved
+
+Native Spark demo artifacts default to **`/tmp/spark-demo-dist`** (not under the git tree).
+
+```bash
+./plugins/engines/spark/src/samples/spark-demo/scripts/prepare-dist.sh
+# → /tmp/spark-demo-dist/{hop-native-spark4-submit.jar,spark-demo.zip,hop-data/,…}
+
+docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
+ up -d --build --scale spark-worker=2
+# hop-data is bind-mounted to /data/hop-data in the cluster (outputs + executions on the host)
+```
+
+Override with `DIST_DIR` / `HOP_DIST_DIR` / `HOP_DATA_HOST_DIR` if you prefer other paths.
diff --git a/docker/integration-tests/integration-tests-base.yaml b/docker/integration-tests/integration-tests-base.yaml
index 2f1c9277c1..4bb32ece03 100644
--- a/docker/integration-tests/integration-tests-base.yaml
+++ b/docker/integration-tests/integration-tests-base.yaml
@@ -33,4 +33,6 @@ services:
- FLASK_ENV=docker
# Optional workflow basename filter (substring or glob). Set by run-tests-docker.sh.
- TEST_FILTER=${TEST_FILTER:-}
+ # Skip Google Sheets ITs when no real service-account JSON key is provided.
+ - SKIP_GOOGLE_SHEETS=${SKIP_GOOGLE_SHEETS:-false}
command: [ "bash", "-c", "/files/scripts/run-tests.sh ${PROJECT_NAME}" ]
diff --git a/docker/integration-tests/integration-tests-beam-base.yaml b/docker/integration-tests/integration-tests-beam-base.yaml
index 8521fd7ee8..f7c82ecdf0 100644
--- a/docker/integration-tests/integration-tests-beam-base.yaml
+++ b/docker/integration-tests/integration-tests-beam-base.yaml
@@ -35,4 +35,5 @@ services:
- HOP_SPARK_CLIENT_VERSION=${HOP_SPARK_CLIENT_VERSION:-}
# Optional workflow basename filter (substring or glob). Set by run-tests-docker.sh.
- TEST_FILTER=${TEST_FILTER:-}
+ - SKIP_GOOGLE_SHEETS=${SKIP_GOOGLE_SHEETS:-false}
command: [ "bash", "-c", "/files/scripts/run-tests.sh ${PROJECT_NAME}" ]
diff --git a/docker/integration-tests/integration-tests-base.yaml b/docker/integration-tests/integration-tests-iceberg.yaml
similarity index 58%
copy from docker/integration-tests/integration-tests-base.yaml
copy to docker/integration-tests/integration-tests-iceberg.yaml
index 2f1c9277c1..f7bd3a3e45 100644
--- a/docker/integration-tests/integration-tests-base.yaml
+++ b/docker/integration-tests/integration-tests-iceberg.yaml
@@ -15,22 +15,13 @@
# specific language governing permissions and limitations
# under the License.
+# Apache Iceberg table ITs on native SparkPipelineEngine (local[2]).
+# Requires engines-spark with lib/iceberg (default assembly).
services:
- integration_test:
- image: hop-base-image
- build:
- context: ../../.
- dockerfile: docker/integration-tests/unit-tests.Dockerfile
- args:
- - JENKINS_USER=jenkins
- - JENKINS_UID=1000
- - JENKINS_GROUP=jenkins
- - JENKINS_GID=1000
- - GCP_KEY_FILE=./docker/integration-tests/resource/dummyfile
- volumes:
- - ../../integration-tests/:/files
+ hop-iceberg-test:
+ extends:
+ file: integration-tests-beam-base.yaml
+ service: integration_test
environment:
- FLASK_ENV=docker
- # Optional workflow basename filter (substring or glob). Set by run-tests-docker.sh.
- - TEST_FILTER=${TEST_FILTER:-}
- command: [ "bash", "-c", "/files/scripts/run-tests.sh ${PROJECT_NAME}" ]
+ - HOP_OPTIONS=-Dspark.ui.enabled=false
diff --git a/docker/integration-tests/integration-tests-base.yaml b/docker/integration-tests/integration-tests-lakehouse.yaml
similarity index 58%
copy from docker/integration-tests/integration-tests-base.yaml
copy to docker/integration-tests/integration-tests-lakehouse.yaml
index 2f1c9277c1..17e9bb4451 100644
--- a/docker/integration-tests/integration-tests-base.yaml
+++ b/docker/integration-tests/integration-tests-lakehouse.yaml
@@ -15,22 +15,13 @@
# specific language governing permissions and limitations
# under the License.
+# Delta Lake table ITs on native SparkPipelineEngine (local[2]).
+# Requires engines-spark with lib/delta (default assembly).
services:
- integration_test:
- image: hop-base-image
- build:
- context: ../../.
- dockerfile: docker/integration-tests/unit-tests.Dockerfile
- args:
- - JENKINS_USER=jenkins
- - JENKINS_UID=1000
- - JENKINS_GROUP=jenkins
- - JENKINS_GID=1000
- - GCP_KEY_FILE=./docker/integration-tests/resource/dummyfile
- volumes:
- - ../../integration-tests/:/files
+ hop-lakehouse-test:
+ extends:
+ file: integration-tests-beam-base.yaml
+ service: integration_test
environment:
- FLASK_ENV=docker
- # Optional workflow basename filter (substring or glob). Set by run-tests-docker.sh.
- - TEST_FILTER=${TEST_FILTER:-}
- command: [ "bash", "-c", "/files/scripts/run-tests.sh ${PROJECT_NAME}" ]
+ - HOP_OPTIONS=-Dspark.ui.enabled=false
diff --git a/docker/integration-tests/integration-tests-spark-native-cluster.yaml b/docker/integration-tests/integration-tests-spark-native-cluster.yaml
new file mode 100644
index 0000000000..bb9750bfd8
--- /dev/null
+++ b/docker/integration-tests/integration-tests-spark-native-cluster.yaml
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Native Spark 4.1 standalone cluster for the spark-demo walk-through.
+#
+# Proves multi-node spark-submit with a Hop project package (Simple Mapping /
+# Workflow Executor under ${PROJECT_HOME}) and shared data on a host-visible
+# bind mount (HOP_DATA=file:///data/hop-data inside the containers).
+#
+# Host data plane (default): ${HOP_DIST_DIR:-/tmp/spark-demo-dist}/hop-data
+# → container /data/hop-data (CSV seeds, out/, packages/, executions/)
+#
+# From the repository root:
+#
+# # Prepare fat jar + package zip into /tmp/spark-demo-dist (see sample scripts)
+# ./plugins/engines/spark/src/samples/spark-demo/scripts/prepare-dist.sh
+#
+# docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
+# up -d --build --scale spark-worker=2
+#
+# # Mapping demo (default)
+# docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
+# exec spark /opt/hop-samples/spark-demo/scripts/spark-submit-demo.sh
+#
+# # Workflow Executor country folders demo
+# docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
+# exec spark /opt/hop-samples/spark-demo/scripts/spark-submit-workflows-demo.sh
+#
+# # Nested Native Spark pipelines (per country) demo
+# docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
+# exec spark /opt/hop-samples/spark-demo/scripts/spark-submit-pipelines-demo.sh
+#
+# # Inspect on the host (no docker exec required)
+# ls -la /tmp/spark-demo-dist/hop-data/out /tmp/spark-demo-dist/hop-data/executions
+#
+# Spark master UI: http://localhost:8080
+# Override SPARK_VERSION if needed (default matches plugins/engines/spark Spark 4.1.x pin).
+# Override data host path: HOP_DATA_HOST_DIR=/somewhere/else docker compose …
+
+services:
+ spark:
+ build:
+ context: ./spark
+ dockerfile: Dockerfile.master
+ args:
+ SPARK_VERSION: ${SPARK_VERSION:-4.1.2}
+ HADOOP_VERSION: ${HADOOP_VERSION:-3}
+ BASE_URL: ${SPARK_BASE_URL:-https://dlcdn.apache.org/spark}
+ hostname: spark
+ environment:
+ - INIT_DAEMON_STEP=setup_spark
+ - SPARK_MASTER_HOST=spark
+ ports:
+ - "8080:8080"
+ - "7077:7077"
+ volumes:
+ # Host-prepared fat jar + project package zip
+ - ${HOP_DIST_DIR:-/tmp/spark-demo-dist}:/opt/hop-dist:ro
+ # Sample project sources + scripts
+ - ../../plugins/engines/spark/src/samples:/opt/hop-samples:ro
+ # Shared data plane (master + workers): bind-mounted so the host can inspect
+ # outputs and execution-info files without docker exec.
+ # Default: /tmp/spark-demo-dist/hop-data (override HOP_DATA_HOST_DIR; if you change
+ # HOP_DIST_DIR, set HOP_DATA_HOST_DIR=${HOP_DIST_DIR}/hop-data as well)
+ - ${HOP_DATA_HOST_DIR:-/tmp/spark-demo-dist/hop-data}:/data/hop-data
+ healthcheck:
+ test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8080/"]
+ interval: 5s
+ timeout: 5s
+ retries: 24
+ start_period: 40s
+
+ spark-worker:
+ build:
+ context: ./spark
+ dockerfile: Dockerfile.worker
+ args:
+ SPARK_VERSION: ${SPARK_VERSION:-4.1.2}
+ HADOOP_VERSION: ${HADOOP_VERSION:-3}
+ BASE_URL: ${SPARK_BASE_URL:-https://dlcdn.apache.org/spark}
+ depends_on:
+ spark:
+ condition: service_healthy
+ environment:
+ - SPARK_MASTER=spark://spark:7077
+ - SPARK_WORKER_CORES=${SPARK_WORKER_CORES:-2}
+ - SPARK_WORKER_MEMORY=${SPARK_WORKER_MEMORY:-2g}
+ volumes:
+ - ${HOP_DATA_HOST_DIR:-/tmp/spark-demo-dist/hop-data}:/data/hop-data
+ ports:
+ - "8081"
diff --git a/docker/integration-tests/integration-tests-base.yaml b/docker/integration-tests/integration-tests-spark-native.yaml
similarity index 58%
copy from docker/integration-tests/integration-tests-base.yaml
copy to docker/integration-tests/integration-tests-spark-native.yaml
index 2f1c9277c1..8827318385 100644
--- a/docker/integration-tests/integration-tests-base.yaml
+++ b/docker/integration-tests/integration-tests-spark-native.yaml
@@ -15,22 +15,15 @@
# specific language governing permissions and limitations
# under the License.
+# Native SparkPipelineEngine smoke (local[2] only — no Spark cluster).
+# Extends hop-beam-image so /tmp/hop-fatjar.jar is present for future cluster tests
+# (spark-local run config leaves fatJar empty for Spark 4.1 plugin classpath).
services:
- integration_test:
- image: hop-base-image
- build:
- context: ../../.
- dockerfile: docker/integration-tests/unit-tests.Dockerfile
- args:
- - JENKINS_USER=jenkins
- - JENKINS_UID=1000
- - JENKINS_GROUP=jenkins
- - JENKINS_GID=1000
- - GCP_KEY_FILE=./docker/integration-tests/resource/dummyfile
- volumes:
- - ../../integration-tests/:/files
+ hop-spark-native-test:
+ extends:
+ file: integration-tests-beam-base.yaml
+ service: integration_test
environment:
- FLASK_ENV=docker
- # Optional workflow basename filter (substring or glob). Set by run-tests-docker.sh.
- - TEST_FILTER=${TEST_FILTER:-}
- command: [ "bash", "-c", "/files/scripts/run-tests.sh ${PROJECT_NAME}" ]
+ # Avoid Spark UI / jersey servlet clash on the driver
+ - HOP_OPTIONS=-Dspark.ui.enabled=false
diff --git a/docker/integration-tests/spark/scripts/master.sh b/docker/integration-tests/spark/scripts/master.sh
index 8d225dbb33..85a7791000 100644
--- a/docker/integration-tests/spark/scripts/master.sh
+++ b/docker/integration-tests/spark/scripts/master.sh
@@ -32,5 +32,6 @@ ln -sf /dev/stdout $SPARK_MASTER_LOG/spark-master.out
# # start history server and thrift server
# /spark/sbin/start-thriftserver.sh
+# Spark 3 accepted --ip; Spark 4 Master only accepts --host / -h (see Master --help).
cd /spark/bin && /spark/sbin/../bin/spark-class org.apache.spark.deploy.master.Master \
- --ip $SPARK_MASTER_HOST --port $SPARK_MASTER_PORT --webui-port $SPARK_MASTER_WEBUI_PORT >> $SPARK_MASTER_LOG/spark-master.out
\ No newline at end of file
+ --host $SPARK_MASTER_HOST --port $SPARK_MASTER_PORT --webui-port $SPARK_MASTER_WEBUI_PORT >> $SPARK_MASTER_LOG/spark-master.out
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-initializing.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-initializing.png
new file mode 100644
index 0000000000..4cdf9f6c65
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-initializing.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-memory-usage-4gb.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-memory-usage-4gb.png
new file mode 100644
index 0000000000..03bdb89426
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-memory-usage-4gb.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-running.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-running.png
new file mode 100644
index 0000000000..f1a2e1de0b
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-running.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-succeeded.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-succeeded.png
new file mode 100644
index 0000000000..113b4d07fd
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-jobs-pipelines-hello-databricks-succeeded.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-pipeline-hello-databricks.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-pipeline-hello-databricks.png
new file mode 100644
index 0000000000..cf56adaed8
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-pipeline-hello-databricks.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-workflow-deploy-to-databricks.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-workflow-deploy-to-databricks.png
new file mode 100644
index 0000000000..a6b57bb107
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-workflow-deploy-to-databricks.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-workspace-catalog-jars-volume.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-workspace-catalog-jars-volume.png
new file mode 100644
index 0000000000..272708a0aa
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/databricks-workspace-catalog-jars-volume.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-logs-to-execution-information-location.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-logs-to-execution-information-location.png
new file mode 100644
index 0000000000..3f561d0014
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-logs-to-execution-information-location.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-pipeline-engine-run-configurtion-editor.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-pipeline-engine-run-configurtion-editor.png
new file mode 100644
index 0000000000..c7363d9c62
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-pipeline-engine-run-configurtion-editor.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-provides-runtime-transform-metrics-and-state.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-provides-runtime-transform-metrics-and-state.png
new file mode 100644
index 0000000000..e24c7f6114
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-provides-runtime-transform-metrics-and-state.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-run-mode-force-driver.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-run-mode-force-driver.png
new file mode 100644
index 0000000000..8cbdb50862
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/native-spark-run-mode-force-driver.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0001-input-output.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0001-input-output.png
new file mode 100644
index 0000000000..a725ab534d
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0001-input-output.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0002-group-by.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0002-group-by.png
new file mode 100644
index 0000000000..0995d4b4cd
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0002-group-by.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0003-filter-rows.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0003-filter-rows.png
new file mode 100644
index 0000000000..6e2cea4c72
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0003-filter-rows.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0004-stream-lookup.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0004-stream-lookup.png
new file mode 100644
index 0000000000..83a2f6ffa6
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0004-stream-lookup.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0005-filter-targets.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0005-filter-targets.png
new file mode 100644
index 0000000000..56e11effa2
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0005-filter-targets.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0005-switch-case.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0005-switch-case.png
new file mode 100644
index 0000000000..970ad2fc1b
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0005-switch-case.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0008-excel-stream-lookup.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0008-excel-stream-lookup.png
new file mode 100644
index 0000000000..25aaf64772
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0008-excel-stream-lookup.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0008-text-file-input-spark-readback.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0008-text-file-input-spark-readback.png
new file mode 100644
index 0000000000..aaee3fe305
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0008-text-file-input-spark-readback.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0099-complex.png b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0099-complex.png
new file mode 100644
index 0000000000..4ece06c757
Binary files /dev/null and b/docs/hop-user-manual/modules/ROOT/assets/images/pipeline/spark/spark-native-0099-complex.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/database.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/database.svg
new file mode 100644
index 0000000000..0ea6675f26
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/database.svg
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" x="0px" y="0px"
+ width="16px" height="16px" viewBox="0 0 16 16" enable-background="new 0 0 16 16" xml:space="preserve">
+<path fill="#FE9C04" d="M1.448,9.916v2.823c0,1.56,2.95,2.823,6.588,2.823c3.638,0,6.588-1.264,6.588-2.823V9.916
+ c0,1.56-2.95,2.823-6.588,2.823C4.398,12.739,1.448,11.476,1.448,9.916z"/>
+<path fill="#FE9C04" d="M1.448,5.21v2.824c0,1.559,2.95,2.824,6.588,2.824c3.638,0,6.588-1.265,6.588-2.824V5.21
+ c0,1.56-2.95,2.824-6.588,2.824C4.398,8.034,1.448,6.77,1.448,5.21z"/>
+<path fill="#FE9C04" d="M14.625,3.328c0,1.56-2.95,2.823-6.588,2.823c-3.639,0-6.588-1.264-6.588-2.823
+ c0-1.559,2.95-2.823,6.588-2.823C11.675,0.504,14.625,1.769,14.625,3.328z"/>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-catalog.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-catalog.svg
new file mode 100644
index 0000000000..dd89035553
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-catalog.svg
@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24">
+ <path d="M 12,12.190476 8,8.1904762 V 11.190476 H 0 v 2 h 8 v 3 m 10,2 V 6.1904762 c 0,-1.11 -0.9,-2 -2,-2 H 4 a 2,2 0 0 0 -2,2 v 3 h 2 v -3 H 16 V 18.190476 H 4 v -3 H 2 v 3 a 2,2 0 0 0 2,2 h 12 a 2,2 0 0 0 2,-2 z"
+ style="fill:#0e3a5a;fill-opacity:1"/>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-input.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-input.svg
new file mode 100644
index 0000000000..dd89035553
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-input.svg
@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24">
+ <path d="M 12,12.190476 8,8.1904762 V 11.190476 H 0 v 2 h 8 v 3 m 10,2 V 6.1904762 c 0,-1.11 -0.9,-2 -2,-2 H 4 a 2,2 0 0 0 -2,2 v 3 h 2 v -3 H 16 V 18.190476 H 4 v -3 H 2 v 3 a 2,2 0 0 0 2,2 h 12 a 2,2 0 0 0 2,-2 z"
+ style="fill:#0e3a5a;fill-opacity:1"/>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-maintenance.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-maintenance.svg
new file mode 100644
index 0000000000..e3fb4293a6
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-maintenance.svg
@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24">
+ <path fill="#0F3B5A" d="M23,12L19,8V11H10V13H19V16M1,18V6C1,4.89 1.9,4 3,4H15A2,2 0 0,1 17,6V9H15V6H3V18H15V15H17V18A2,2 0 0,1 15,20H3A2,2 0 0,1 1,18Z"/>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-merge.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-merge.svg
new file mode 100644
index 0000000000..e3fb4293a6
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-merge.svg
@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24">
+ <path fill="#0F3B5A" d="M23,12L19,8V11H10V13H19V16M1,18V6C1,4.89 1.9,4 3,4H15A2,2 0 0,1 17,6V9H15V6H3V18H15V15H17V18A2,2 0 0,1 15,20H3A2,2 0 0,1 1,18Z"/>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-output.svg b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-output.svg
new file mode 100644
index 0000000000..e3fb4293a6
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/images/transforms/icons/spark-lake-table-output.svg
@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24">
+ <path fill="#0F3B5A" d="M23,12L19,8V11H10V13H19V16M1,18V6C1,4.89 1.9,4 3,4H15A2,2 0 0,1 17,6V9H15V6H3V18H15V15H17V18A2,2 0 0,1 15,20H3A2,2 0 0,1 1,18Z"/>
+</svg>
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc b/docs/hop-user-manual/modules/ROOT/nav.adoc
index 7afae79f25..63fca1915d 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -49,6 +49,11 @@ under the License.
**** xref:pipeline/beam/beam-samples-spark.adoc[Apache Spark]
**** xref:pipeline/beam/beam-samples-dataflow.adoc[Google Cloud Dataflow]
*** xref:pipeline/beam/flink-k8s-operator-running-hop-pipeline.adoc[Running a Hop pipeline using the Flink Kubernetes Operator]
+** xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark engine]
+*** xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems]
+*** xref:pipeline/spark/cluster-mapping-walkthrough.adoc[Cluster mapping walk-through]
+*** xref:pipeline/spark/lakehouse.adoc[Lakehouse tables (Delta / Iceberg)]
+*** xref:pipeline/spark/databricks.adoc[Native Spark on Databricks]
** xref:pipeline/pipeline-metrics.adoc[Pipeline Metrics]
** xref:pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc[Pipeline Run Configurations]
*** xref:pipeline/pipeline-run-configurations/beam-dataflow-pipeline-engine.adoc[Beam Google DataFlow]
@@ -57,6 +62,7 @@ under the License.
*** xref:pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.adoc[Beam Spark]
*** xref:pipeline/pipeline-run-configurations/native-local-pipeline-engine.adoc[Native Local]
*** xref:pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc[Native Remote]
+*** xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native Spark]
** xref:pipeline/pipeline-unit-testing.adoc[Pipeline Unit Tests]
** xref:pipeline/metadata-injection.adoc[Metadata Injection]
** xref:pipeline/specify-copies.adoc[Specify copies]
@@ -252,6 +258,10 @@ under the License.
*** xref:pipeline/transforms/sort.adoc[Sort Rows]
*** xref:pipeline/transforms/sortedmerge.adoc[Sorted Merge]
*** xref:pipeline/transforms/sortedschemamerge.adoc[Sorted Schema Merge]
+*** xref:pipeline/transforms/spark-lake-table-input.adoc[Spark Lake Table Input]
+*** xref:pipeline/transforms/spark-lake-table-maintenance.adoc[Spark Lake Table Maintenance]
+*** xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark Lake Table Merge]
+*** xref:pipeline/transforms/spark-lake-table-output.adoc[Spark Lake Table Output]
*** xref:pipeline/transforms/splitfields.adoc[Split Fields]
*** xref:pipeline/transforms/splitfieldtorows.adoc[Split fields to rows]
*** xref:pipeline/transforms/splunkinput.adoc[Splunk Input]
@@ -321,6 +331,8 @@ under the License.
*** xref:workflow/actions/copyfiles.adoc[Copy Files]
*** xref:workflow/actions/createfolder.adoc[Create a folder]
*** xref:workflow/actions/createfile.adoc[Create file]
+*** xref:workflow/actions/databricks-job-run.adoc[Databricks Job Run]
+*** xref:workflow/actions/databricks-job-wait.adoc[Databricks Job Wait]
*** xref:workflow/actions/pgpdecryptfiles.adoc[Decrypt files with PGP]
*** xref:workflow/actions/deletefile.adoc[Delete file]
*** xref:workflow/actions/deleteresultfilenames.adoc[Delete filenames from result]
@@ -462,6 +474,7 @@ under the License.
** xref:metadata-types/rest-connection.adoc[REST Connection]
** xref:metadata-types/salesforce-connection.adoc[Salesforce Connection]
** xref:metadata-types/splunk-connection.adoc[Splunk Connection]
+** xref:metadata-types/spark-catalog.adoc[Spark Catalog]
** xref:metadata-types/variable-resolver/index.adoc[Variable Resolver]
*** xref:metadata-types/variable-resolver/azure-key-vault-variable-resolver.adoc[Azure Key Vault variable resolver]
*** xref:metadata-types/variable-resolver/google-secret-manager-variable-resolver.adoc[Google Secret Manager variable resolver]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/databricks-connection.adoc b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/databricks-connection.adoc
new file mode 100644
index 0000000000..ba8aee6cce
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/databricks-connection.adoc
@@ -0,0 +1,62 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:imagesdir: ../../assets/images/
+:page-pagination:
+:description: Databricks Connection metadata for the Jobs / workspace REST API (PAT), used by Databricks Job Run actions.
+
+= Databricks Connection
+
+== Description
+
+*Databricks Connection* stores workspace credentials for the **Databricks Jobs / workspace REST API**.
+It is **not** the same as the xref:database/databases/databricks.adoc[Databricks JDBC] relational connection (SQL warehouse).
+
+Use this metadata type when orchestrating jobs from Hop (for example the upcoming **Databricks Job Run** workflow action) or when testing Jobs API access from the connection editor.
+
+== Options
+
+[options="header",cols="1,3"]
+|===
+|Option |Description
+
+|Name
+|Hop metadata name (how actions reference this connection).
+
+|Description
+|Optional free-text description.
+
+|Workspace host
+|Workspace URL host, e.g. `https://adb-1234567890123456.7.azuredatabricks.net`. The `https://` scheme is optional.
+
+|Personal access token
+|Databricks PAT (stored as a password field). Prefer Hop variables / secret backends for production. Tokens are never written to Hop logs by the Jobs client.
+
+|API base path
+|Jobs API prefix. Default: `/api/2.1`.
+|===
+
+== Test connection
+
+Use **Test connection** in the editor to call the workspace identity endpoint (SCIM *Me*) or fall back to listing jobs. A successful test shows the resolved user name or host.
+
+== Related
+
+* xref:pipeline/spark/databricks.adoc[Native Spark on Databricks] — Deploy & run, classic clusters, Volumes
+* xref:workflow/actions/databricks-job-run.adoc[Databricks Job Run] — trigger and wait for Jobs API runs
+* xref:database/databases/databricks.adoc[Databricks database (JDBC)] — SQL warehouse access
+* xref:pipeline/spark/getting-started-with-native-spark.adoc[Native Spark pipeline engine] — fat jar / MainSpark on cluster
+* xref:metadata-types/spark-catalog.adoc[Spark Catalog] — Unity Catalog templates for lakehouse TABLE mode
diff --git a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc
index ca4b5ecd1a..c4c716ac2d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc
@@ -41,6 +41,7 @@ By default, Hop contains the following metadata types:
* xref:metadata-types/cassandra/cassandra-connection.adoc[Cassandra Connection]: Describes a connection to a Cassandra cluster
* xref:metadata-types/data-set.adoc[Data Set]: This defines a data set, a static pre-defined collection of rows
xref:metadata-types/data-stream/data-stream.adoc[Data Stream]: A Data Stream can be used for Inter Process Communication (IPC) or fast data serialization.
+* xref:metadata-types/databricks-connection.adoc[Databricks Connection]: Workspace connection for the Databricks Jobs / REST API (PAT); not the JDBC SQL warehouse connection
* xref:metadata-types/execution-data-profile.adoc[Execution Data Profile]: Collects and profiles data as it flows through a pipeline using configurable samplers for insight into value ranges, nulls, and row samples.
* xref:metadata-types/execution-information-location.adoc[Execution Information Location]: Defines where and how Apache Hop stores execution metadata, supporting local files, remote servers, Neo4j, or Elastic for later inspection and analysis.
* xref:metadata-types/google-storage-authentication.adoc[Google Storage Connection]: A Google Cloud Storage connection type.
@@ -60,6 +61,7 @@ xref:metadata-types/data-stream/data-stream.adoc[Data Stream]: A Data Stream can
* xref:metadata-types/rest-connection.adoc[REST Connection]: Describes all the metadata needed to connect to a REST api.
* xref:metadata-types/salesforce-connection.adoc[Salesforce Connection]: Describes a reusable Salesforce connection that can be used across multiple Salesforce transforms
* xref:metadata-types/splunk-connection.adoc[Splunk Connection]: Describes a Splunk connection
+* xref:metadata-types/spark-catalog.adoc[Spark Catalog]: Spark SQL catalog configuration for Delta Lake / Apache Iceberg TABLE mode on the native Spark engine
* xref:metadata-types/static-schema-definition.adoc[Static Schema Definition]: Defines a reusable data stream layout to ensure consistency across multiple pipelines and simplify schema management.
* xref:metadata-types/variable-resolver/index.adoc[Variable Resolver]: Use plugins to resolve variable values with a pipeline, a key store, a vaults, or secret managers.
* xref:hop-server/web-service.adoc[Web Service]: Allows to run a pipeline to generate output for a servlet on Hop Server
diff --git a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/spark-catalog.adoc b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/spark-catalog.adoc
new file mode 100644
index 0000000000..79fb3ab916
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/spark-catalog.adoc
@@ -0,0 +1,150 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:imagesdir: ../../assets/images/
+:page-pagination:
+:description: Spark Catalog metadata configures Spark SQL catalogs for Delta Lake and Apache Iceberg TABLE mode on the native Spark pipeline engine.
+
+= Spark Catalog
+
+== Description
+
+*Spark Catalog* is a reusable Hop metadata type (category **Connections**) that describes a Spark SQL catalog for open table formats on the xref:pipeline/spark/getting-started-with-native-spark.adoc[native Spark pipeline engine].
+
+Lake Table transforms in **TABLE** mode can reference a Spark Catalog by name. Hop expands the entry into `spark.sql.catalog.<catalogName>.*` configuration when building or preparing the session (`LakeSessionPlan` / `SparkCatalogApplier`).
+
+PATH-mode Iceberg tables do **not** require this metadata type — the engine auto-registers a built-in Hadoop catalog named `hop_iceberg`. Use Spark Catalog for production Iceberg warehouses, REST catalogs, and other named catalogs.
+
+Connector packaging and session matrix: xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine].
+
+== Load catalog template
+
+Use **Load catalog template** at the top of the form to fill fields from a named scenario. Placeholders (bucket, host, URI) must still be adjusted for your environment. Secrets are never stored in templates — put tokens in **Credential**.
+
+Every template writes a commented documentation link as the first line of **Extra conf**, for example:
+
+----
+# docs: https://iceberg.apache.org/docs/latest/spark-configuration/
+----
+
+`#` lines are ignored when Hop builds the Spark session, so the link is guidance only.
+
+[options="header",cols="1,2,3"]
+|===
+|Template |Type |What it sets
+
+|Iceberg Hadoop (local)
+|`hadoop`
+|Catalog `lake`, warehouse `file:///tmp/hop-warehouse`
+
+|Iceberg Hadoop (object store)
+|`hadoop`
+|Catalog `lake`, warehouse `s3a://bucket/warehouse`, S3 `io-impl` in conf extra
+
+|Iceberg REST
+|`rest`
+|Catalog `lake`, URI `https://catalog.example.com/v1`
+
+|Iceberg REST (authenticated)
+|`rest`
+|Same as REST; conf extra notes that **Credential** maps to catalog `.token`
+
+|Hive Metastore (advanced)
+|`hive`
+|Catalog `hive`, thrift URI placeholder (extra deps required)
+
+|AWS Glue (advanced)
+|`glue`
+|Catalog `glue`, S3 warehouse + IO placeholders (extra deps required)
+
+|Nessie (advanced)
+|`custom`
+|Catalog `nessie`, Nessie `catalog-impl` / URI / `ref` placeholders
+
+|Databricks Unity Catalog (advanced)
+|`custom`
+|Catalog `unity`, commented workspace URI/warehouse examples
+
+|Delta named catalog (advanced)
+|`custom`
+|`DeltaCatalog` implementation; prefer `spark_catalog` when coexisting with Iceberg
+|===
+
+The Hop metadata **Name** (how transforms select this entry) is not changed by a template — only catalog fields are.
+
+== Options
+
+[options="header",cols="1,3"]
+|===
+|Option |Description
+
+|Name
+|Hop metadata name (how transforms reference this entry).
+
+|Load catalog template
+|Button that applies a named scenario (see above). Confirms before replacing non-empty fields.
+
+|Catalog name
+|Spark SQL catalog name — first segment of `catalog.namespace.table` (e.g. `lake`).
+
+|Catalog type
+|`hadoop` — Iceberg Hadoop warehouse (default). +
+`rest` — Iceberg REST catalog. +
+`custom` — supply implementation + conf extra. +
+`hive` / `glue` — advanced; conf-extra oriented, not packaged by default.
+
+|Implementation
+|Spark catalog class. Default for Iceberg is `org.apache.iceberg.spark.SparkCatalog`.
+
+|Warehouse
+|Warehouse path for Hadoop (and similar) catalogs, e.g. `file:///data/warehouse` or an object-store URI.
+
+|URI
+|Catalog service URI (REST and similar).
+
+|Credential
+|Optional secret (password field). For REST catalogs, applied as `spark.sql.catalog.<name>.token` when set.
+
+|Conf extra
+|Additional `key=value` lines (one per line; `#` comments and blank lines ignored). +
+Short keys (e.g. `io-impl`) become `spark.sql.catalog.<catalogName>.io-impl`. +
+Keys that already start with `spark.` are applied as full Spark conf keys.
+|===
+
+== How it is applied
+
+For a catalog named `lake` of type `hadoop` with warehouse `file:///tmp/wh`, Hop effectively sets:
+
+----
+spark.sql.catalog.lake=org.apache.iceberg.spark.SparkCatalog
+spark.sql.catalog.lake.type=hadoop
+spark.sql.catalog.lake.warehouse=file:///tmp/wh
+----
+
+Lake Table transforms then use identifiers such as `lake.db.orders`.
+
+When both Delta and Iceberg are used in one session, keep **Delta** on `spark_catalog` (`DeltaCatalog`) and put Iceberg under a **different** catalog name (e.g. `lake` or `hop_iceberg`).
+
+== spark-submit
+
+Export project metadata JSON that includes every Spark Catalog entry referenced by the pipeline (and any used only via run configuration). Alternatively, pass equivalent `--conf spark.sql.catalog.…` flags on the submit command. Paths and warehouses must be reachable from the driver and executors.
+
+== Related pages
+
+* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine]
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark Lake Table Input]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark Lake Table Output]
+* xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark pipeline engine]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc
new file mode 100644
index 0000000000..fa00b1929a
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc
@@ -0,0 +1,218 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[NativeSparkPipelineEngine]]
+:imagesdir: ../../../assets/images
+:description: Apache Hop native Spark pipeline engine run configuration options (Spark 4.1.x without Beam).
+
+= Native Spark Pipeline Engine
+
+== Overview
+
+The native Spark pipeline engine executes batch Hop pipelines on *Apache Spark 4.1.x* without Apache Beam.
+It builds a Spark SQL Dataset graph from the pipeline and runs transforms via native Dataset operators or generic `mapPartitions` mini-pipelines.
+
+For a full introduction — concepts, `local[*]`, fat jar, `spark-submit`, Spark File I/O, `${Internal.Transform.ID}`, metrics, limitations, and integration-test examples — see
+xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark pipeline engine].
+
+Paths on *Spark File* and *Lake Table* transforms are **Spark/Hadoop URIs**, not Hop VFS schemes — see
+xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems on native Spark].
+
+This page is *not* the same as xref:pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.adoc[Beam Spark], which runs Hop over the Apache Beam Spark runner (Spark 3.5.x line).
+
+image::pipeline/spark/native-spark-pipeline-engine-run-configurtion-editor.png[Native Spark pipeline run configuration editor,width=70%]
+
+=== Requirements
+
+* Java 21
+* Spark 4.1.x (bundled under `plugins/engines/spark/lib` for `local[*]`)
+
+=== Options
+
+==== Load configuration template
+
+Use **Load configuration template** to fill the fields below from a common deployment style:
+
+* *Local execution* — `local[*]`, Spark UI off (embedded Spark in Hop)
+* *Local (capped cores)* — `local[4]`
+* *Spark standalone server* — `spark://host:7077` (replace host), sample driver/executor memory
+* *spark-submit (cluster)* — master **empty** so `spark-submit --master` wins; use a `native-provided` fat jar
+* *Databricks* — submit-oriented placeholders (master empty); set workspace-specific values yourself
+* *YARN client* — master `yarn`, client deploy mode
+* *Lakehouse local* — same as local; Delta/Iceberg connectors live on the plugin classpath
+
+Templates replace the current engine options after confirmation when the form already has custom values. The form fields update immediately after you apply a template. Edit values afterward as needed.
+
+For named Iceberg/Delta catalogs used by Lake Table transforms in **TABLE** mode, see xref:metadata-types/spark-catalog.adoc[Spark Catalog] (**Load catalog template**).
+
+[options="header",cols="1,3,1"]
+|===
+|Option |Description |Default
+
+|Spark master
+|Spark master URL: `local[*]`, `local[4]`, or `spark://host:7077`. Leave empty under `spark-submit` so the CLI master is used.
+|`local[*]`
+
+|Application name
+|Spark application name.
+|`Apache Hop`
+
+|Fat jar file location
+|Optional Hop fat jar staged for executors when Hop starts the Spark session. Leave empty for pure `spark-submit` with a native-provided fat jar.
+|
+
+|Spark config (key=value lines)
+|Additional SparkConf entries, one `key=value` per line.
+|
+
+|Driver memory
+|`spark.driver.memory` (e.g. `2g`).
+|
+
+|Executor memory
+|`spark.executor.memory`.
+|
+
+|Executor cores
+|`spark.executor.cores`.
+|
+
+|Temp location
+|Directory for temporary files.
+|JVM temp dir
+
+|Plugins to stage
+|Optional comma-separated plugin folders to stage on the cluster.
+|
+
+|Path scheme map (`from=to` lines)
+|Optional URI scheme rewrite for *Spark File* / *Lake PATH* only (e.g. `s3=s3a`, `minio=s3a`). Does not apply to classic Hop VFS transforms. Configure target FS (e.g. `spark.hadoop.fs.s3a.*`) separately. See xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems].
+|
+
+|Generic transform run mode
+|Default for Hop transforms that run as `mapPartitions` mini-pipelines (Workflow Executor, most classic transforms). `DISTRIBUTED` fans work out across Spark partitions/nodes. `DRIVER_ONLY` materializes the transform input on the Spark driver and runs the mini-pipeline once there, so nested workflows do not execute on multiple nodes. Override per transform with the *Spark Run Mode* context action on the canvas.
+|`DISTRIBUTED`
+|===
+
+Also configure *Execution information location* and *Execution data profile* on the run configuration main tab when you want GUI metrics history and row samples.
+
+==== Generic transform run mode and per-transform override
+
+Most classic Hop transforms (including *Workflow Executor*, *Pipeline Executor*, and mappings) execute on Native Spark as generic `mapPartitions` work. When the input Dataset has many partitions, Spark can run that nested work on multiple executors at once. That is often desirable for pure row transforms, but nested workflows or other non-reentrant side effects may need a single place of execution.
+
+* **Run configuration default** — *Generic transform run mode*: `DISTRIBUTED` (default) or `DRIVER_ONLY`.
+* **Per-transform override** — right-click a transform → *Spark Run Mode*:
+** *Inherit* — use the run configuration default
+** *Force distributed* — always use `mapPartitions` across partitions
+** *Force Driver Only* — always run this transform's mini-pipeline on the Spark driver
++
+The override is stored on the transform (`attributes` group `spark` / key `run_mode`) and is saved with the pipeline. When a force override is set, a small badge is drawn at the **bottom-left** of the transform icon.
+
+image::pipeline/spark/native-spark-run-mode-force-driver.png[Transform with Force Driver Only Spark run-mode badge,width=70%]
+
+Native Dataset handlers (Spark File / Lake Table, Memory Group By, Sort, Unique, Merge Join) ignore this setting.
+
+[NOTE]
+====
+`DRIVER_ONLY` collects that transform's input rows on the driver. Prefer `DISTRIBUTED` for high-volume data; use driver-only for nested execution control (for example a Workflow Executor over a modest set of control rows).
+====
+
+==== Nested Native Spark pipelines (orchestrator pattern)
+
+A parent Native Spark pipeline can start **child** Native Spark pipelines via **Pipeline Executor** (run configuration = Native Spark):
+
+* Put **Force Driver Only** (or run-config `DRIVER_ONLY`) on the Pipeline Executor so children start on the driver.
+* Each child **reuses the parent `SparkSession`** (same Spark application) — it does **not** start a second `spark-submit`.
+* Parameterize the child with control keys (country, month, partition) for paths and filters.
+* Nested engines **must not stop** the shared session when they finish.
+
+Do **not** run nested Native Spark executors under **DISTRIBUTED** mapPartitions on workers (no safe nested SparkContext). For multi-application isolation (separate Spark apps per key), use an external job launcher instead.
+
+Demo: sample project `spark-demo` pipeline `pipelines/03-run-pipelines.hpl` (see xref:pipeline/spark/cluster-mapping-walkthrough.adoc[Cluster demo walk-through]).
+
+[[transforms-not-supported-on-native-spark]]
+=== Transforms not supported on Native Spark
+
+Most classic Hop transforms run on Native Spark via native Dataset handlers or generic `mapPartitions`.
+The list below is taken from each transform’s *Supported Engines* table (**Native Spark** = Not Supported).
+When in doubt, open the transform page — that table is the source of truth.
+
+For broader design limits (copy/distribute, streaming, Spark SQL, MLlib), see
+xref:pipeline/spark/getting-started-with-native-spark.adoc#limitations-and-design-rules[Getting started — Limitations and design rules].
+
+==== Beam-only transforms
+
+Use a xref:pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.adoc[Beam] engine instead.
+
+* xref:pipeline/transforms/beambigqueryinput.adoc[BigQuery Input]
+* xref:pipeline/transforms/beambigqueryoutput.adoc[BigQuery Output]
+* xref:pipeline/transforms/beambigtableinput.adoc[Beam Bigtable Input]
+* xref:pipeline/transforms/beambigtableoutput.adoc[Beam Bigtable Output]
+* xref:pipeline/transforms/beamfileinput.adoc[Beam File Input]
+* xref:pipeline/transforms/beamfileoutput.adoc[Beam File Output]
+* xref:pipeline/transforms/beamgcppublisher.adoc[Beam GCP Pub/Sub : Publish]
+* xref:pipeline/transforms/beamgcpsubscriber.adoc[Beam GCP Pub/Sub : Subscribe]
+* xref:pipeline/transforms/beamhivecataloginput.adoc[Beam Hive Catalog Input]
+* xref:pipeline/transforms/beamkafkaconsume.adoc[Beam Kafka Consume]
+* xref:pipeline/transforms/beamkafkaproduce.adoc[Beam Kafka Produce]
+* xref:pipeline/transforms/beamkinesisconsume.adoc[Beam Kinesis Consume]
+* xref:pipeline/transforms/beamkinesisproduce.adoc[Beam Kinesis Produce]
+* xref:pipeline/transforms/beamtimestamp.adoc[Beam Timestamp]
+* xref:pipeline/transforms/beamwindow.adoc[Beam Window]
+
+==== Barriers, stream ordering, and merge-style transforms
+
+These rely on waiting for other transforms, full-stream ordering, or multi-stream merge semantics that do not map cleanly to a Spark Dataset graph.
+
+* xref:pipeline/transforms/analyticquery.adoc[Analytic Query]
+* xref:pipeline/transforms/append.adoc[Append Streams]
+* xref:pipeline/transforms/blockingtransform.adoc[Blocking transform]
+* xref:pipeline/transforms/blockuntiltransformsfinish.adoc[Blocking until transforms finish]
+* xref:pipeline/transforms/detectemptystream.adoc[Detect Empty Stream]
+* xref:pipeline/transforms/identifylastrow.adoc[Identify last row in a stream]
+* xref:pipeline/transforms/mergerows.adoc[Merge rows (diff)]
+* xref:pipeline/transforms/multimerge.adoc[Multiway Merge Join]
+* xref:pipeline/transforms/sortedmerge.adoc[Sorted Merge]
+* xref:pipeline/transforms/sortedschemamerge.adoc[Sorted Schema Merge]
+* xref:pipeline/transforms/streamschemamerge.adoc[Stream Schema Merge]
+
+==== Sorted aggregation and hash-set unique
+
+* xref:pipeline/transforms/groupby.adoc[Group By] — use xref:pipeline/transforms/memgroupby.adoc[Memory Group By] (native Spark `groupBy`) instead
+* xref:pipeline/transforms/uniquerowsbyhashset.adoc[Unique Rows (HashSet)] — use xref:pipeline/transforms/uniquerows.adoc[Unique Rows] where possible
+
+==== Workflow / pipeline result and control transforms
+
+Result-row and result-file hops, injectors, and metadata injection are not supported as Native Spark graph nodes.
+
+* xref:pipeline/transforms/copyrowstoresult.adoc[Copy rows to result]
+* xref:pipeline/transforms/filesfromresult.adoc[Files from result]
+* xref:pipeline/transforms/filestoresult.adoc[Files to result]
+* xref:pipeline/transforms/getrowsfromresult.adoc[Get Rows from Result]
+* xref:pipeline/transforms/injector.adoc[Injector]
+* xref:pipeline/transforms/metainject.adoc[Metadata Injection]
+* xref:pipeline/transforms/pipeline-data-probe.adoc[Pipeline Data Probe]
+* xref:pipeline/transforms/repeatfields.adoc[Repeat Fields]
+
+==== Other
+
+* xref:pipeline/transforms/kafkaconsumer.adoc[Kafka Consumer] — no native Spark structured-streaming / Kafka Dataset I/O; use Beam Kafka transforms on a Beam engine for Kafka pipelines
+
+=== Related
+
+* xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark pipeline engine]
+* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine] (Delta Lake / Apache Iceberg)
+* xref:pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.adoc[Beam Spark pipeline engine]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc
index a92a49d52f..660797a34e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc
@@ -58,6 +58,7 @@ The available engine types are:
* *xref:pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.adoc[Beam Spark pipeline engine]*: this configuration runs pipelines on Apache Spark over Apache Beam
* *xref:pipeline/pipeline-run-configurations/native-local-pipeline-engine.adoc[Hop local pipeline engine]*: this configuration runs pipelines locally in the native Hop engine
* *xref:pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc[Hop remote pipeline engine]*: this configuration runs pipelines in the native Hop engine on a remote machine
+* *xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native Spark pipeline engine]*: this configuration runs batch pipelines on Apache Spark 4.x *without* Beam (see xref:pipeline/spark/getting-started-with-native-spark.adoc[getting started])
* *xref:pipeline/pipeline-run-configurations/single-threaded-pipeline-engine.adoc[Hop local single threaded pipeline engine]*: this configuration runs pipelines locally in a single-threaded fashion
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipelines.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipelines.adoc
index 0ce83d4c85..3f692b967d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipelines.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipelines.adoc
@@ -56,4 +56,6 @@ Pipelines are an extensive topic. Check the pages below to learn more about work
* xref:pipeline/metadata-injection.adoc[Metadata Injection]
* xref:pipeline/partitioning.adoc[Partitioning]
* xref:pipeline/beam/getting-started-with-beam.adoc[Getting started with Apache Beam]
+* xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark pipeline engine]
+* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine (Delta / Iceberg)]
* xref:pipeline/transforms.adoc[Transforms]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/cluster-mapping-walkthrough.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/cluster-mapping-walkthrough.adoc
new file mode 100644
index 0000000000..7121823357
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/cluster-mapping-walkthrough.adoc
@@ -0,0 +1,328 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[NativeSparkClusterMappingWalkthrough]]
+:imagesdir: ../../../assets/images
+:description: Walk-through: run Native Spark pipelines (Simple Mapping and Workflow Executor) on a Docker Spark 4.1 cluster using a project package.
+
+= Cluster demo walk-through (Native Spark)
+
+This walk-through shows that the Native Spark **project package** design works on a *real multi-node* Spark 4.1 cluster — not only on `local[*]`.
+
+You will:
+
+. Build a **`native-provided`** fat jar (cluster provides Spark).
+. Export a **Native Spark project package** (definitions + metadata).
+. Start a Docker Compose cluster (master + workers, shared data volume).
+. `docker exec` **spark-submit** with `MainSpark --HopProjectPackage=…`.
+. See a **Simple Mapping** child under `${PROJECT_HOME}` execute on workers via **SparkFiles**.
+. Optionally run a **Workflow Executor** demo that creates per-country folders and instance marker files on the shared volume.
+. Optionally run a **nested Native Spark** demo: Pipeline Executor on the driver starts a full child Spark pipeline per country (shared `SparkSession`).
+
+Sample project (in source tree):
+
+`plugins/engines/spark/src/samples/spark-demo`
+
+Compose file:
+
+`docker/integration-tests/integration-tests-spark-native-cluster.yaml`
+
+== What this proves
+
+[options="header",cols="1,2"]
+|===
+|Claim |How the demo shows it
+
+|Nested mappings need files on every executor
+|Child path is `${PROJECT_HOME}/pipelines/mappings/upper-name.hpl` — not embedded in the parent
+
+|Nested workflows need the same package
+|Workflow Executor child is `${PROJECT_HOME}/workflows/create-country-folder.hwf`
+
+|Project package is the right unit
+|Package zip is exported specifically for Native Spark (GUI or `hop-conf --export-spark-project`)
+
+|Multi-host shipping works
+|Engine calls `SparkContext.addFile`; workers resolve with `SparkFiles.get` then extract
+
+|Data ≠ definitions
+|Input/output and workflow side effects use **`HOP_DATA=file:///data/hop-data`** (shared volume), not package `PROJECT_HOME`
+|===
+
+Related design docs: xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems], xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark engine].
+
+== Prerequisites
+
+* Docker with Compose v2
+* Hop client **2.19+** with the native Spark plugin (from a built assembly or install)
+* Enough disk to download Spark 4.1.x images (~1 GB first build)
+* From a git checkout of Hop (paths below assume **repository root**)
+
+== 1. Prepare the fat jar and package on the host
+
+[source,bash]
+----
+# Optional: point at your Hop install
+export HOP_HOME=/path/to/hop # directory that contains hop-conf.sh
+
+./plugins/engines/spark/src/samples/spark-demo/scripts/prepare-dist.sh
+----
+
+By default this writes into **`/tmp/spark-demo-dist/`** (outside the git tree; override with `DIST_DIR`):
+
+* `hop-native-spark4-submit.jar` — fat jar with token **`native-provided`** (no embedded Spark)
+* `spark-demo.zip` — Native Spark project package
+* `cluster-env.json` — sets `HOP_DATA` for the cluster
+* `data/customers-sample.csv` — mapping demo CSV
+* `data/countries.csv` — workflow demo CSV (Genovia, Wakanda, Latveria)
+
+You do **not** need to register `spark-demo` in the system Hop configuration.
+`prepare-dist.sh` exports with:
+
+[source,bash]
+----
+hop-conf.sh --export-spark-project=/tmp/spark-demo-dist/spark-demo.zip \
+ --export-spark-project-home=…/plugins/engines/spark/src/samples/spark-demo
+----
+
+[NOTE]
+====
+Hop has two different project flags:
+
+* **`-j <name>`** — *enable* a registered project for this command (sets `PROJECT_HOME`)
+* **`-p` / `--project <name>`** — only *names* a project for create/delete/list management
+
+`export-spark-project` needs a home folder via `--export-spark-project-home=…` or a project already enabled with **`-j`**. Using only `--project=` does not set `PROJECT_HOME`.
+====
+
+Alternatively in the GUI:
+
+* *Tools* → *Generate a Hop fat jar* with native-provided packaging, and
+* *Tools* → *Export project package for Native Spark…* after opening the sample project.
+
+== 2. Start the Spark 4.1 cluster
+
+[source,bash]
+----
+docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
+ up -d --build --scale spark-worker=2
+----
+
+Compose mounts:
+
+* `${HOP_DIST_DIR:-/tmp/spark-demo-dist}` → `/opt/hop-dist` (fat jar + project package; master)
+* `${HOP_DATA_HOST_DIR:-/tmp/spark-demo-dist/hop-data}` → `/data/hop-data` (shared data plane; **master and workers**, host-visible)
+
+If you used a non-default `DIST_DIR` in `prepare-dist.sh`, set both `HOP_DIST_DIR` and `HOP_DATA_HOST_DIR=${HOP_DIST_DIR}/hop-data` for compose.
+
+* Master UI: http://localhost:8080 (two workers should register).
+* Images use the existing `docker/integration-tests/spark` Dockerfiles with **`SPARK_VERSION=4.1.2`** (override with env if needed).
+* After a run you can inspect **on the host** without `docker exec`, for example:
+
+[source,bash]
+----
+ls -la /tmp/spark-demo-dist/hop-data/out
+ls -la /tmp/spark-demo-dist/hop-data/executions # if an execution information location is configured
+----
+
+== 3. Submit the demo job
+
+[source,bash]
+----
+docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
+ exec spark /opt/hop-samples/spark-demo/scripts/spark-submit-demo.sh
+----
+
+What the script does:
+
+. Seeds `/data/hop-data/customers-sample.csv` and `/data/hop-data/countries.csv` on the shared volume.
+. Runs `spark-submit --master spark://spark:7077 --deploy-mode client --class org.apache.hop.spark.run.MainSpark …`
+. Passes `--HopProjectPackage=/opt/hop-dist/spark-demo.zip` and
+ `--HopPipelinePath=pipelines/01-enrich-with-mapping.hpl` (default).
+. Loads `cluster-env.json` so `HOP_DATA=file:///data/hop-data`.
+
+In the logs you should see package materialization / distribution and a normal pipeline finish.
+
+== 4. Verify mapping output
+
+On the host (bind mount):
+
+[source,bash]
+----
+ls -la /tmp/spark-demo-dist/hop-data/out/enriched
+head -n 20 /tmp/spark-demo-dist/hop-data/out/enriched/*
+----
+
+Or inside the container (`/data/hop-data` is the same directory):
+
+[source,bash]
+----
+docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
+ exec spark ls -la /data/hop-data/out/enriched
+
+docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
+ exec spark sh -c 'head -n 20 /data/hop-data/out/enriched/*'
+----
+
+Expect a header plus rows with an uppercase `displayName` column (from the mapping child).
+
+== 5. Workflow Executor demo (optional)
+
+Submit the second sample pipeline (same package and cluster):
+
+[source,bash]
+----
+docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
+ exec spark /opt/hop-samples/spark-demo/scripts/spark-submit-workflows-demo.sh
+----
+
+Inspect country folders and marker files (host path):
+
+[source,bash]
+----
+find /tmp/spark-demo-dist/hop-data/out/countries -type f | sort
+----
+
+Expect:
+
+[source]
+----
+/tmp/spark-demo-dist/hop-data/out/countries/Genovia/<Internal.Pipeline.ID>.txt
+/tmp/spark-demo-dist/hop-data/out/countries/Wakanda/<Internal.Pipeline.ID>.txt
+/tmp/spark-demo-dist/hop-data/out/countries/Latveria/<Internal.Pipeline.ID>.txt
+----
+
+* Three country directories mean Workflow Executor ran the child workflow **once per input row**.
+* Marker basenames come from `${Internal.Pipeline.ID}` (parent context via `inherit_all_vars`).
+** Same basename in every folder → one parent transform/pipeline instance handled all rows (typical single partition or **DRIVER_ONLY**).
+** Different basenames → multiple Spark partition instances of Workflow Executor (**DISTRIBUTED** fan-out).
+
+The Native Spark run configuration **Generic transform run mode** (`DISTRIBUTED` / `DRIVER_ONLY`) and the transform context action *Spark Run Mode* control whether Workflow Executor runs as distributed `mapPartitions` or on the driver. See xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native Spark pipeline engine].
+
+A 3-row CSV may still use a single partition under `DISTRIBUTED`; the folders still prove package path resolution and shared-volume side effects.
+
+== 6. Nested Native Spark pipelines (optional)
+
+For **heavy** per-key work (Dataset I/O, shuffles), use **Pipeline Executor** with a **Native Spark** child run configuration and **Force Driver Only** on the executor transform — not a new `spark-submit` per key.
+
+[source,bash]
+----
+docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
+ exec spark /opt/hop-samples/spark-demo/scripts/spark-submit-pipelines-demo.sh
+----
+
+[source,bash]
+----
+find /tmp/spark-demo-dist/hop-data/out/by-country -type f | sort
+----
+
+Expect `out/by-country/{Genovia,Wakanda,Latveria}/…` under the host hop-data dir, and logs:
+`Nested Native Spark pipeline reusing parent SparkSession`.
+
+Child definition: `${PROJECT_HOME}/pipelines/nested/enrich-by-country.hpl` (in the project package). Control plane stays on the driver; each child builds a Dataset graph on the **same** Spark session.
+
+== Pipeline shapes (why they matter)
+
+=== Mapping
+
+[source]
+----
+Spark File Input (${HOP_DATA}/customers-sample.csv)
+ │
+ ▼
+Simple Mapping (filename = ${PROJECT_HOME}/pipelines/mappings/upper-name.hpl)
+ │ ↑ only works if the package was extracted on this executor
+ ▼
+Spark File Output (${HOP_DATA}/out/enriched)
+----
+
+The child mapping concatenates first + last name and uppercases the result. If the package were missing on a worker, the job would fail opening the child `.hpl` — that is the gap this design closes.
+
+=== Workflow Executor
+
+[source]
+----
+Spark File Input (${HOP_DATA}/countries.csv)
+ │
+ ▼
+Workflow Executor (filename = ${PROJECT_HOME}/workflows/create-country-folder.hwf)
+ parameter COUNTRY_NAME ← country_name
+ → Create Folder ${HOP_DATA}/out/countries/${COUNTRY_NAME}
+ → Create File …/${Internal.Pipeline.ID}.txt
+----
+
+Side-effect paths use **`HOP_DATA`** (shared volume). The workflow definition uses **`PROJECT_HOME`** (package).
+
+=== Nested Native Spark (orchestrator)
+
+[source]
+----
+Spark File Input (${HOP_DATA}/countries.csv)
+ │
+ ▼
+Pipeline Executor Force Driver Only; runConfiguration = spark-cluster (Native Spark)
+ │ filename = ${PROJECT_HOME}/pipelines/nested/enrich-by-country.hpl
+ │ parameter COUNTRY_NAME ← country_name
+ ▼
+Child SparkPipelineEngine (reuses parent SparkSession)
+ Spark File Input → Spark File Output (${HOP_DATA}/out/by-country/${COUNTRY_NAME})
+----
+
+== Tear down
+
+[source,bash]
+----
+docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml down
+----
+
+Host data under `/tmp/spark-demo-dist/hop-data` is left in place so you can keep inspecting outputs and execution-info JSON. Delete that directory manually if you want a clean slate.
+
+== Troubleshooting
+
+[options="header",cols="1,2"]
+|===
+|Symptom |What to check
+
+|Fat jar / package not found in container
+|`HOP_DIST_DIR` must point at the host folder mounted as `/opt/hop-dist`. Re-run `prepare-dist.sh`.
+
+|Workers cannot open mapping file
+|Confirm logs mention distributing the package / `SparkFiles`. Scale workers ≥ 1 and ensure submit used `--HopProjectPackage`.
+
+|Spark File Input path not found
+|`HOP_DATA` must be the **shared** volume path (`file:///data/hop-data`). Do not point Dataset paths at package `PROJECT_HOME`.
+
+|ClassNotFoundException / wrong Spark
+|Fat jar must be **`native-provided`**. Cluster image must be Spark **4.1.x** + **Java 21**. Do not mix Beam Spark 3 fat jars.
+
+|Driver cannot reverse-connect workers
+|Submit script sets `spark.driver.host=spark` (master hostname on the compose network). Adjust if you change service names.
+
+|Spark download slow / fails on build
+|Dockerfiles fall back to archive.apache.org. Set `SPARK_BASE_URL` or pre-build images once.
+|===
+
+== Follow-ups (not required for this walk-through)
+
+* MinIO + `s3a://` data plane (needs `hadoop-aws` on the Spark classpath).
+* HDFS instead of a bind mount.
+* CI job that runs prepare + compose + submit non-interactively.
+
+== Related pages
+
+* xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark engine]
+* xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems on native Spark]
+* xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native Spark pipeline engine]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/databricks.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/databricks.adoc
new file mode 100644
index 0000000000..c62ef8dabc
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/databricks.adoc
@@ -0,0 +1,544 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[NativeSparkOnDatabricks]]
+:imagesdir: ../../../assets/images
+:description: Run Apache Hop native Spark (MainSpark JAR) jobs on Databricks — workspace tiers, classic clusters, UC Volumes, compute modes, and Deploy & run lessons learned.
+:toc:
+:toclevels: 3
+
+= Native Spark on Databricks
+
+This page captures practical lessons for running **Hop native Spark** pipelines on a **Databricks** workspace via the xref:workflow/actions/databricks-job-run.adoc[Databricks Job Run] **Deploy & run** path (`MainSpark` JAR task).
+
+It is not a full Databricks admin guide. It focuses on what usually goes wrong when connecting Hop to Free / trial / serverless-oriented workspaces versus a **classic-cluster-capable** (e.g. Premium) workspace suitable for Spark 4.1 and modern Java.
+
+For general native Spark concepts (engine, fat jar, `local[*]` first), start with
+xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark engine].
+
+== Sample project shape
+
+A minimal Hop project for this path typically includes:
+
+* A small pipeline to run on the cluster (e.g. generate rows → random UUID → dummy)
+* A workflow with **Databricks Job Run** (Deploy & run) and optional **Databricks Job Wait**
+* A `work/` folder with the **native-provided** fat jar
+* Metadata: Databricks Connection, Native Spark pipeline run configuration
+
+.Sample `hello-databricks.hpl` in the project explorer (Generate rows → UUID → Dummy)
+image::pipeline/spark/databricks-pipeline-hello-databricks.png[Hello Databricks sample pipeline in Hop GUI: Generate rows, Random value (uuid), Dummy,width=85%]
+
+== What Deploy & run does
+
+When mode is **Deploy & run**, the Job Run action:
+
+. Uploads artifacts to a workspace path (UC Volume or classic DBFS). Names are chosen so concurrent deploys and library refreshes stay correct:
+ * fat jar → **content-addressed** `hop-native-<sha12>.jar` (first 12 hex chars of the local jar’s SHA-256) plus a matching `.sha256` sidecar — skip upload when that remote path already matches size + sidecar (see <<faq-skip-jar-upload>> and <<lesson-fat-jar-library-uri>>)
+ * pipeline → `pipeline-{stem}.hpl` (simple mode; may be omitted when a package-relative path is used)
+ * exported Hop metadata → `metadata-{stem}.json` (simple mode; embedded in the package when package mode is on)
+ * optional Spark project package → `hop-spark-package-{stem}.zip` (nested Simple Mapping / Pipeline Executor)
+ * optional environment config → `env-config-{stem}.json` (MainSpark `--HopConfigFile`)
+ * `{stem}` = sanitized pipeline file basename without extension (e.g. `hello-mapping-databricks`)
+. Creates or updates a Databricks **Jobs** definition with a `spark_jar_task`:
+ * main class: `org.apache.hop.spark.run.MainSpark`
+ * parameters: classic positional `pipeline`, `metadata`, `runConfig` — or named `--Hop*` args when a project package and/or env file is present
+. Calls `jobs/run-now` and optionally waits for completion (or fire-and-forget +
+ xref:workflow/actions/databricks-job-wait.adoc[Databricks Job Wait]).
+
+Credentials come from xref:metadata-types/databricks-connection.adoc[Databricks Connection]
+metadata (workspace host + personal access token). That is **not** the
+xref:database/databases/databricks.adoc[Databricks JDBC] SQL warehouse connection.
+
+.Deploy & run dialog: connection, fat jar / pipeline paths, UC Volume upload base, classic `new_cluster` (e.g. DBR `18.2.x-scala2.13`, AWS `i3.xlarge`)
+image::pipeline/spark/databricks-workflow-deploy-to-databricks.png[Databricks Job Run dialog: Deploy and run with fat jar, pipeline, Volume base path, and new_cluster settings,width=90%]
+
+== Workspace and tier: classic clusters vs serverless
+
+Hop’s Deploy & run path is built for a **classic compute** story: either an **existing
+all-purpose cluster** or a classic **job cluster** (`new_cluster` in the Jobs API).
+
+[options="header",cols="1,2,2"]
+|===
+|Workspace style |Typical compute |Hop Deploy & run JAR jobs
+
+|Free / some trial / serverless-only
+|No (or very limited) user-created classic clusters; serverless Jobs only
+|*Not* a good fit for full MainSpark JAR deploy today. Serverless has different job shape (environments / `environment_key` / `java_dependencies`), runtime limits (often older Java / constrained Spark), and no dynamic classic cluster create.
+
+|Premium (or equivalent) with **classic clusters** enabled
+|All-purpose clusters and/or classic job clusters (e.g. AWS `m5.*`, DBR with Spark 4.x)
+|**Supported path.** Create or attach a classic cluster that can run your Spark / Java line, then use existing cluster id or `new_cluster`.
+|===
+
+=== Lessons learned
+
+* **Free tier / “cannot create a cluster”** — many Free or UI-first workspaces never expose a classic cluster id. Deploy & run then has nothing valid to put in `existing_cluster_id`. Putting the literal string `new_cluster` into that field without a nested object produces errors such as _Cluster new_cluster does not exist_ (the API treated it as a cluster id).
+* **Serverless-only error** — `Only serverless compute is supported in the workspace` means classic `new_cluster` / existing all-purpose clusters are blocked. Serverless needs job-level `environments` and task `environment_key`, and **must not** send task-level `libraries` the same way classic jobs do.
+* **Spark 4.1 / Java 21 style workloads** — for Hop’s **native** Spark 4.x engine and a fat jar built for modern runtimes, provision a workspace that **allows classic clusters** (e.g. AWS Premium workspace) and a DBR / job cluster configuration aligned with your Spark and JVM expectations. Do not assume Free/serverless will match local `native-provided` fat-jar testing.
+* **New workspace for classic** — if the current workspace is permanently serverless-oriented, creating a **new** classic-capable workspace (cloud account + tier that allows classic compute) is often simpler than fighting Free-tier product limits.
+* [[lesson-fat-jar-library-uri]]**Fat jar library URI must change when jar bytes change (Dedicated / long-lived cluster “caching crapout”)** — overwriting a **fixed** Volume path such as `/Volumes/…/hop-native.jar` (even with Files API `?overwrite=true` and a fresh `.sha256` sidecar) is **not** enough for Databricks to always run the new classes. A long-lived **Dedicated** (or other all-purpose) cluster often keeps **previously installed job libraries** on the driver/executor classpath ( [...]
+ **Volume object is minutes old and matches local `sha256sum`**, but MainSpark driver log still shows **old behavior** (e.g. missing `>>>>>> Spark project package distribution build: pkg-dist-…` right after `Initializing Hop`, or obsolete package-distribution log strings such as `cluster-shared … (no SparkFiles)`). Hop Deploy & run therefore uploads a **content-addressed** library path `hop-native-<sha12>.jar` and points the job’s `libraries[].jar` at that URI so each content change is [...]
+
+== Connection and secrets
+
+* Create a **Databricks Connection** (Jobs / PAT), not only a JDBC entry.
+* Prefer Hop **variables** for host and token (project or lifecycle environment JSON).
+* Password-style fields and variable values may be stored as `Encrypted …`. The Jobs client resolves variables and then calls `Encr.decryptPasswordOptionallyEncrypted` so both plain and encrypted PAT values work.
+* **Test connection** in the metadata editor uses the same variable space as Hop GUI (`MetadataManager` / `HopGui` variables). After changing the plugin, rebuild the **client assembly** so the running `plugins/tech/databricks` jar matches your source (evaluate-in-IDE vs stale plugin jar is a common false lead).
+
+Example environment variables:
+
+----
+DATABRICKS_HOST=https://dbc-….cloud.databricks.com
+DATABRICKS_TOKEN=Encrypted …
+----
+
+Connection fields:
+
+----
+host = ${DATABRICKS_HOST}
+token = ${DATABRICKS_TOKEN}
+----
+
+CLI (with projects/environments registered in Hop config):
+
+[source,bash]
+----
+cd $HOP_HOME # e.g. assemblies/client/target/hop
+export HOP_CONFIG_FOLDER=/path/to/your/hop-config-dir
+./hop run -j my-project -e my-databricks-env \
+ -f main-deploy-to-databricks.hwf -r local -l BASIC
+----
+
+[[artifact-storage]]
+== Artifact storage: UC Volumes vs DBFS
+
+Modern workspaces often have **public DBFS root disabled**. Uploads to
+`dbfs:/FileStore/…` then fail with:
+
+----
+PERMISSION_DENIED: Public DBFS root is disabled
+----
+
+=== Prefer Unity Catalog Volumes
+
+. Create a volume (e.g. catalog `apache-hop`, schema `default`, volume `jars`).
+. Set **Upload base directory** on Job Run Deploy & run to something like:
+
+----
+/Volumes/apache-hop/default/jars
+----
+
+Artifacts land as (names depend on jar content and entry pipeline stem):
+
+----
+/Volumes/apache-hop/default/jars/hop-native-2f80e51734a5.jar
+/Volumes/apache-hop/default/jars/hop-native-2f80e51734a5.jar.sha256
+/Volumes/apache-hop/default/jars/pipeline-hello-databricks.hpl
+/Volumes/apache-hop/default/jars/metadata-hello-databricks.json
+/Volumes/apache-hop/default/jars/hop-spark-package-hello-databricks.zip
+----
+
+The `.sha256` sidecar next to the content-addressed jar lets later deploys **skip re-uploading** the same bytes (see <<faq-skip-jar-upload>>). The content-addressed **filename** is what forces Databricks to treat a new jar as a new job library (see <<lesson-fat-jar-library-uri>>).
+
+.UC Volume after deploy: fat jar, SHA-256 sidecar, exported metadata, and pipeline
+image::pipeline/spark/databricks-workspace-catalog-jars-volume.png[Databricks Catalog Explorer volume jars showing hop-native jar, checksum sidecar, metadata, and pipeline artifacts,width=80%]
+
+=== Which upload API Hop uses
+
+[options="header",cols="1,2,2"]
+|===
+|Path prefix |API |Notes
+
+|`/Volumes/…` or `/Workspace/…`
+|**Files API** — `PUT /api/2.0/fs/files{path}?overwrite=true` with `Content-Type: application/octet-stream`
+|Required for UC Volumes. Single streaming PUT (up to ~5 GiB). Parent folders/volume must exist and be writable by the PAT.
+
+|Classic `dbfs:/FileStore/…` (or other non-Volume paths)
+|**Legacy DBFS** — `dbfs/create` → `add-block` → `close`
+|Only where public DBFS root is still enabled. Not interchangeable with Volumes.
+|===
+
+If logs still show `POST /api/2.0/dbfs/create` for a `/Volumes/…` path, the running plugin jar is almost certainly **older than the Files API fix** — rebuild/copy `hop-tech-databricks` into the client `plugins/tech/databricks` folder and restart Hop.
+
+Job library and MainSpark parameter paths for volumes stay as absolute `/Volumes/…` (no `dbfs:` scheme).
+
+== Compute modes (Deploy & run)
+
+The **Cluster / compute** field accepts one of:
+
+[options="header",cols="1,2,2"]
+|===
+|Value |Jobs API shape |When to use
+
+|**Existing cluster id**
+|Task: `existing_cluster_id` + task `libraries` (jar)
+|You already started a classic all-purpose cluster. Paste its cluster id.
+
+|Literal **`new_cluster`**
+|Task: nested `new_cluster` object (`spark_version`, `node_type_id`, `num_workers`, …) + task `libraries`
+|Classic **job cluster**: Databricks creates a cluster for the run and tears it down afterward (similar idea to ephemeral Dataflow workers). **Not** the string `"existing_cluster_id": "new_cluster"`.
+
+|Literal **`serverless`**
+|Job: `environments` with `environment_key` + `spec.client` + `java_dependencies`; task: `environment_key` only (no cluster fields, no task `libraries`)
+|Serverless-only workspaces. JAR-on-serverless is a different product surface (runtime limits, library rules). Prefer classic for Hop native Spark 4.x / Java 21 style deploy until you explicitly target serverless.
+|===
+
+=== Classic job cluster (`new_cluster`) defaults
+
+When the field is `new_cluster`, Hop can build:
+
+[source,json]
+----
+"new_cluster": {
+ "spark_version": "18.2.x-scala2.13",
+ "node_type_id": "i3.xlarge",
+ "num_workers": 1
+}
+----
+
+* **`spark_version`** — Databricks Runtime string. Example: `18.2.x-scala2.13` (DBR line shipping Spark 4.1.x / Scala 2.13). Align with the runtime you validated for the fat jar.
+* **`node_type_id`** — cloud-specific. On **AWS**, prefer **`i3.xlarge`** (local NVMe SSD; no extra EBS block required). Avoid bare `m5.xlarge` without EBS (see <<faq-ebs-m5>>). Azure example: `Standard_DS3_v2`. Wrong type → create-job 400.
+* **`num_workers`** — worker count (string fields support variables).
+* Optional **full new_cluster JSON** overrides the structured fields when you need more knobs (Spark conf, single-node, EBS volumes, etc.).
+
+[[spark-conf-cores]]
+==== Match Spark cores and parallelism to the node
+
+Hop’s simple `new_cluster` defaults only set `spark_version`, `node_type_id`, and `num_workers`. Without explicit Spark conf, DBR may under-use the instance CPUs and multi-million-row Hop pipelines can crawl even when the cluster looks “idle.”
+
+**Tune `spark_conf` so cores and default parallelism match the machine you bought.** Example for **one worker `i3.xlarge` (4 vCPUs)** with the driver and executor sharing that capacity:
+
+[source,json]
+----
+"new_cluster": {
+ "spark_version": "18.2.x-scala2.13",
+ "node_type_id": "i3.xlarge",
+ "num_workers": 1,
+ "spark_conf": {
+ "spark.executor.cores": "3",
+ "spark.driver.cores": "1",
+ "spark.default.parallelism": "12",
+ "spark.sql.shuffle.partitions": "12"
+ }
+}
+----
+
+| Setting | Example (1× `i3.xlarge`, 4 cores) | Intent |
+| --- | --- | --- |
+| `spark.executor.cores` | `3` | Executor threads on the worker |
+| `spark.driver.cores` | `1` | Driver share of the same node (3+1 = 4 cores available) |
+| `spark.default.parallelism` | `12` | Default RDD partitions (often a few × total cores) |
+| `spark.sql.shuffle.partitions` | `12` | Shuffle partitions; keep in the same ballpark |
+
+Put this under **optional new_cluster JSON** on the Job Run **Compute** tab (cluster field = `new_cluster`). Scale the numbers when you change node type or worker count (more workers → more total cores → raise parallelism/partitions).
+
+**Observed impact (Hello-style multi-million-row Native Spark pipeline on classic job compute):** pipeline **execution** dropped from on the order of **~10 minutes** to about **~41 seconds** after applying the conf above — cold start / Pending time is unchanged (see <<job-cluster-cold-start>>).
+
+[[job-cluster-cold-start]]
+==== Job cluster cold start is slow
+
+With **`new_cluster`**, Databricks provisions a fresh classic job cluster for the run. That is **not** instant: cloud VMs, DBR image pull, and Spark/JVM startup often take **several minutes**. A small sample such as *Hello Databricks* can sit in **Pending / Initializing** for **~8 minutes or more** before the pipeline itself starts — especially on the first run of a node type/runtime in the account.
+
+.Jobs UI: *Hello Databricks* run stuck in Initializing while the classic job cluster starts (normal; often many minutes)
+image::pipeline/spark/databricks-jobs-pipelines-hello-databricks-initializing.png[Databricks Jobs run for Hello Databricks showing pipeline Initializing while the job cluster starts,width=90%]
+
+Once the cluster is up, the same run moves to **Running** and a tiny pipeline usually finishes quickly:
+
+.Jobs UI: *Hello Databricks* pipeline Running after the job cluster is ready
+image::pipeline/spark/databricks-jobs-pipelines-hello-databricks-running.png[Databricks Jobs run for Hello Databricks showing pipeline Running,width=90%]
+
+When the fat jar includes the TrapExit-safe `MainSpark` exit path, the same run ends as **Succeeded**:
+
+.Jobs UI: *Hello Databricks* pipeline Succeeded (after cluster start + pipeline run)
+image::pipeline/spark/databricks-jobs-pipelines-hello-databricks-succeeded.png[Databricks Jobs run for Hello Databricks showing pipeline Succeeded,width=90%]
+
+Even a trivial *Hello Databricks* job still pays **Spark / JVM baseline** on the driver container: metrics often show on the order of **~4 GB** of container memory used (not “a few MB for three transforms”). That is expected overhead for DBR + Spark + loading the fat jar — size your node type for the **platform floor**, not only for row volume.
+
+.Metrics: *Hello Databricks* container memory used ~4 GB vs a much higher limit on `i3.xlarge`
+image::pipeline/spark/databricks-jobs-pipelines-hello-databricks-memory-usage-4gb.png[Databricks container memory usage chart for Hello Databricks showing about 4 GB used,width=90%]
+
+Plan timeouts accordingly:
+
+* On **Databricks Job Run** (wait mode) or **Databricks Job Wait**, set **timeout** high enough for **jar upload (first time) + cluster start + pipeline run** — e.g. **3600s** (1 hour), not a few minutes.
+* Prefer fire-and-forget + Job Wait if you do not want the workflow action to hold a long poll in one step.
+* **Existing all-purpose cluster** (already Running) avoids most of this wait; **`new_cluster`** always pays the cold-start cost each run.
+
+=== Existing classic cluster
+
+. In Databricks UI: **Compute** → create/start a classic all-purpose cluster on the new workspace.
+. **Access mode must be Dedicated (single user)** — not **Standard** (shared) and not Serverless. Standard/Serverless use **Spark Connect**, which does **not** expose `SparkContext`. Hop Native Spark needs a real context for RDDs, accumulators, and mapPartitions.
+. Wait until **Running**.
+. Copy **Cluster ID**.
+. Set **Cluster / compute** to that id (not `new_cluster` / not `serverless`).
+. Leave job id empty on first create; after success, store `${DatabricksJobId}` if you enable **Update existing job**.
+
+If you see:
+
+----
+[UNSUPPORTED_CONNECT_FEATURE.SESSION_SPARK_CONTEXT] Feature is not supported in Spark Connect:
+SparkContext is not available on Standard access mode or Serverless compute.
+Switch to Dedicated access mode to access it.
+----
+
+…the cluster is the wrong access mode. Either switch the all-purpose cluster to **Dedicated**, or use Deploy & run with **`new_cluster`** (classic job cluster), which provides a full SparkContext.
+
+== FAQ
+
+[[faq-ebs-m5]]
+=== Create job fails: “At least one EBS volume must be attached … node type m5.xlarge”
+
+**Symptom**
+
+----
+Databricks API HTTP 400 for POST /api/2.1/jobs/create
+INVALID_PARAMETER_VALUE
+At least one EBS volume must be attached for clusters created with node type m5.xlarge.
+----
+
+**Cause**
+
+On AWS, **`m5.xlarge`** (and many non-`i3` instance types) do not ship local instance store disks that Databricks can use as worker/driver local storage for a classic job cluster. Databricks then requires an **EBS volume** attachment in the `new_cluster` definition. Hop’s simple defaults only set `spark_version`, `node_type_id`, and `num_workers` — no EBS block — so `m5.xlarge` fails at `jobs/create`.
+
+**Fix (simplest, recommended on AWS)**
+
+Switch **`node_type_id`** to **`i3.xlarge`** (or another **`i3.*`** size that fits your budget).
+
+* `i3` instances include **local NVMe SSDs**, so Databricks does **not** require extra EBS configuration for a basic job cluster.
+* `i3.xlarge` is a commonly used worker type for Databricks jobs.
+
+In the Job Run dialog (Deploy & run, cluster field = `new_cluster`):
+
+* Set **new_cluster node_type_id** to `i3.xlarge`
+
+Or in optional **new_cluster JSON**:
+
+[source,json]
+----
+{
+ "spark_version": "18.2.x-scala2.13",
+ "node_type_id": "i3.xlarge",
+ "num_workers": 1
+}
+----
+
+**Alternative**
+
+Keep `m5.xlarge` (or similar) and supply a full `new_cluster` JSON that includes the required **EBS volume** settings for your account/cloud (see Databricks Jobs API / cluster create docs for `aws_attributes` / disk specs). Prefer `i3.xlarge` unless you have a reason to stay on EBS-backed types.
+
+=== Other common create/deploy errors
+
+[options="header",cols="1,2"]
+|===
+|Message (summary) |What to do
+
+|Public DBFS root is disabled / FileStore access denied
+|Use a UC Volume path (`/Volumes/…`), not `dbfs:/FileStore/…`. See <<artifact-storage, artifact storage>> above.
+
+|`/Volumes/… cannot be opened for writing` with `dbfs/create`
+|Volume path must use the **Files API**, not legacy DBFS. Rebuild/update the Databricks plugin; upload base should start with `/Volumes/`.
+
+|Cluster `new_cluster` does not exist
+|Do not put the word `new_cluster` as an existing cluster id without the nested object. Use cluster field **`new_cluster`** so Hop emits a real `new_cluster` block.
+
+|Only serverless compute is supported
+|Workspace is serverless-only. Use classic-capable workspace/tier for MainSpark JAR, or `serverless` compute mode (different job shape; not the default for Hop native Spark 4.x / Java 21 style deploy).
+|===
+
+[[faq-exit-marks-failed]]
+=== Pipeline log says success but Databricks marks the job **Failed**
+
+**Symptom**
+
+Driver stdout looks healthy, for example:
+
+----
+Spark pipeline finished, result row count=1000
+>>>>>> Execution finished...
+----
+
+…then Databricks Jobs still shows the run as **Failed**. You may also see:
+
+----
+Error running native Spark pipeline: Program attempted to exit with code 0
+com.databricks.backend.daemon.driver.ExitSecurityException: Program attempted to exit with code 0
+ at com.databricks.backend.daemon.driver.TrapExitSecurityManager.checkExit(...)
+ at java.base/java.lang.System.exit(...)
+ at org.apache.hop.spark.run.MainSpark.main(...)
+----
+
+Logs may mention cleaning a REPL wrapper. A line such as `Hop configuration file not found … /databricks/driver/config/hop-config.json` is **benign** (Hop skips serializing config when that path is missing).
+
+**Cause**
+
+Databricks installs **`TrapExitSecurityManager`**: any `System.exit` (including **code 0**) throws **`ExitSecurityException`** instead of ending the process. Older `MainSpark` called `System.exit(0)` after a successful pipeline; the security manager blocked it, and the generic catch block mis-reported that as *Error running native Spark pipeline*, so Jobs marked the run **Failed** even though Spark finished (`result row count=1000`). Same class of issue as Databricks KB *“Job fails, but [...]
+
+**Fix**
+
+* Use a fat jar where `MainSpark` **returns from `main` without `System.exit(0)`** on Databricks (env / `SPARK_HOME` under `/databricks` / TrapExit security manager), and treats a trapped exit-0 as success if exit is still attempted.
+* Rebuild **native-provided** fat jar, re-upload (or refresh via size/checksum skip), and re-run.
+* On plain `spark-submit` (non-Databricks), success still uses `System.exit(0)` so non-daemon Spark threads do not hang the JVM.
+
+**Healthy driver log (success)** — Jobs UI should show **Succeeded**:
+
+----
+>>>>>> Initializing Hop
+Hop configuration file not found, not serializing: /databricks/driver/config/hop-config.json
+Argument : Pipeline path (.hpl) : /Volumes/apache-hop/default/jars/pipeline.hpl
+Argument : Metadata export (.json) : /Volumes/apache-hop/default/jars/metadata.json
+Argument : Pipeline run configuration : databricks-native
+>>>>>> Loading pipeline metadata
+>>>>>> Validating native Spark engine plugin in fat jar...
+>>>>>> SPARK_HOME=/databricks/spark
+>>>>>> Pipeline execution starting (native Spark)...
+2026/07/18 13:06:41 - pipeline - Prepared native Spark pipeline engine with run configuration 'databricks-native' (master=)
+2026/07/18 13:06:41 - pipeline - Reusing active SparkSession (version=4.1.0, spark-submit or existing driver context)
+2026/07/18 13:06:53 - General - Handled transform '5M rows' with generic Spark DISTRIBUTED (plugin id=RowGenerator)
+2026/07/18 13:06:54 - General - Handled transform 'uuid' with generic Spark DISTRIBUTED (plugin id=RandomValue)
+2026/07/18 13:06:54 - General - Handled transform 'results' with generic Spark DISTRIBUTED (plugin id=Dummy)
+2026/07/18 13:16:43 - pipeline - Spark pipeline finished, result row count=5000000
+>>>>>> Execution finished...
+>>>>>> Skipping System.exit(0) (Databricks / TrapExit security manager)
+----
+
+Notes on that log:
+
+* `Hop configuration file not found … hop-config.json` — **benign**.
+* `Reusing active SparkSession` — expected on Databricks (do not `spark.stop()` / do not force a second context).
+* `Handled transform …` lines are **graph build** (seconds); wall time until `result row count=…` is the real work.
+* Final line **`Skipping System.exit(0) (Databricks / TrapExit security manager)`** is the success marker for the exit fix above; the Jobs UI should show **Succeeded** (see the screenshot under <<job-cluster-cold-start>>).
+* Example timing: without core/parallelism conf, **5 000 000** rows can take on the order of **~10 minutes** on classic job compute; with <<spark-conf-cores, cores matched to the node>> (e.g. 3+1 on `i3.xlarge`), the same style workload has been seen around **~40 seconds** of *execution* time. A pure Spark SQL `range` job would still be faster; Hop generic transforms are not that path (see also <<faq-low-cpu-rowgen>>).
+
+[[faq-low-cpu-rowgen]]
+=== Low cluster CPU / slow Row Generator (millions of rows)
+
+**Symptom**
+
+Databricks metrics show only **~5–7%** cluster CPU while a multi-million-row pipeline runs for many minutes. You expected “a few seconds” for 5M rows.
+
+**Cause**
+
+Source transforms such as **Generate rows** are wired as **one empty driver row → `repartition(1)` → one `mapPartitions` task**. Downstream generic transforms (e.g. Random value, Dummy) stay on that **single partition** unless something shuffles. Each partition runs a **single-threaded Hop mini-pipeline** (not Catalyst `range`), and the engine materializes with `Dataset.count()` at the end. Cluster-wide CPU stays low because **most cores never get a task**.
+
+**What to expect**
+
+* Spark UI: source transforms such as Generate rows still often start as **1 task** (see engine design).
+* Still set <<spark-conf-cores, executor/driver cores and shuffle partitions>> so DBR uses the CPUs you pay for — under-configured clusters show low CPU and multi-minute runs for millions of rows even after the cluster is “Running”.
+* Wall clock for generic Hop transforms remains higher than pure Spark SQL; local clusters show the same architectural limits.
+
+**When you need real scale**
+
+Prefer partitioned sources (files, lake tables, or a multi-partition generator), keep high-volume work in native Spark I/O where possible, and treat generic Hop `mapPartitions` as **compatibility**, not as a bulk SQL engine.
+
+[[faq-grpc-context]]
+=== Job fails at init: `NoClassDefFoundError: io/grpc/Context`
+
+**Symptom**
+
+----
+Error running native Spark pipeline:
+java.lang.NoClassDefFoundError: io/grpc/Context
+ at … GoogleCredentials.getApplicationDefault …
+ at org.apache.hop.vfs.gs.GoogleStorageFileProvider …
+ at org.apache.hop.core.HopEnvironment.init …
+ at org.apache.hop.spark.run.MainSpark.main …
+----
+
+**Cause**
+
+During `HopEnvironment.init`, Hop registers VFS plugins. The **Google Storage** VFS provider used to call **Application Default Credentials** immediately. That path probes the GCE metadata service and needs **`io.grpc.Context`**. A **native-provided** fat jar typically includes Beam’s *vendored* gRPC (`org.apache.beam.vendor.grpc…`) but **not** plain `io.grpc.*`. On **Databricks AWS** (not GCP) ADC is unnecessary for a simple Hello pipeline, but the missing class still aborted Hop startup.
+
+**Fix**
+
+* Use a Hop build where Google Storage VFS **does not fail Hop init** when ADC/gRPC is unavailable (credentials loaded only when a key is configured, and linkage errors are swallowed for the default provider).
+* Rebuild the fat jar from that install and re-upload (or let skip-upload refresh the sidecar after you replace the jar).
+* If you actually need `gs://` on the cluster, either ship a service account key config or ensure full Google auth/gRPC libraries are on the driver classpath (not only Beam-vendored gRPC).
+
+== Fat jar and pipeline run configuration
+
+. Build a **native-provided** fat jar (Spark provided by the cluster, not bundled):
+
+[source,bash]
+----
+./hop-conf.sh \
+ --generate-fat-jar=/path/to/hop-native-spark-submit.jar \
+ --spark-client-version=native-provided
+----
+
+. Create a **Native Spark** pipeline run configuration in project metadata (name is exported into `metadata.json`). Deploy & run passes that name as the third MainSpark argument.
+. Point Deploy & run at:
+ * fat jar path (variables OK, e.g. `${PROJECT_HOME}/work/….jar`)
+ * pipeline `.hpl`
+ * that run configuration name
+ * upload base under a Volume
+ * classic compute (existing id or `new_cluster`)
+
+First jar upload is slow (hundreds of MB). Classic **`new_cluster`** cold start is also slow (often **~8+ minutes** of Pending/Initializing before a tiny pipeline runs — see above). Timeouts on the action should allow for upload **plus** cluster start **plus** the pipeline (e.g. **3600s**).
+
+[[faq-skip-jar-upload]]
+=== Fat jar re-upload every run is slow — can Hop skip it?
+
+**Yes (content-addressed path + size + checksum sidecar).** Deploy & run computes the local fat jar’s SHA-256 and targets:
+
+----
+{upload-base}/hop-native-<first-12-hex-of-sha>.jar
+{upload-base}/hop-native-<first-12-hex-of-sha>.jar.sha256
+----
+
+. Probe that **content-addressed** remote path for **file size** (Files API metadata / DBFS get-status).
+. If size matches, download the tiny sidecar and compare to the local **SHA-256**.
+. If both match → log *Skipping fat jar upload* and leave the remote jar as-is (same library URI as last time).
+. If the local jar content changed → the remote **filename** changes → upload a new object and point the job library at the new URI (see <<lesson-fat-jar-library-uri>>).
+
+Pipeline (`.hpl`) and metadata JSON are still uploaded each run when used (they are small). Project package zips are per-pipeline stem.
+
+**Manual check** (without Hop):
+
+[source,bash]
+----
+ls -l work/hop-native-spark4-submit.jar
+sha256sum work/hop-native-spark4-submit.jar
+# First 12 hex chars of that sum → hop-native-<12hex>.jar on the Volume
+
+# Remote size (example Files API HEAD)
+curl -sS -I -H "Authorization: Bearer $DATABRICKS_TOKEN" \
+ "https://$DATABRICKS_HOST/api/2.0/fs/files/Volumes/.../hop-native-<12hex>.jar"
+----
+
+Databricks does **not** expose a standard content MD5/SHA for volume files comparable to S3 ETag, so Hop stores its own `.sha256` sidecar next to the jar. The **filename prefix** is the operational identity for job libraries; the sidecar is for skip-upload efficiency.
+
+== End-to-end checklist
+
+* [ ] Workspace allows **classic** compute (Premium or equivalent); not Free/serverless-only for JAR deploy
+* [ ] Databricks Connection + variables for host/token; Test connection succeeds
+* [ ] UC Volume exists and PAT can write; upload base is `/Volumes/…` (not disabled FileStore)
+* [ ] Client assembly includes current `hop-tech-databricks` plugin (Files API + compute modes)
+* [ ] Fat jar `native-provided`; Native Spark run config name matches deploy field
+* [ ] After a fat jar rebuild: deploy log shows content-addressed `hop-native-<sha12>.jar` (or restart Dedicated cluster if still on a fixed library path); driver log shows expected `pkg-dist-…` / MainSpark fingerprint when debugging library freshness
+* [ ] Cluster / compute = existing classic id **or** `new_cluster` with valid `node_type_id` / `spark_version`
+* [ ] Job id empty on first create; wait mode and timeout sized for upload + **job cluster cold start** (often 8+ minutes) + pipeline run
+* [ ] After success: note Job ID, Run ID, run page URL variables; open run in Databricks UI
+
+== Related
+
+* xref:workflow/actions/databricks-job-run.adoc[Databricks Job Run]
+* xref:workflow/actions/databricks-job-wait.adoc[Databricks Job Wait]
+* xref:metadata-types/databricks-connection.adoc[Databricks Connection]
+* xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark engine]
+* xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native Spark run configuration]
+* xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems]
+* xref:database/databases/databricks.adoc[Databricks JDBC] (SQL warehouse — separate use case)
+* xref:metadata-types/spark-catalog.adoc[Spark Catalog] (Unity Catalog templates for lakehouse TABLE mode)
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
new file mode 100644
index 0000000000..d8bfda450b
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
@@ -0,0 +1,706 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[GettingStartedWithNativeSpark]]
+:imagesdir: ../../../assets/images
+:description: Getting started with the native Apache Spark pipeline engine in Apache Hop (Spark 4.x, without Apache Beam).
+
+:toc:
+:toclevels: 3
+
+= Getting started with the native Spark pipeline engine
+
+This guide is for data engineers who already know the basics of Hop pipelines (transforms, hops, run configurations, metadata) and want to run *batch* pipelines on https://spark.apache.org[Apache Spark] using Hop’s *native* Spark engine.
+
+It walks you from “what is this?” through a `local[*]` first run, cluster `spark-submit`, Spark-oriented I/O, metrics, limitations, and concrete examples from the `integration-tests/spark-native` project.
+
+For **Delta Lake** and **Apache Iceberg** (ACID tables, catalogs, time travel, MERGE, maintenance), see the dedicated xref:pipeline/spark/lakehouse.adoc[lakehouse guide] after you have a working native Spark run configuration.
+
+Before putting object-store paths in *Spark File* or *Lake Table* transforms, read
+xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems on native Spark] — Hop VFS (`s3://`, named MinIO) and Spark/Hadoop URIs (`s3a://`, …) are different dialects.
+
+For **Simple Mapping** / **Pipeline Executor** child pipelines on `spark-submit`, export a
+xref:pipeline/spark/paths-and-filesystems.adoc#spark-project-package[Spark project package]
+(`hop-conf -j … --export-spark-project=…`) and pass `--HopProjectPackage=…` to `MainSpark`.
+
+Hands-on multi-node demo (Docker Compose + sample project `spark-demo` — Simple Mapping and Workflow Executor):
+xref:pipeline/spark/cluster-mapping-walkthrough.adoc[Cluster demo walk-through].
+
+To deploy **MainSpark** JAR jobs to a **Databricks** workspace (classic clusters, UC Volumes, Job Run Deploy & run):
+xref:pipeline/spark/databricks.adoc[Native Spark on Databricks].
+
+== What is the native Spark pipeline engine?
+
+=== In one sentence
+
+The native Spark pipeline engine takes the *metadata* of a Hop pipeline and executes it as a Spark SQL `Dataset` graph on *Apache Spark 4.1.x* — without translating the pipeline through Apache Beam.
+
+=== How that differs from Beam Spark
+
+Apache Hop already had a way to run pipelines on Spark: the xref:pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.adoc[Beam Spark pipeline engine]. That path converts your Hop graph into an https://beam.apache.org[Apache Beam] pipeline, then uses Beam’s Spark *runner* (Spark *3.5.x* line).
+
+The *native* engine skips Beam entirely:
+
+* Hop’s `SparkPipelineEngine` builds a logical plan of Spark Datasets on the driver.
+* Each Hop transform becomes either a *native handler* (Spark APIs such as `groupBy`, `join`, `read` / `write`) or a *generic* `mapPartitions` stage that runs a small single-threaded Hop mini-pipeline once per Spark partition.
+* You use Hop’s *Spark File Input* / *Spark File Output* transforms for bulk Dataset I/O (not Beam File Input/Output).
+
+[options="header",cols="1,2,2"]
+|===
+|Topic |Native Spark engine |Beam Spark engine
+
+|Purpose
+|Batch Hop pipelines on Spark Datasets / mapPartitions
+|Beam pipelines on Spark (batch *and* streaming)
+
+|Spark line
+|https://spark.apache.org/docs/4.1.2/[4.1.x] (Scala 2.13)
+|3.5.x (Scala 2.12; see Beam support matrix)
+
+|Programming model
+|Spark SQL `Dataset` + Hop mini-pipelines
+|Apache Beam PCollections + runners
+
+|Streaming / windows
+|Out of scope — use Beam
+|Supported via Beam
+
+|Preferred bulk I/O
+|*Spark File Input* / *Spark File Output*
+|*Beam File Input* / *Beam File Output*
+
+|Side data (lookups)
+|Info streams (broadcast), e.g. Stream Lookup
+|Beam side inputs
+
+|Conditional fan-out
+|Target streams (Filter Rows, Switch/Case)
+|Beam multi-output tags
+
+|Engine id (plugin)
+|`SparkPipelineEngine` — “Native Spark pipeline engine”
+|Beam Spark run configuration
+|===
+
+If you need streaming, windowing, or Beam-only connectors, stay on the Beam engines. If you want a straightforward batch path on modern Spark 4 with Dataset-oriented I/O, use native Spark.
+
+=== How a Hop pipeline becomes a Spark job
+
+. You design a normal Hop pipeline (`.hpl`) and select a *Native Spark* pipeline run configuration.
+. On prepare, the engine walks the *enabled* transform graph in order.
+. For each transform it either:
+** applies a *native handler* that rewrites the hop as Spark Dataset operations, or
+** wraps the transform in `mapPartitions` so each Spark partition runs that transform in a tiny local Hop pipeline.
+. File outputs and other Spark *actions* materialise the graph (read → transform → write).
+. Transform metrics are collected via Spark accumulators and shown in Hop like any other engine.
+
+You still design pipelines *visually* in Hop. Spark’s role is the distributed execution engine and the Dataset runtime under the hood.
+
+=== Version support matrix
+
+[NOTE]
+====
+*Java:* Hop requires *Java 21* for the GUI, `hop-run`, and the fat jar executed on the Spark driver/executors. Your Spark cluster must also run a JVM that can load that bytecode (Java 21).
+
+*Spark:* The native engine is built and tested against *Spark 4.1.x* (currently *4.1.2* in the Hop source tree, Scala *2.13*). For `local[*]` runs, Spark libraries are packaged under `plugins/engines/spark/lib`, so you do *not* need a separate Spark install on your laptop.
+====
+
+[options="header",cols="1,1,2,2"]
+|===
+|Hop version |Native Spark engine |Target Spark |Notes
+
+|2.19.0+
+|Included (`plugins/engines/spark`)
+|4.1.x (Scala 2.13)
+|Independent of Beam’s Spark 3.5.x client packs
+
+|Earlier Hop versions
+|—
+|—
+|Native Spark engine not available; use Beam Spark if needed
+|===
+
+Do *not* mix this engine with Beam’s Spark 3.5 fat-jar packs or an accidental `SPARK_HOME` that points at Spark 3.x when you intend to use native Spark 4.1.
+
+== Prerequisites
+
+* Apache Hop *2.19+* with the *engines-spark* plugin (standard Hop client / assembly includes it).
+* *Java 21*.
+* Familiarity with Hop pipelines, metadata, and run configurations.
+* For cluster submit:
+** A *Spark 4.1.x* distribution (`SPARK_HOME` pinned to that tree).
+** Network access to the master and workers.
+** Paths for data and metadata that workers can reach (HDFS, object store, shared FS).
+* Optional but recommended: an *execution information location* and *execution data profile* for history, metrics, and row samples in the GUI.
+
+== How to use it
+
+=== Create a `local[*]` pipeline run configuration
+
+This is the fastest way to learn: Spark runs *inside* the Hop process using all local cores (`local[*]`), with no cluster and no fat jar.
+
+TIP: On the Native Spark engine options tab, **Load configuration template** can pre-fill common setups (*Local execution*, *Spark standalone server*, *spark-submit*, *Databricks*, *YARN*, …). See xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native Spark pipeline engine].
+
+TIP: For Iceberg **TABLE** mode, create a xref:metadata-types/spark-catalog.adoc[Spark Catalog] entry and use **Load catalog template** (Hadoop local/object store, REST, Hive, Glue, Nessie, …). Each preset starts **Extra conf** with a commented `# docs:` link to vendor documentation.
+
+. Open *Metadata* → *Pipeline run configuration* → *New*.
+. Give it a clear name, for example `spark-local`.
+. Set *Engine type* to *Native Spark pipeline engine*.
+. On the engine tab, set at least:
+
+[options="header",cols="1,3"]
+|===
+|Option |Suggested first value
+
+|Spark master
+|`local[*]` (or `local[4]` to cap cores)
+
+|Application name
+|`Apache Hop` (or your project name)
+
+|Fat jar file location
+|Leave *empty* for pure `local[*]`
+
+|Spark config
+|Optional; one `key=value` per line (advanced tuning)
+
+|Driver / executor memory & cores
+|Optional; leave empty for local defaults
+
+|Temp location
+|Optional; defaults to the JVM temp directory
+|===
+
+. On the main run-configuration tab, optionally attach:
+** *Execution information location* — where Hop stores run history
+** *Execution data profile* — which row samples to capture
+. Save the configuration.
+
+image::pipeline/spark/native-spark-pipeline-engine-run-configurtion-editor.png[Native Spark pipeline run configuration editor,width=80%]
+
+Full option reference: xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native Spark pipeline engine].
+
+=== First pipeline on `local[*]`
+
+. Create a small batch pipeline, for example:
+** *Spark File Input* — CSV path under your project, header and separator, field list
+** *Filter Rows* or *Select Values* — any familiar transform
+** *Spark File Output* — path, format (`csv` or `parquet`), save mode *Overwrite*
+. Run the pipeline and choose your `spark-local` run configuration.
+. When it finishes, check:
+** Transform metrics on the pipeline graph (input / output / written / errors)
+** Execution Information perspective if you configured a location
+
+You do not need `spark-submit` or a fat jar for this path. Hop starts an embedded Spark session and loads Spark from `plugins/engines/spark/lib`.
+
+=== Run from the command line (local)
+
+Same run configuration, no GUI:
+
+[source,bash]
+----
+./hop-run.sh \
+ --project=your-project \
+ --runconfig=spark-local \
+ --file=/path/to/your-pipeline.hpl
+----
+
+=== Build a fat jar for cluster submit
+
+On a real cluster you must *not* embed a second copy of Spark inside the application jar. The cluster already provides Spark 4.1. Embedding Spark leads to dual classpaths and failures on Java 21.
+
+Use the special fat-jar token *`native-provided`*: Hop and plugins go into the jar; *Spark stays out*.
+
+==== From Hop GUI
+
+. *Tools* → *Generate a Hop fat jar* (when prompted for Spark client version for native submit, use the *native-provided* style packaging for cluster submit — or use the CLI below for an explicit flag).
+. Save the jar *outside* the project (or gitignore it). Fat jars are large.
+
+==== From the command line (recommended, explicit)
+
+[source,bash]
+----
+# From your Hop installation directory
+./hop-conf.sh \
+ --generate-fat-jar=/tmp/hop-native-spark4-submit.jar \
+ --spark-client-version=native-provided
+----
+
+[options="header",cols="1,2,2"]
+|===
+|Token |When to use |Spark runtime
+
+|`native-provided`
+|`spark-submit` on a cluster
+|Cluster provides Spark 4.1.x (*do not* embed)
+
+|`native`
+|Rare: fat jar that *includes* Spark 4 from `plugins/engines/spark/lib` (not for normal cluster submit)
+|Embedded Spark 4
+|===
+
+=== Export project metadata
+
+Cluster runs need your Hop metadata (run configurations, execution locations, connections, etc.) as a single JSON file.
+
+* GUI: *Tools* → *Export metadata to JSON*
+* CLI:
+
+[source,bash]
+----
+./hop-conf.sh \
+ --project=your-project \
+ --export-metadata=/tmp/hop-metadata.json
+----
+
+Keep this export out of git if it is a point-in-time dump of secrets or large metadata; your project `metadata/` folder is already the source of truth.
+
+=== Detailed `spark-submit` instructions
+
+. Install or unpack *Spark 4.1.x* (Hadoop 3 build is typical).
+. *Pin `SPARK_HOME`* to that directory. If `SPARK_HOME` still points at an older Spark (for example 3.3), even a `spark-submit` binary from a 4.1 tree can re-exec the wrong `spark-class` and load the wrong version.
+. Build the *native-provided* fat jar and export metadata (previous sections).
+. Ensure the pipeline run configuration used on the cluster:
+** Engine = *Native Spark pipeline engine* (not Beam Spark)
+** *Master* blank (so `spark-submit --master` wins) *or* set to `spark://host:7077`
+** *Fat jar* field empty (the submit jar already carries Hop)
+. Submit:
+
+[source,bash]
+----
+export SPARK_HOME=/path/to/spark-4.1.x-bin-hadoop3
+
+"${SPARK_HOME}/bin/spark-submit" \
+ --master spark://your-master:7077 \
+ --deploy-mode client \
+ --class org.apache.hop.spark.run.MainSpark \
+ /tmp/hop-native-spark4-submit.jar \
+ /path/to/pipeline.hpl \
+ /tmp/hop-metadata.json \
+ YourNativeSparkRunConfig \
+ /path/to/cluster-env-config.json
+----
+
+*Main class:* `org.apache.hop.spark.run.MainSpark`
+
+*Positional arguments:*
+
+. Path to the pipeline (`.hpl`) — must be readable on the *driver*
+. Path to metadata JSON — readable on the driver
+. Name of the native Spark pipeline run configuration (as in metadata)
+. Optional: *environment configuration file* (JSON) — extra variables for the run (see below)
+
+*Named arguments* (equivalent):
+
+[source,bash]
+----
+"${SPARK_HOME}/bin/spark-submit" \
+ --master spark://your-master:7077 \
+ --class org.apache.hop.spark.run.MainSpark \
+ /tmp/hop-native-spark4-submit.jar \
+ --HopPipelinePath=/path/to/pipeline.hpl \
+ --HopMetadataPath=/tmp/hop-metadata.json \
+ --HopRunConfigurationName=YourNativeSparkRunConfig \
+ --HopConfigFile=/path/to/cluster-env-config.json
+----
+
+==== Environment configuration file (4th argument)
+
+On a laptop, Hop projects usually set `PROJECT_HOME` to a local folder. In a *clustered* `spark-submit` job that path often has no meaning: drivers and executors do not share your desktop filesystem, and paths in pipelines are typically `${PROJECT_HOME}/…` or other project variables.
+
+Pass an optional **environment configuration file** as the 4th positional argument (or `--HopConfigFile=…`). It uses the same JSON format as a Hop environment / `hop-config` variables file. `MainSpark` loads every described variable into the pipeline variable space before execution.
+
+Example `cluster-env-config.json`:
+
+[source,json]
+----
+{
+ "variables": [
+ {
+ "name": "PROJECT_HOME",
+ "value": "s3a://hop-project/",
+ "description": "Project root on object storage for the cluster run (Spark/Hadoop S3A URI, not Hop VFS s3://)"
+ },
+ {
+ "name": "OUTPUT_ROOT",
+ "value": "s3a://hop-project/output/",
+ "description": "Shared output prefix visible to all executors"
+ }
+ ]
+}
+----
+
+Use this for any value that must differ between local development and the cluster: connection endpoints, bucket prefixes, `${PROJECT_HOME}`, credentials placeholders you resolve elsewhere, and so on.
+
+[IMPORTANT]
+====
+* The config file path must be readable on the *driver* (local path on the submit machine, or a URI Hop VFS can open there).
+* Values such as `s3a://hop-project/` only help if workers can actually read/write that location (correct Spark/Hadoop **S3A** configuration, IAM roles, etc.). Prefer **`s3a://`** for Spark Dataset I/O — Hop VFS uses `s3://` for a *different* stack. See xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems on native Spark].
+* Variables in this file override defaults for *this* submit only; they do not rewrite your project’s GUI environment on disk.
+====
+
+*Data paths* in the pipeline (files, tables, object stores) must be reachable from *workers*, not only from your laptop. Prefer HDFS, S3/GS/ABFS, or a shared mount — and set `PROJECT_HOME` (and similar) in the environment config so `${PROJECT_HOME}/…` resolves correctly on the cluster.
+
+*Resource flags* (optional examples):
+
+[source,bash]
+----
+"${SPARK_HOME}/bin/spark-submit" \
+ --master spark://your-master:7077 \
+ --executor-memory 4g \
+ --executor-cores 2 \
+ --driver-memory 2g \
+ --class org.apache.hop.spark.run.MainSpark \
+ /tmp/hop-native-spark4-submit.jar \
+ /path/to/pipeline.hpl \
+ /tmp/hop-metadata.json \
+ YourNativeSparkRunConfig \
+ /path/to/cluster-env-config.json
+----
+
+You can also put memory/core settings in the run configuration’s engine options; CLI flags are useful for per-job overrides.
+
+[#_building_spark_oriented_pipelines]
+== Building Spark-oriented pipelines
+
+=== Prefer Spark File Input and Spark File Output
+
+For large or production Dataset I/O, use the dedicated transforms in the *Big Data* category:
+
+* *Spark File Input* — `spark.read.format(...).load`
+* *Spark File Output* — `df.write.format(...).save`
+
+Typical first pipeline shape:
+
+image::pipeline/spark/spark-native-0001-input-output.png[Spark File Input to Spark File Output sample,width=70%]
+
+==== Spark File Input
+
+Common options:
+
+[options="header",cols="1,3"]
+|===
+|Option |Notes
+
+|Spark path (Hadoop URI)
+|File, directory, or glob visible to Spark (local path on `local[*]`; cluster URI such as `hdfs://…` or `s3a://…` on submit). **Not** a Hop VFS path — see xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems].
+
+|Format
+|`csv`, `parquet`, `json`, `orc`, `text`
+
+|Header / separator / quote
+|CSV and text
+
+|Field list
+|Preferred for stable schemas. With a header, columns are selected and cast *by name*. Without a header, Spark names columns `_c0`, `_c1`, … and Hop maps them *by position* to your field list.
+
+|Infer schema
+|Optional; explicit fields are safer for production
+|===
+
+==== Spark File Output
+
+[options="header",cols="1,3"]
+|===
+|Option |Notes
+
+|Spark path (Hadoop URI)
+|Output directory/prefix Spark should write to (same Spark/Hadoop dialect as input; not Hop VFS)
+
+|Format
+|`csv`, `parquet`, `json`, `orc`, `text`
+
+|Save mode
+|Overwrite, Append, Ignore, ErrorIfExists
+
+|Header
+|CSV/text
+
+|Coalesce
+|e.g. `1` for a single part file (costs a shuffle)
+
+|Partition by
+|Optional partition column list for directory partitioning
+|===
+
+Writes run as Spark *actions* when the engine materialises the graph. Prefer *Spark File Output* for large partitioned writes to object stores or HDFS.
+
+=== Open table formats (Delta Lake / Apache Iceberg)
+
+Classic *Spark File* transforms cover `csv` / `parquet` / `json` / `orc` / `text` only. For ACID lake tables use the dedicated **Lake Table** transforms (Big Data category):
+
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark Lake Table Input] / xref:pipeline/transforms/spark-lake-table-output.adoc[Spark Lake Table Output]
+* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark Lake Table Merge] / xref:pipeline/transforms/spark-lake-table-maintenance.adoc[Spark Lake Table Maintenance]
+* xref:metadata-types/spark-catalog.adoc[Spark Catalog] metadata for Iceberg TABLE mode
+
+Connector JARs **ship in the default Hop assembly** under `plugins/engines/spark/lib/{delta,iceberg}` (and ride a *native-provided* fat jar for cluster submit). Full session conf, PATH vs TABLE, and samples: xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine].
+
+=== Classic Hop I/O and `${Internal.Transform.ID}`
+
+You can still use classic transforms such as *Text File Input*, *Text File Output*, *Get File Names*, or *Microsoft Excel Input* on the generic mapPartitions path (useful for POCs and small lookups). Those plugins are staged for the Spark engine via `plugins/engines/spark/dependencies.xml`.
+
+*Sources with no previous hop* (for example Get File Names, or Excel/Text Input reading a fixed file list) run on a *single* partition so the same small file is not listed or opened once per executor — the same idea as Beam’s single-thread pattern for non-Beam inputs.
+
+==== Unique filenames per partition
+
+Each Spark partition runs its own copy of a classic writer. If every copy uses the same path, files overwrite each other.
+
+*Recommended:* put `${Internal.Transform.ID}` in the filename template, for example:
+
+[source,text]
+----
+${PROJECT_HOME}/out/export_${Internal.Transform.ID}.csv
+----
+
+On the native Spark engine, that variable is set to a *readable* value:
+
+[source,text]
+----
+{transform-name}-{partition-id}
+----
+
+`${Internal.Transform.CopyNr}` and `${Internal.Transform.BundleNr}` are set to the Spark partition id as well.
+
+*Also:* the engine enables Beam-parity parallel-file context so *Text File Output* can auto-append `_<transformId>_<partition>` to the built filename (same idea as Beam’s `beamContext` suffix). Relying on `${Internal.Transform.ID}` in the path remains the clearest approach for operators.
+
+[NOTE]
+====
+For large distributed writes, prefer *Spark File Output* (Spark manages part files). Classic Text File Output is ideal for small POC side-loads and local/shared paths where unique names per partition are enough.
+====
+
+==== Accept filenames from a previous transform
+
+*Get File Names* → *Text File Input* (accept filenames from field) works on native Spark: filename rows are the main Dataset into Text File Input; the mini-pipeline rebinds the accept source to the partition injector and preloads filename rows before the file reader drains them.
+
+Example shape from the integration tests:
+
+image::pipeline/spark/spark-native-0008-text-file-input-spark-readback.png[Get File Names to Text File Input to Spark File Output,width=70%]
+
+== Metrics and execution information
+
+=== Live metrics in the pipeline editor
+
+While a native Spark pipeline runs (and when it finishes), Hop shows transform-level metrics on the graph — lines read/written/input/output, errors, and running/finished state — similar to the local engine. Values are aggregated on the *driver* from Spark accumulators.
+
+image::pipeline/spark/native-spark-provides-runtime-transform-metrics-and-state.png[Runtime transform metrics and state on a native Spark pipeline,width=90%]
+
+=== Execution information location
+
+To keep *history* after the run (and to drill into past executions):
+
+. Create an xref:metadata-types/execution-information-location.adoc[Execution information location] (file, caching file, remote Hop Server, …).
+. Optionally create an xref:metadata-types/execution-data-profile.adoc[Execution data profile] (first/last/random rows, etc.).
+. On the *Native Spark* pipeline run configuration, set both on the main tab.
+. Run a pipeline, then open the xref:hop-gui/perspective-execution-information.adoc[Execution Information] perspective.
+
+image::pipeline/spark/native-spark-logs-to-execution-information-location.png[Execution information logged from a native Spark run,width=90%]
+
+How data is produced:
+
+* *State and metrics* (`updateExecutionState`) — computed on the *driver* from `EngineMetrics` / accumulators for all transform types.
+* *Row samples* (`registerData`) — collected on *executors* for generic `mapPartitions` transforms when a data profile is set, then merged for the GUI (parent/child execution hierarchy, similar to Beam).
+
+[IMPORTANT]
+====
+If the location is file-based and you need *samples from cluster workers*, the root folder must be *writable from every executor* (shared filesystem or object store). A path that exists only on the driver will still get pipeline/transform *state*, but executor samples will be missing or incomplete.
+====
+
+On `local[*]`, Spark’s Jackson libraries must not shadow Hop’s Jackson under the plugin classloader. The Spark plugin packaging excludes core Jackson jars so sampling and `HopJson` keep working.
+
+== How transforms map to Spark
+
+=== Native handlers
+
+These use Spark Dataset APIs on the driver graph (no per-row Hop mini-pipeline for the core of the work):
+
+[options="header",cols="1,2"]
+|===
+|Transform |Spark mapping
+
+|Memory Group By
+|`groupBy` + aggregations (SUM, AVG, MIN, MAX, COUNTs, FIRST/LAST, …)
+
+|Merge Join
+|`Dataset.join` (inner / left / right / full outer)
+
+|Unique Rows
+|`dropDuplicates` (optional count via groupBy)
+
+|Sort Rows
+|`orderBy`
+
+|Spark File Input / Output
+|`spark.read` / `df.write`
+|===
+
+=== Generic mapPartitions transforms
+
+Most other transforms — Filter Rows, Switch/Case, Stream Lookup, Calculator, Select Values, Dummy, classic file transforms, Excel, and many plugins — run inside `mapPartitions`:
+
+. Each Spark partition starts a *single-threaded* Hop mini-pipeline for that transform.
+. Partition rows are injected; outputs are collected back into a Dataset.
+. Plugin JARs used by those transforms must be on the Spark classpath (`dependencies.xml` for the engine plugin, and the fat jar on the cluster).
+
+=== Info streams and target streams
+
+*Info streams* (side data)::
+Transforms such as *Stream Lookup* need a full secondary dataset while processing the main stream. The engine *broadcasts* the info Dataset to partitions (same idea as Beam `View.asList`). Configure the Stream Lookup *from* transform name exactly.
+
+*Target streams* (conditional routing)::
+*Filter Rows* (true/false) and *Switch/Case* (cases + default) declare named targets. Each input row goes to *one* successor. Downstream hops use keys of the form `Previous - TARGET - Current`.
+
+*Multiple previous transforms*::
+If several transforms hop into one (branches rejoin), row layouts must match (names, order, types). The engine *unions* those Datasets after a layout check (Beam Flatten equivalent).
+
+[[limitations-and-design-rules]]
+== Limitations and design rules
+
+Keep these in mind when moving a local pipeline to native Spark.
+
+[options="header",cols="1,3"]
+|===
+|Topic |Behaviour on native Spark
+
+|Transforms marked *Not Supported*
+|See the authoritative list on
+xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc#transforms-not-supported-on-native-spark[Native Spark Pipeline Engine — Transforms not supported].
+It is derived from each transform page’s *Supported Engines* table (*Native Spark* row).
+Examples: sorted *Group By* (use *Memory Group By*), Beam-only I/O, barriers (*Blocking until transforms finish*), result-row hops, *Kafka Consumer*, multi-way / sorted merges.
+
+|Sorted *Group By*
+|*Not supported.* Use *Memory Group By* (native `groupBy`).
+
+|Copy vs Distribute on multi-hop output
+|*Not implemented* (same stance as Beam). Named *target streams* route by condition; plain multi-next hops without target metadata are rejected. Round-robin distribute is not available.
+
+|Streaming / windows
+|Out of scope. Use the Beam engines.
+
+|Native Spark Kafka
+|*Not supported.* There is no Hop transform that maps to Spark’s structured streaming / Kafka source-sink APIs. For Kafka-oriented Beam pipelines, use the Beam engines (e.g. Beam Kafka Consumer/Producer). Hop’s own Kafka transforms (if used) are not native-Spark Dataset I/O.
+
+|Spark SQL
+|*Not supported* as a first-class surface. You cannot submit arbitrary Spark SQL / `DataFrame` queries from Hop metadata; the engine only builds a Dataset graph from the Hop transform graph (plus native handlers for a fixed set of transforms).
+
+|MLlib / Spark ML
+|*Not supported.* There are no native handlers or transforms for training, pipelines, or model inference via Spark MLlib / `spark.ml`. Keep ML workloads in Spark jobs outside Hop, or call external services from ordinary Hop transforms where that fits.
+
+|Large Stream Lookup tables
+|Fully *broadcast* — size carefully.
+
+|Unique Rows (error stream)
+|Reject-to-error style options may not apply; prefer Dataset-oriented dedup semantics.
+
+|Transform copies / partitioning GUI
+|Parallelism comes from *Spark partitions*, not Hop “copies” in the local sense.
+
+|Plugins in mapPartitions
+|Must be present on the Spark classpath (staged list + fat jar). Missing plugins fail at runtime on the executor.
+
+|Beam-only transforms
+|Beam File I/O, Beam Kafka/Kinesis/BigQuery/… are for Beam engines, not native Spark.
+|===
+
+== Examples from the integration tests
+
+The Hop source tree includes a self-contained project:
+
+[source,text]
+----
+integration-tests/spark-native/
+----
+
+Open that folder as a Hop project (or copy patterns into your own). Workflows are named `main-00xx-*.hwf` and typically: clean output → run the Spark pipeline with `spark-local` → validate on the local engine.
+
+=== 0001 — Spark File Input / Output
+
+Minimal path: read a CSV with *Spark File Input*, write with *Spark File Output*.
+
+image::pipeline/spark/spark-native-0001-input-output.png[0001 input output,width=70%]
+
+=== 0002 — Memory Group By
+
+Aggregation via the native Memory Group By handler (Spark `groupBy`).
+
+image::pipeline/spark/spark-native-0002-group-by.png[0002 group by,width=70%]
+
+=== 0003 — Filter Rows
+
+Row filtering in mapPartitions.
+
+image::pipeline/spark/spark-native-0003-filter-rows.png[0003 filter rows,width=70%]
+
+=== 0004 — Stream Lookup (info stream)
+
+Main stream plus a lookup Dataset broadcast into *Stream Lookup* — exercises info streams end-to-end.
+
+image::pipeline/spark/spark-native-0004-stream-lookup.png[0004 stream lookup,width=75%]
+
+=== 0005 — Filter Rows with target streams
+
+True/false targets (conditional fan-out without Copy/Distribute).
+
+image::pipeline/spark/spark-native-0005-filter-targets.png[0005 filter targets,width=75%]
+
+=== 0006 — Switch / Case
+
+Multi-way routing with named cases and default.
+
+image::pipeline/spark/spark-native-0005-switch-case.png[0006 switch case,width=75%]
+
+=== 0007 — Classic Text File Input and Spark write-back
+
+Get File Names → Text File Input (accept filenames) → Spark File Output (e.g. Parquet). Validates classic I/O and partition-safe readback.
+
+image::pipeline/spark/spark-native-0008-text-file-input-spark-readback.png[0007 text file input spark readback,width=70%]
+
+=== 0008 — Excel info stream into Stream Lookup
+
+Main data from multiple Parquet files (*Spark File Input*); product dimension from *Microsoft Excel Input* into *Stream Lookup*; CSV result via *Spark File Output*.
+
+image::pipeline/spark/spark-native-0008-excel-stream-lookup.png[0008 excel stream lookup,width=75%]
+
+=== 0099 — Complex sample
+
+A larger pipeline combining several of the patterns above (good stress check for a full Hop + Spark install).
+
+image::pipeline/spark/spark-native-0099-complex.png[0099 complex sample,width=95%]
+
+== Checklist
+
+* [ ] Java 21; Hop 2.19+ with `plugins/engines/spark`
+* [ ] Run configuration engine = *Native Spark* (not Beam Spark)
+* [ ] First success on `local[*]` with Spark File Input → transform → Spark File Output
+* [ ] Execution information location attached if you need history and samples
+* [ ] Cluster: fat jar with `--spark-client-version=native-provided`
+* [ ] Cluster: `SPARK_HOME` pinned to Spark *4.1.x*
+* [ ] Cluster: metadata JSON exported; run configuration name matches submit args
+* [ ] No sorted Group By; no reliance on Copy/Distribute multi-hop behaviour
+* [ ] Classic writers use `${Internal.Transform.ID}` (or accept Spark File Output for bulk writes)
+* [ ] Lakehouse (optional): confirm `plugins/engines/spark/lib/{delta,iceberg}` present; see xref:pipeline/spark/lakehouse.adoc[lakehouse guide]
+
+== Related pages
+
+* xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native Spark pipeline engine] (option reference)
+* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine] (Delta / Iceberg)
+* xref:pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.adoc[Beam Spark pipeline engine] (contrast)
+* xref:pipeline/beam/getting-started-with-beam.adoc[Getting started with Apache Beam]
+* xref:pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc[Pipeline run configurations]
+* xref:metadata-types/execution-information-location.adoc[Execution information location]
+* xref:metadata-types/execution-data-profile.adoc[Execution data profile]
+* xref:metadata-types/spark-catalog.adoc[Spark Catalog]
+* xref:hop-gui/perspective-execution-information.adoc[Execution Information perspective]
+* xref:hop-tools/hop-conf/hop-conf.adoc[hop-conf] (fat jar, metadata export)
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/lakehouse.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/lakehouse.adoc
new file mode 100644
index 0000000000..57b850ee32
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/lakehouse.adoc
@@ -0,0 +1,314 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[NativeSparkLakehouse]]
+:imagesdir: ../../../assets/images
+:description: Open table formats (Delta Lake and Apache Iceberg) on the Apache Hop native Spark pipeline engine
+:toc:
+:toclevels: 3
+
+= Lakehouse tables on the native Spark engine
+
+This guide covers **Delta Lake** and **Apache Iceberg** support on Hop’s xref:pipeline/spark/getting-started-with-native-spark.adoc[native Spark pipeline engine] (Spark *4.1.x*, no Beam).
+
+It assumes you already have a working *Native Spark* pipeline run configuration (`local[*]` or cluster `spark-submit`). Classic file formats (CSV, Parquet, JSON, ORC, text) stay on *Spark File Input* / *Spark File Output*. Lakehouse work uses dedicated **Lake Table** transforms.
+
+== What you get
+
+[options="header",cols="2,1"]
+|===
+| Capability | How
+
+| Read / write ACID tables | xref:pipeline/transforms/spark-lake-table-input.adoc[Spark Lake Table Input] / xref:pipeline/transforms/spark-lake-table-output.adoc[Spark Lake Table Output]
+
+| Path-based tables | Identifier mode **PATH** (local, HDFS, object-store URIs Spark can open)
+
+| Catalog tables | Identifier mode **TABLE** + xref:metadata-types/spark-catalog.adoc[Spark Catalog] metadata (Iceberg-first). Use **Load catalog template** on the metadata editor for Hadoop/REST/Hive/Glue/Nessie skeletons.
+
+| Time travel | Version / snapshot id or timestamp on Input
+
+| Upsert | xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark Lake Table Merge] (`MERGE INTO`)
+
+| Compaction / vacuum / expire | xref:pipeline/transforms/spark-lake-table-maintenance.adoc[Spark Lake Table Maintenance]
+|===
+
+*Not in scope:* Hudi; Beam Spark lakehouse.
+
+== Version pins
+
+[options="header",cols="2,1"]
+|===
+| Component
+| Coordinate / version
+
+| Spark (Hop native engine)
+| `spark-sql_2.13` *4.1.2* (Scala *2.13*)
+
+| Delta Lake
+| `io.delta:delta-spark_4.1_2.13:*4.3.1*`
+
+| Apache Iceberg
+| `org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:*1.11.0*`
+
+|===
+
+The `_4.1_2.13` suffix means **Spark 4.1 / Scala 2.13**, not the product major version of Delta or Iceberg.
+
+== Connectors (shipped by default)
+
+Default Hop client assemblies package Delta Lake and Apache Iceberg under the **Spark engine plugin classloader**:
+
+[source,shell]
+----
+$HOP_HOME/plugins/engines/spark/lib/delta/ # Delta jars
+$HOP_HOME/plugins/engines/spark/lib/iceberg/ # Iceberg runtime jar
+----
+
+`SparkPipelineEngine` builds `SparkSession` under that classloader (`lib/` recursive + `dependencies.xml` folders). A sibling plugin folder alone is **not** enough.
+
+hop-run / Hop Gui (`local[*]`) therefore need **no manual connector install** after a normal Hop install or rebuild of `plugins/engines/spark`.
+
+=== Shipped jar list
+
+[options="header",cols="2,1"]
+|===
+|Jar |Role
+
+|`delta-spark_4.1_2.13-4.3.1.jar`
+|Delta Spark SQL / DataSource
+
+|`delta-storage-4.3.1.jar`
+|Delta storage layer
+
+|`delta-kernel-api-4.3.1.jar`
+|Delta Kernel API
+
+|`delta-kernel-defaults-4.3.1.jar`
+|Delta Kernel defaults
+
+|`delta-kernel-unitycatalog-4.3.1.jar`
+|Transitive from delta-spark 4.3.1
+
+|`unitycatalog-client-0.5.0.jar`
+|Transitive (UC-related)
+
+|`unitycatalog-hadoop-0.5.0.jar`
+|Transitive (UC-related)
+
+|`iceberg-spark-runtime-4.1_2.13-1.11.0.jar`
+|Shaded Iceberg Spark runtime
+|===
+
+Jackson, Spark, Scala, and slf4j are **not** duplicated into these folders (Hop parent CL + existing spark lib provide them).
+
+=== spark-submit (cluster)
+
+Prefer a **native-provided** fat jar (Hop + plugins + lakehouse connectors — **not** Spark). Fat-jar generation walks `plugins/engines/spark/lib` (including `delta/` and `iceberg/`) and keeps those connectors while excluding the Spark runtime.
+
+[source,bash]
+----
+export SPARK_HOME=/path/to/spark-4.1.x-bin-hadoop3
+
+./hop-conf.sh --generate-fat-jar=/tmp/hop-native-provided.jar \
+ --spark-client-version=native-provided
+
+# Export project metadata (run config + any Spark Catalog entries) to JSON, then:
+"${SPARK_HOME}/bin/spark-submit" \
+ --master spark://your-master:7077 \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
+ --class org.apache.hop.spark.run.MainSpark \
+ /tmp/hop-native-provided.jar \
+ /path/to/pipeline.hpl \
+ /path/to/metadata.json \
+ YourNativeSparkRunConfig
+----
+
+Optional: if you strip connectors from the fat jar or use site-managed `$SPARK_HOME/jars`, you can still add:
+
+----
+--packages io.delta:delta-spark_4.1_2.13:4.3.1,org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0
+----
+
+When both formats are used, keep **Delta** on `spark_catalog` and register Iceberg catalogs under **other** names (for example `lake`). Export full project metadata so `SparkCatalog` entries travel with the job.
+
+If connectors are missing, Hop fails with an actionable `HopException` (`SparkLakeConnectorProbe`).
+
+== Session configuration
+
+Hop’s `LakeSessionPlan` scans Lake Table transforms before the session is used and applies the conf needed for hop-run. Under pure `spark-submit`, extensions and catalogs must already be correct on the active session (CLI `--conf`, site defaults, or exported metadata that Hop can still register).
+
+[options="header",cols="1,2"]
+|===
+|When |Session conf (summary)
+
+|Any Delta use (PATH or TABLE)
+|`spark.sql.extensions` includes `io.delta.sql.DeltaSparkSessionExtension` **and** `spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog`
+
+|Any Iceberg PATH use
+|Iceberg extensions + built-in Hadoop catalog **`hop_iceberg`** (`type=hadoop`, warehouse under tmp)
+
+|Iceberg TABLE
+|`spark.sql.catalog.<name>=org.apache.iceberg.spark.SparkCatalog` (+ `type`, `warehouse` / `uri`, …) from xref:metadata-types/spark-catalog.adoc[Spark Catalog] metadata
+|===
+
+IMPORTANT: Delta PATH I/O on Spark 4.1 + Delta 4.3 **requires DeltaCatalog**, not only the extension. Iceberg bare `format("iceberg").load(path)` is **not** used — it defaults toward HiveCatalog and fails without HMS. PATH mode uses `hop_iceberg.\`file:///…\`` (or equivalent URI) instead.
+
+[[path-vs-table]]
+== PATH vs TABLE
+
+[options="header",cols="1,2,2"]
+|===
+|Mode
+|Identifier
+|Typical use
+
+|**PATH**
+|Directory / table root Spark can open (`file:///…`, `s3a://…`, …) — a **Spark/Hadoop URI**, not a Hop VFS path (`s3://`, named MinIO, …). See xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems].
+|Local lakes, object-store folders, quick prototypes
+
+|**TABLE**
+|Multi-part name (`catalog.namespace.table`) + optional Spark Catalog metadata name
+|Production catalogs (Iceberg Hadoop warehouse, REST, …)
+
+|===
+
+[options="header",cols="3"]
+|===
+| Format
+| PATH read
+| PATH write
+
+| **Delta**
+| `spark.read.format("delta").load(path)`
+| `format("delta").mode(…).save(path)`
+
+| **Iceberg**
+| SQL over `hop_iceberg.uri`
+| `writeTo(hop_iceberg.uri).using("iceberg")`
+
+|===
+
+
+[options="header",cols="3"]
+|===
+| Format
+| TABLE read
+| TABLE write
+
+| **Iceberg** (primary)
+| `SELECT * FROM catalog.ns.table`
+| `writeTo(id).using("iceberg")`
+
+| **Delta** (advanced)
+| `format("delta").table(id)`
+| `format("delta").saveAsTable(id)`
+
+
+|===
+
+
+== Lake Table transforms
+
+All four live under the **Big Data** category and are **native Spark only** (not Beam Spark, not the local Hop engine).
+
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark Lake Table Input] — read, optional projection, time travel
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark Lake Table Output] — append / overwrite / …; **default save mode is ErrorIfExists**
+* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark Lake Table Merge] — single upstream hop is the source; target is PATH or TABLE
+* xref:pipeline/transforms/spark-lake-table-maintenance.adoc[Spark Lake Table Maintenance] — zero-input action (OPTIMIZE, VACUUM, expire, …)
+
+Writes, merges, and maintenance run as Spark **actions** during graph materialisation and register empty leaf Datasets so a later engine `count()` does not re-execute them (same contract as *Spark File Output*).
+
+=== Topology rules
+
+* **Output / Merge** need exactly one main-stream upstream hop (the data to write or the MERGE source).
+* **Maintenance** is a zero-input sink — no hop required.
+* Prefer a single Lake Table Output / Merge / Maintenance leaf per branch; avoid graphs that would re-trigger the same ACID action.
+* Do not use *Spark File Output* with format `delta` / `iceberg` — use Lake Table transforms so session plan, probes, and SQL identifiers stay consistent.
+
+=== Metadata injection (MDI)
+
+All four Lake Table transforms support xref:pipeline/metadata-injection.adoc[metadata injection] so complex multi-table landings, CDC upserts, and fleet maintenance can use **one template pipeline** driven by an injecting pipeline (table path/identifier, format, merge condition, OPTIMIZE/VACUUM parameters, etc.).
+
+Injection keys are documented on each transform page. Shared identity keys: `FORMAT`, `IDENTIFIER_MODE`, `TABLE_PATH`, `TABLE_IDENTIFIER`, `CATALOG_METADATA_NAME`.
+
+=== Quick PATH round-trip (Delta)
+
+. Install Delta jars (see above) and restart Hop.
+. Create a *Native Spark* run configuration (`local[*]`).
+. Pipeline: *Row Generator* (or *Spark File Input*) → *Spark Lake Table Output*
+ ** Format: `delta`
+ ** Identifier mode: `PATH`
+ ** Table path: e.g. `${PROJECT_HOME}/output/delta-orders`
+ ** Save mode: `Overwrite` for a first write (default is *ErrorIfExists*)
+. Second pipeline or hop: *Spark Lake Table Input* on the same path → *Spark File Output* or preview.
+. Run with the native Spark run configuration.
+
+=== Time travel sketch
+
+Write twice with *Overwrite*, then set Input **Time travel** to `VERSION` and **Version** to `0` (Delta) or the first snapshot id from Iceberg’s `{table}.snapshots`. See the Input transform page for TIMESTAMP mode.
+
+=== MERGE sketch
+
+*Spark File Input* / generator → *Spark Lake Table Merge*:
+
+* Target: PATH or TABLE of an existing Delta/Iceberg table
+* Merge condition: e.g. `t.id = s.id` (`t` = target, `s` = source)
+* When matched: `UPDATE_ALL`; when not matched: `INSERT_ALL`
+
+=== Maintenance sketch
+
+Standalone *Spark Lake Table Maintenance*:
+
+* Operation `OPTIMIZE` (safe compaction)
+* For `VACUUM` / `EXPIRE_SNAPSHOTS` / `DELETE_WHERE`: set retention / WHERE as required and tick **Acknowledge destructive**
+
+== Troubleshooting
+
+[options="header",cols="1,2"]
+|===
+|Symptom |What to check
+
+|Probe / classpath error naming Delta or Iceberg classes
+|Standard install has jars under `plugins/engines/spark/lib/{delta,iceberg}`? Rebuild engines-spark / reinstall Hop? Fat jar built with current install? Optional cluster `--packages` / site jars?
+
+|`DELTA_CONFIGURE_SPARK_SESSION_WITH_EXTENSION_AND_CATALOG`
+|`spark_catalog` must be `DeltaCatalog` when using Delta (hop-run sets this when Delta transforms are present)
+
+|Iceberg PATH fails with HiveCatalog / HMS
+|Do not use bare `format("iceberg")` outside Hop; ensure Iceberg transforms run under hop-run so `hop_iceberg` is registered, or pass equivalent conf on submit
+
+|TABLE unresolved
+|Spark Catalog metadata exported? Catalog name matches the first segment of `catalog.ns.table`? Warehouse / URI reachable from driver and executors?
+
+|Destructive maintenance rejected
+|Set retention hours / WHERE and enable acknowledge flag
+
+|Writes seem to run twice
+|Only one action sink should materialise the write; check for extra Output hops or dual engine re-runs
+|===
+
+== Related pages
+
+* xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark pipeline engine]
+* xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native Spark pipeline engine]
+* xref:metadata-types/spark-catalog.adoc[Spark Catalog]
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark Lake Table Input]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark Lake Table Output]
+* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark Lake Table Merge]
+* xref:pipeline/transforms/spark-lake-table-maintenance.adoc[Spark Lake Table Maintenance]
+* Developer notes and spike findings: `plugins/engines/spark/README.md` in the Hop source tree
+* Integration tests: `integration-tests/lakehouse` (Delta) and `integration-tests/iceberg` (Iceberg) in the Hop source tree
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/paths-and-filesystems.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/paths-and-filesystems.adoc
new file mode 100644
index 0000000000..b98f8bcc25
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/paths-and-filesystems.adoc
@@ -0,0 +1,312 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[NativeSparkPathsAndFilesystems]]
+:imagesdir: ../../../assets/images
+:description: Hop VFS URIs versus Spark/Hadoop FileSystem URIs when using the native Spark pipeline engine.
+
+= Paths and file systems on native Spark
+
+Apache Hop uses *two different path languages*. They look similar (both use `scheme://…` URIs) but resolve through different stacks, credentials, and plugins. Mixing them is a common source of confusion when you move from the local engine to the xref:pipeline/spark/getting-started-with-native-spark.adoc[native Spark pipeline engine].
+
+== Two dialects
+
+[options="header",cols="1,2,2,2"]
+|===
+|Dialect
+|Resolved by
+|Typical schemes
+|Used for
+
+|**Hop VFS URI**
+|Apache Commons VFS via `HopVfs` and Hop VFS plugins
+|`file://`, `s3://`, `azure://` / `azfs://`, `gs://`, `googledrive://`, `dropbox://`, `webdav4(s)://`, **named** connections such as `minio://` or `myconn://`
+|Hop GUI file dialog, classic transforms (Text File Input/Output, Get File Names, …), loading pipeline/metadata on the Spark driver
+
+|**Spark URI** (Hadoop FileSystem URI)
+|Spark SQL Dataset I/O → Hadoop `FileSystem` on the *driver and every executor*
+|`file:///…`, `hdfs://…`, **`s3a://…`**, `abfs://…` / `wasbs://…`, connector-specific `gs://…`, …
+|xref:pipeline/spark/getting-started-with-native-spark.adoc#_building_spark_oriented_pipelines[Spark File Input/Output], Lake Table **PATH** mode, Iceberg/Delta warehouse locations
+|===
+
+[IMPORTANT]
+====
+A path that opens in the Hop *File* dialog (Hop VFS) is *not* automatically valid for *Spark File Input*.
+Spark never calls `HopVfs` for Dataset `load` / `save`. The engine only resolves Hop variables, then hands the string to Spark.
+====
+
+For the full list of Hop VFS schemes, see xref:vfs.adoc[Virtual File Systems].
+
+== When each stack is used on native Spark
+
+[options="header",cols="2,1,3"]
+|===
+|Feature |Path dialect |Notes
+
+|Spark File Input / Spark File Output
+|**Spark URI**
+|`spark.read…load(path)` / `df.write…save(path)`
+
+|Spark Lake Table (PATH mode)
+|**Spark URI**
+|e.g. `file:///…`, `s3a://bucket/table` — see xref:pipeline/spark/lakehouse.adoc[Lakehouse]
+
+|Spark Catalog warehouse / Iceberg `io-impl`
+|**Spark URI**
+|Templates use `s3a://` and `file:///`
+
+|Classic transforms via `mapPartitions` (Text File Input, Excel, …)
+|**Hop VFS URI**
+|Each partition runs a small Hop pipeline; VFS plugins and named connections work if the *executor* classpath and credentials allow it
+
+|`MainSpark` loading `.hpl` / metadata JSON
+|**Hop VFS URI** (driver only)
+|Driver must open the definition files; *data* paths still must work on executors
+|===
+
+== Scheme collision matrix
+
+[options="header",cols="1,2,2,2"]
+|===
+|You write |Hop VFS meaning |Spark meaning |What to do on native Spark File/Lake
+
+|`s3://bucket/key`
+|AWS S3 via Hop’s S3 VFS plugin (AWS SDK)
+|Often legacy or unused; production Spark expects **S3A**
+|Use **`s3a://bucket/key`** and configure `spark.hadoop.fs.s3a.*` (or cluster `core-site`)
+
+|`s3a://bucket/key`
+|**Not** a Hop VFS scheme (unless you add a bridge plugin)
+|Hadoop S3A FileSystem
+|Correct for Spark; configure IAM / keys / endpoint for S3A
+
+|`minio:///bucket/…` or `myconn://…`
+|Named MinIO or other VFS connection metadata
+|Unknown scheme unless Hadoop is registered for that name
+|Do *not* use named VFS schemes in Spark File path. Use `s3a://` + `fs.s3a.endpoint` (and path-style access) for MinIO-compatible stores
+
+|`azure://` / `azfs://`
+|Hop Azure VFS
+|Spark usually wants **`abfs://`** or **`wasbs://`**
+|Use Azure Hadoop schemes + connector conf for Dataset I/O
+
+|`gs://`
+|Hop Google Cloud Storage VFS
+|Spark GCS connector (same scheme, *different* jars/credentials)
+|Works only if the Spark GCS connector and credentials are on the cluster
+
+|`$\{PROJECT_HOME\}/data.csv`
+|Resolved, then Hop VFS or local rules
+|Resolved, then Hadoop default FS / local path
+|On cluster submit, set `PROJECT_HOME` in the env config to a **Spark-visible** URI (for object storage prefer `s3a://…`)
+|===
+
+== MinIO and S3-compatible storage
+
+Hop’s xref:metadata-types/minio-connection.adoc[Minio Connection] metadata registers the **connection name** as a VFS scheme (for example connection `minio` → `minio:///demo/…`). That path works with classic Hop transforms and the file dialog.
+
+On **Spark File Input/Output** and **Lake PATH**:
+
+. Put objects under a URI Spark’s S3A client can open, typically `s3a://bucket/prefix/…`.
+. Point S3A at your endpoint with run-configuration *Spark config* lines, for example:
+
+[source]
+----
+spark.hadoop.fs.s3a.endpoint=https://minio.example.com:9000
+spark.hadoop.fs.s3a.path.style.access=true
+spark.hadoop.fs.s3a.access.key=...
+spark.hadoop.fs.s3a.secret.key=...
+----
+
+(Exact keys depend on your Hadoop/Spark version and security model; prefer IAM roles on cloud clusters over embedding keys when possible.)
+
+[[spark-project-package]]
+== Project package (nested pipelines / mappings)
+
+Native Spark ships only the *top-level* pipeline XML plus metadata JSON by default. Transforms such as *Simple Mapping* and *Pipeline Executor* load child `.hpl` files at runtime via Hop VFS using paths like `$\{PROJECT_HOME\}/mappings/child.hpl`. On a cluster those files must exist where every executor can open them.
+
+=== Export a Spark project package
+
+This package is **specifically for Native Spark execution** (nested Simple Mapping / Pipeline Executor on the cluster). It is not the same as *File → Export current project to zip*.
+
+**GUI:** *Tools → Export project package for Native Spark…* (requires an active project / `PROJECT_HOME`).
+
+**CLI:**
+
+[source,bash]
+----
+# Preferred for scripts (no project registration required):
+./hop-conf.sh --export-spark-project=/tmp/my-project-spark.zip \
+ --export-spark-project-home=/path/to/project
+
+# Or enable a registered project first (-j, not --project/-p):
+./hop-conf.sh -j my-project --export-spark-project=/tmp/my-project-spark.zip
+----
+
+(Short option for the zip path: `-esp`.)
+
+Zip layout:
+
+* `metadata.json` — Hop metadata (run configs, connections, catalogs)
+* `hop-spark-package.properties` — marker (`projectRoot=project`)
+* `project/…` — project home files (see filters below)
+
+==== Package include / exclude filters
+
+Export does **not** read `.gitignore`. Built-in skipped directory basenames (case-insensitive):
+
+`work`, `datasets`, `target`, `.git`, `node_modules`, `.idea`, `.settings`
+
+Configure extras with **`spark-package.json`** in the project home (Tools → *Edit Native Spark package filters…* in Hop GUI, or edit the file by hand):
+
+[source,json]
+----
+{
+ "excludeDirs": [ "tmp", "screenshots" ],
+ "excludeGlobs": [ "**/*.jar", "**/*.zip" ],
+ "includePaths": [ "work/cluster-env.json" ],
+ "replaceDefaultExcludeDirs": false
+}
+----
+
+|===
+|Field |Meaning
+
+|`excludeDirs`
+|Extra directory basenames to skip (merged with built-ins unless replace is true)
+
+|`excludeGlobs`
+|Globs relative to project home (e.g. `**/*.jar`)
+
+|`includePaths`
+|Relative paths force-included even under a skipped dir
+
+|`replaceDefaultExcludeDirs`
+|If true, only the listed `excludeDirs` apply (built-ins off)
+|===
+
+**hop-conf** (merged on top of `spark-package.json`):
+
+[source,bash]
+----
+./hop-conf.sh --export-spark-project=/tmp/pkg.zip \
+ --export-spark-project-home=/path/to/project \
+ --export-spark-project-exclude-dirs=tmp,screenshots \
+ --export-spark-project-exclude-globs='**/*.jar' \
+ --export-spark-project-include=work/cluster-env.json
+----
+
+Databricks Deploy & run auto-export uses the same rules (reads `spark-package.json` when present).
+
+=== Run with MainSpark
+
+[source,bash]
+----
+"${SPARK_HOME}/bin/spark-submit" \
+ --class org.apache.hop.spark.run.MainSpark \
+ /tmp/hop-native-spark4-submit.jar \
+ --HopProjectPackage=/tmp/my-project-spark.zip \
+ --HopPipelinePath=pipelines/run-on-spark.hpl \
+ --HopRunConfigurationName=YourNativeSparkRunConfig \
+ [--HopConfigFile=/path/cluster-env.json]
+----
+
+**Databricks Deploy & run:** the xref:workflow/actions/databricks-job-run.adoc[Databricks Job Run] action can **upload** this package (export from `PROJECT_HOME` or an existing zip) and an optional environment config file to the Volume base path, then pass the same named MainSpark arguments on the JAR task. Enable **Upload Spark project package** on the action when the pipeline needs nested definitions.
+
+On Databricks, packages under **`/Volumes/…`** (or `/dbfs/…`) are treated as **cluster-shared**: Hop does **not** call `SparkContext.addFile` for them. Executors open the same Volume path (UC FUSE mount). That avoids failures such as `Failed to fetch spark://…/files/hop-spark-package.zip` / `Stream '/files/…' was not found` when the driver cannot serve SparkFiles to job-cluster workers.
+
+Flow:
+
+. Driver materializes the package under `${java.io.tmpdir}/hop-spark-pkg-*/` and sets `PROJECT_HOME`.
+. When the Native Spark session starts, the engine calls `SparkContext.addFile` so **every executor** receives the zip (`SparkFiles`).
+. Each mini-pipeline init (`HopMapPartitionsFn`) resolves the zip via `SparkFiles.get`, extracts once per worker JVM, and sets `PROJECT_HOME` for Hop VFS loads.
+
+Variables:
+
+* `HOP_SPARK_PROJECT_PACKAGE` — local path to the zip (after staging)
+* `HOP_SPARK_PROJECT_PACKAGE_SPARK_FILE` — basename for `SparkFiles.get`
+* `PROJECT_HOME` — extracted definition root on that JVM
+
+[IMPORTANT]
+====
+With a project package, **`PROJECT_HOME` is the extracted definition tree**, not your object-store data root.
+
+* Nested mappings: `$\{PROJECT_HOME\}/mappings/…` ✅
+* Spark File Input data: use `HOP_DATA`, `OUTPUT_ROOT`, or explicit `s3a://…` / `hdfs://…` — **not** `$\{PROJECT_HOME\}/files/…` unless those files are small resources inside the package and you accept local extract paths.
+
+The path scheme map (`s3=s3a`) is independent: it rewrites Dataset schemes only.
+====
+
+Limitations: Workflow Executor and dynamic Pipeline Executor filenames remain experimental. The package must be readable on the *driver* at submit time so `addFile` can stage it (local path or VFS URI Hop can open).
+
+=== Path scheme map (run configuration)
+
+The Native Spark pipeline run configuration includes an optional **Path scheme map** field: multi-line `from=to` entries applied **only** to *Spark File Input/Output* and *Lake Table PATH* URIs after variable resolution (not to classic Hop VFS / mapPartitions paths).
+
+Example:
+
+[source]
+----
+# Hop VFS / named schemes -> Spark/Hadoop schemes
+s3=s3a
+minio=s3a
+azure=abfs
+----
+
+* Tokens may be written as `s3` or `s3://` (normalized to bare scheme names).
+* Lines starting with `#` are comments.
+* Mapping only rewrites the scheme; it does **not** inject credentials or endpoints — still set `spark.hadoop.fs.s3a.*` (or other FS config) for the *target* scheme.
+* Prefer putting the target URI in pipeline variables when you can; use the map when the same pipeline must keep Hop-style schemes in metadata.
+
+== Cluster environment variables
+
+When you `spark-submit` with an environment configuration file, remap project variables to **Spark URIs**, not Hop-only VFS schemes:
+
+[source,json]
+----
+{
+ "variables": [
+ {
+ "name": "PROJECT_HOME",
+ "value": "s3a://hop-project/",
+ "description": "Project root on object storage (Hadoop S3A URI for executors)"
+ },
+ {
+ "name": "OUTPUT_ROOT",
+ "value": "s3a://hop-project/output/",
+ "description": "Shared output prefix visible to all executors"
+ }
+ ]
+}
+----
+
+See xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark engine] for the full submit flow.
+
+== Practical rules of thumb
+
+. **Spark File / Lake path fields** → use **Spark URIs** (`file:///`, `hdfs://`, `s3a://`, `abfs://`, …).
+. **Classic Hop file transforms** (even on the Spark engine) → use **Hop VFS URIs** and ensure plugins/credentials exist on executors.
+. Prefer *Spark File Output* for large distributed writes; classic Text File Output is for POCs and needs unique names per partition (`${Internal.Transform.ID}`).
+. Do not assume `s3://` means the same thing in Hop and Spark — in Hop docs for object storage on Spark, **`s3a://` is canonical**.
+
+== Related pages
+
+* xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark engine]
+* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables (Delta / Iceberg)]
+* xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native Spark pipeline engine]
+* xref:vfs.adoc[Virtual File Systems]
+* xref:metadata-types/minio-connection.adoc[Minio Connection]
+* xref:metadata-types/spark-catalog.adoc[Spark Catalog]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms.adoc
index bbb11f3fa9..5c2b495a40 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms.adoc
@@ -208,6 +208,10 @@ The pages nested under this topic contain information on how to use the transfor
* xref:pipeline/transforms/sort.adoc[Sort Rows]
* xref:pipeline/transforms/sortedmerge.adoc[Sorted Merge]
* xref:pipeline/transforms/sortedschemamerge.adoc[Sorted Schema Merge]
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark Lake Table Input]
+* xref:pipeline/transforms/spark-lake-table-maintenance.adoc[Spark Lake Table Maintenance]
+* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark Lake Table Merge]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark Lake Table Output]
* xref:pipeline/transforms/splitfields.adoc[Split Fields]
* xref:pipeline/transforms/splitfieldtorows.adoc[Split fields to rows]
* xref:pipeline/transforms/splunkinput.adoc[Splunk Input]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/abort.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/abort.adoc
index b445447c68..8909014d74 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/abort.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/abort.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Abort tranform aborts a running pipeline as soon as it receives input data. The main use case for this transform is to throw an error when an unexpected or unwanted situation occurs.
+:description: The Abort tranform aborts a running pipeline.
= image:transforms/icons/abort.svg[Abort Icon, role="image-doc-icon"] Abort
@@ -34,9 +34,11 @@ For example, you can use this transform so that a pipeline can be aborted after
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/accessoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/accessoutput.adoc
index ba01946829..da2214b151 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/accessoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/accessoutput.adoc
@@ -32,9 +32,11 @@ The Microsoft Access Output allows you to create a new Access database file as a
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addchecksum.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addchecksum.adoc
index 4ec4389eea..c65ed5e6b9 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addchecksum.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addchecksum.adoc
@@ -32,9 +32,11 @@ The Add a Checksum transform calculates checksums for one or more fields in the
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addconstant.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addconstant.adoc
index 6d3f536831..a9f0d2819c 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addconstant.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addconstant.adoc
@@ -31,9 +31,11 @@ The Add Constant Values transform is a simple and high performance way to add co
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addfieldschangesequence.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addfieldschangesequence.adoc
index a73e568898..e4981ff98c 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addfieldschangesequence.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addfieldschangesequence.adoc
@@ -31,9 +31,11 @@ The Add Fields Changing Sequence transform simply adds a sequence value which re
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addsequence.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addsequence.adoc
index 53f52252f5..41c86337b6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addsequence.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addsequence.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Add Sequence transform adds a sequence to the Hop stream. A sequence is an ever-changing integer value with a specific start and increment value. Sequences can be generated by Hop or retrieved from a database.
+:description: The Add Sequence transform adds a sequence to the Hop stream.
= image:transforms/icons/addsequence.svg[Add Sequence Icon, role="image-doc-icon"] Add Sequence
@@ -37,9 +37,11 @@ Also, they are not stored, so the values start back at the same value every time
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addxml.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addxml.adoc
index 3d36ef6218..3df3de478e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addxml.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addxml.adoc
@@ -32,9 +32,11 @@ This XML is added to the row in the form of a String field.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/analyticquery.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/analyticquery.adoc
index 2d4e8e2826..31463785dc 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/analyticquery.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/analyticquery.adoc
@@ -36,9 +36,11 @@ Examples of common use cases are:
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/apache-tika.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/apache-tika.adoc
index 6e2c8e0fe4..38dcfa2324 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/apache-tika.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/apache-tika.adoc
@@ -35,9 +35,11 @@ If you need specific pieces of information from this metadata, you can extract t
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/append.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/append.adoc
index 4b9fd82148..82baaf16d0 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/append.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/append.adoc
@@ -35,9 +35,11 @@ As always, the row layout for the input data coming from both transforms has to
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-decode.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-decode.adoc
index d0d899195f..ace9b35b18 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-decode.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-decode.adoc
@@ -31,9 +31,11 @@ The Avro Decode transform allows you to decode an Avro field and convert it to H
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-encode.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-encode.adoc
index 81058f44e9..859c2439d3 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-encode.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-encode.adoc
@@ -31,9 +31,11 @@ The Avro Encode transform allows you to encode a new Avro Record field using a s
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-file-input.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-file-input.adoc
index 5893f773e5..71dd5779c6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-file-input.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-file-input.adoc
@@ -33,9 +33,11 @@ Each record is encapsulated in an Avro field, each value has its own Schema and
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-file-output.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-file-output.adoc
index e75f136caa..10de2c9c69 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-file-output.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/avro-file-output.adoc
@@ -31,9 +31,11 @@ The Avro File Output transform can write Apache Avro messages to a file or field
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/aws-sns-notify.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/aws-sns-notify.adoc
index e90f8fa5f5..e02b1859c2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/aws-sns-notify.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/aws-sns-notify.adoc
@@ -31,9 +31,11 @@ The AWS SNS Notify transform enables you to send notifications from an Apache Ho
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Supported, 24]
-!Flink! image:question_mark.svg[Supported, 24]
-!Dataflow! image:question_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/aws-sqs-reader.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/aws-sqs-reader.adoc
index 1453f08f6e..e479b85591 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/aws-sqs-reader.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/aws-sqs-reader.adoc
@@ -31,9 +31,11 @@ The AWS SQS Reader transform enables you to receive messages from Amazon Web Ser
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Supported, 24]
-!Flink! image:question_mark.svg[Supported, 24]
-!Dataflow! image:question_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/azure-event-hubs-listener.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/azure-event-hubs-listener.adoc
index 8526768895..b21d7b3c99 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/azure-event-hubs-listener.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/azure-event-hubs-listener.adoc
@@ -31,9 +31,11 @@ The Azure Event Hubs Listener transform listens indefinitely to an Event Hub on
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/azure-event-hubs-writer.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/azure-event-hubs-writer.adoc
index 5cd411b759..c54184b6dc 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/azure-event-hubs-writer.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/azure-event-hubs-writer.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Azure Event Hubs Writer transform allows you to write messages (events) to a streaming service bus called Event Hubs on the Microsoft Azure cloud platform.
+:description: The Azure Event Hubs Writer transform allows you to write messages (events) to Event Hubs on the Microsoft Azure cloud platform.
= image:transforms/icons/azure.svg[Azure Event Hubs Writer Icon, role="image-doc-icon"] Azure Event Hubs Writer
@@ -31,9 +31,11 @@ The Azure Event Hubs Writer transform allows you to write messages (events) to a
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryinput.adoc
index 85909cb6eb..87ec43301c 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryinput.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The BigQuery Input transform reads data from Google Cloud BigQuery. It runs on the local Hop engine as well as every Beam engine (Direct, Dataflow, Spark, Flink).
+:description: The BigQuery Input transform reads data from Google Cloud BigQuery.
= image:transforms/icons/beam-bq-input.svg[BigQuery Input Icon, role="image-doc-icon"] BigQuery Input
@@ -31,9 +31,11 @@ The BigQuery Input transform reads data from a link:https://cloud.google.com/big
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryoutput.adoc
index bc1576b101..e6e7b9e48b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigqueryoutput.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The BigQuery Output transform writes data to a Google Cloud BigQuery table. It runs on the local Hop engine (via streaming inserts) as well as every Beam engine (Direct, Dataflow, Spark, Flink).
+:description: The BigQuery Output transform writes data to a Google Cloud BigQuery table.
= image:transforms/icons/beam-bq-output.svg[BigQuery Output Icon, role="image-doc-icon"] BigQuery Output
@@ -31,9 +31,11 @@ The BigQuery Output transform writes data to a link:https://cloud.google.com/big
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigtableinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigtableinput.adoc
index b63fc971a0..b0ec391d67 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigtableinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigtableinput.adoc
@@ -32,9 +32,11 @@ The Beam Bigtable Input transform can be used to input data from link:https://cl
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigtableoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigtableoutput.adoc
index d7c1c8889f..b5e21b712f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigtableoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beambigtableoutput.adoc
@@ -32,9 +32,11 @@ The Beam Bigtable Output transform can be used to write data to a link:https://c
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileinput.adoc
index 2283f0819b..58bb80918d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileinput.adoc
@@ -32,9 +32,11 @@ The Beam File Input transform reads files using a file definition with the Beam
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileoutput.adoc
index 391ac6d83b..4886502117 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamfileoutput.adoc
@@ -31,9 +31,11 @@ The Beam File Output transform writes files using a file definition with the Bea
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamgcppublisher.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamgcppublisher.adoc
index af4f85b8ed..f178e7ebaf 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamgcppublisher.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamgcppublisher.adoc
@@ -32,9 +32,11 @@ The Beam GCP Pub/Sub : Publish transform publishes messages to a link:https://cl
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamgcpsubscriber.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamgcpsubscriber.adoc
index 64408ce01e..de8fd470c6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamgcpsubscriber.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamgcpsubscriber.adoc
@@ -32,9 +32,11 @@ The Beam GCP Pub/Sub : Subscribe transform gets messages from a link:https://clo
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamhivecataloginput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamhivecataloginput.adoc
index 591c6969b1..a0b360349e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamhivecataloginput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamhivecataloginput.adoc
@@ -33,9 +33,11 @@ WARNING: This transform is in an experimental state, backwards compatibility bet
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkafkaconsume.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkafkaconsume.adoc
index dd2152fae5..c9ee46c235 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkafkaconsume.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkafkaconsume.adoc
@@ -32,9 +32,11 @@ The Beam Kafka Consume transform link:https://kafka.apache.org/23/javadoc/index.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkafkaproduce.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkafkaproduce.adoc
index 6ecf9e945a..b4e5b2d655 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkafkaproduce.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkafkaproduce.adoc
@@ -31,9 +31,11 @@ The Beam Kafka Produce transform link:https://kafka.apache.org/25/javadoc/index.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkinesisconsume.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkinesisconsume.adoc
index aa65bb1d2c..67fcd18eef 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkinesisconsume.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkinesisconsume.adoc
@@ -31,9 +31,11 @@ The Beam link:https://aws.amazon.com/kinesis/[Kinesis] Consume transform consume
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkinesisproduce.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkinesisproduce.adoc
index ae9189caf4..6bdb2e6bb6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkinesisproduce.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamkinesisproduce.adoc
@@ -31,9 +31,11 @@ The Beam link:https://aws.amazon.com/kinesis/[Kinesis] Produce transform sends d
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamtimestamp.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamtimestamp.adoc
index 585cd23581..8b2c5e7b0a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamtimestamp.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamtimestamp.adoc
@@ -31,9 +31,11 @@ The Beam Timestamp transform adds a custom timestamp using the Beam execution en
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamwindow.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamwindow.adoc
index 23ae9522b5..6129c3edcc 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamwindow.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/beamwindow.adoc
@@ -31,9 +31,11 @@ The Beam Window transform adds event-time-based window functions using the Beam
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:cross.svg[Not Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/blockingtransform.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/blockingtransform.adoc
index ee8ccb14ae..546e0c8733 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/blockingtransform.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/blockingtransform.adoc
@@ -37,9 +37,11 @@ Use the Blocking transform for triggering plugins, stored procedures, Java scrip
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/blockuntiltransformsfinish.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/blockuntiltransformsfinish.adoc
index af99b4ca1a..829773f477 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/blockuntiltransformsfinish.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/blockuntiltransformsfinish.adoc
@@ -34,9 +34,11 @@ You can use it to avoid the natural concurrency (parallelism) that exists betwee
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/calculator.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/calculator.adoc
index caa6125ebe..dd7de1ac04 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/calculator.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/calculator.adoc
@@ -37,12 +37,14 @@ TIP: The execution speed of the Calculator is far better than the speed provided
| == Supported Engines
-[%noheader,cols="2,1a",frame=none,role="table-supported-engines"]
+[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
-!Hop Engine! image:check_mark.svg[Supported,24]
-!Spark! image:check_mark.svg[Supported,24]
-!Flink! image:check_mark.svg[Supported,24]
-!Dataflow! image:check_mark.svg[Supported,24]
+!Hop Engine! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/calldbproc.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/calldbproc.adoc
index 8388babe0b..34388808b6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/calldbproc.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/calldbproc.adoc
@@ -34,9 +34,11 @@ Stored procedures and functions can only return values through their function ar
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cassandra-input.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cassandra-input.adoc
index 2a730ff4d2..bf20c34f5e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cassandra-input.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cassandra-input.adoc
@@ -34,9 +34,11 @@ The Cassandra Input transform reads data from a Cassandra table of an Apache Cas
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cassandra-output.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cassandra-output.adoc
index bc9737a703..72270d2abb 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cassandra-output.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cassandra-output.adoc
@@ -34,9 +34,11 @@ The Cassandra Output transform writes data to a Cassandra table of an Apache Cas
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/changefileencoding.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/changefileencoding.adoc
index c3f17e4b64..09949328a6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/changefileencoding.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/changefileencoding.adoc
@@ -32,9 +32,11 @@ The Change File Encoding transform changes a text file from one encoding to anot
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/checkfilelocked.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/checkfilelocked.adoc
index b917647351..c8f15d513e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/checkfilelocked.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/checkfilelocked.adoc
@@ -32,9 +32,11 @@ The Check If A File Is Locked transform tries to determine if a file is locked b
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/checkwebserviceavailable.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/checkwebserviceavailable.adoc
index 0b4e6d34d5..3de36000b2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/checkwebserviceavailable.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/checkwebserviceavailable.adoc
@@ -34,9 +34,11 @@ Further information of the failing reason can be found in the log when debug log
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/clonerow.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/clonerow.adoc
index c9f65ba46e..c4f1404418 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/clonerow.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/clonerow.adoc
@@ -32,9 +32,11 @@ The Clone Row transform creates copies (clones) of a row and outputs them direct
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/closure.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/closure.adoc
index b9ceb22861..16a73bd218 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/closure.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/closure.adoc
@@ -34,9 +34,11 @@ It attaches the distance (in levels) from parent to child.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/coalesce.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/coalesce.adoc
index 2c3010fee7..f271e64eab 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/coalesce.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/coalesce.adoc
@@ -32,9 +32,11 @@ The Coalesce transform lets you list multiple fields and returns the first non-n
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/columnexists.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/columnexists.adoc
index 026e44302f..c9c3c3dc09 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/columnexists.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/columnexists.adoc
@@ -32,9 +32,11 @@ The Column Exists transforms allows you to verify the existence of a specific co
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/combinationlookup.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/combinationlookup.adoc
index 70a733f013..51df52a924 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/combinationlookup.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/combinationlookup.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Combination Lookup-Update transform allows you to store information in a junk-dimension table. The transform can possibly also be used to maintain Kimball pure Type 1 dimensions.
+:description: The Combination Lookup-Update transform allows you to store information in a junk-dimension table.
= image:transforms/icons/combinationlookup.svg[Combination lookup/update transform Icon, role="image-doc-icon"] Combination lookup/update
@@ -49,9 +49,11 @@ This can speed up lookup performance dramatically while limiting the fields to i
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/concatfields.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/concatfields.adoc
index f68e890bfa..6b2a9ae7b5 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/concatfields.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/concatfields.adoc
@@ -32,9 +32,11 @@ The Concat Fields transform concatenates multiple fields into one target field.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/copyrowstoresult.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/copyrowstoresult.adoc
index 3e1de1325d..fbc1bfb07b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/copyrowstoresult.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/copyrowstoresult.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Copy Rows To Result transform allows you to transfer rows of data (in memory) to the next pipeline or workflow action via an internal result row set. Remember that values or variables will not be copied, only row data.
+:description: The Copy Rows To Result transform allows you to transfer rows of data (in memory) to the next pipeline or workflow action.
= image:transforms/icons/rowstoresult.svg[Copy rows to result transform Icon, role="image-doc-icon"] Copy rows to result
@@ -30,9 +30,11 @@ The Copy Rows To Result transform allows you to transfer rows of data (in memory
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cratedb-bulkloader.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cratedb-bulkloader.adoc
index db3167eedb..b58bbf0f2f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cratedb-bulkloader.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/cratedb-bulkloader.adoc
@@ -36,9 +36,11 @@ The CrateDB Bulk Loader transform loads data from Apache Hop to CrateDB using tw
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/creditcardvalidator.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/creditcardvalidator.adoc
index 44a399da15..683294ad1a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/creditcardvalidator.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/creditcardvalidator.adoc
@@ -35,9 +35,11 @@ The Credit Card Validator transform will help you check the following:
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/csvinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/csvinput.adoc
index 135f08e817..9cae15f2a6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/csvinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/csvinput.adoc
@@ -48,9 +48,11 @@ For information on valid date and numeric formats used in this transform, view t
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/data-stream-input.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/data-stream-input.adoc
index b6a291b3b0..b2efe91414 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/data-stream-input.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/data-stream-input.adoc
@@ -18,14 +18,29 @@ under the License.
:language: en_US
:description: The Data Stream Input transform reads from an IPC Data Stream.
-= Data Stream Input
-
-== Overview
+[%noheader,cols="3a,1a", role="table-no-borders" ]
+|===
+|
+== Description
The **Data Stream Input** transform reads rows from a configured xref:metadata-types/data-stream/data-stream.adoc[Data Stream] into a Hop pipeline.
It supports all available Data Stream implementations (Arrow File Stream, Arrow Random Access File, and Arrow Flight).
+|
+== Supported Engines
+[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
+!===
+!Hop Engine! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
+!===
+|===
+
+
== Options
[cols="1,3"]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/data-stream-output.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/data-stream-output.adoc
index aec06edefa..4fe11ab2dc 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/data-stream-output.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/data-stream-output.adoc
@@ -18,14 +18,28 @@ under the License.
:language: en_US
:description: The Data Stream Output transform writes to an IPC Data Stream.
-= Data Stream Output
-
-== Overview
+[%noheader,cols="3a,1a", role="table-no-borders" ]
+|===
+|
+== Description
The **Data Stream Output** transform writes rows from a Hop pipeline to a configured xref:/metadata-types/data-stream/data-stream.adoc[Data Stream].
It supports all available Data Stream implementations (Arrow File Stream, Arrow Random Access File, and Arrow Flight).
+|
+== Supported Engines
+[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
+!===
+!Hop Engine! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
+!===
+|===
+
== Options
[cols="1,3"]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/databasejoin.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/databasejoin.adoc
index f1b11039cc..ec4358b538 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/databasejoin.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/databasejoin.adoc
@@ -32,9 +32,11 @@ The Database Join transform allows you to run a query against a database using d
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/databaselookup.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/databaselookup.adoc
index 5f469aa6f4..123eedfb1a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/databaselookup.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/databaselookup.adoc
@@ -32,9 +32,11 @@ Lookup values are added as new fields onto the stream.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/datagrid.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/datagrid.adoc
index 7404864a7d..3244f607d2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/datagrid.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/datagrid.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Data Grid transform allows you to enter a static list of rows in a grid, similar to an in-pipeline spreadsheet. This is usually done for testing, reference or demo purposes.
+:description: The Data Grid transform allows you to enter a static list of rows in a grid.
= image:transforms/icons/datagrid.svg[Data Grid transform Icon, role="image-doc-icon"] Data Grid
@@ -35,9 +35,11 @@ Place a Data grid onto the data orchestration grid. On the Meta tab, add the fie
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/datasetinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/datasetinput.adoc
index e63c66ea31..54d1250f1e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/datasetinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/datasetinput.adoc
@@ -32,9 +32,11 @@ The Data Set Input transform allows you to read from a data set in a pipeline.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dbimpactinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dbimpactinput.adoc
index dc5ba55ec5..ed2553cfee 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dbimpactinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dbimpactinput.adoc
@@ -20,7 +20,7 @@ under the License.
:openvar: ${
:closevar: }
-= image:images/icons/database.svg[Database Impact Input, role="image-doc-icon"] Database Impact Input
+= image:transforms/icons/database.svg[Database Impact Input Icon, role="image-doc-icon"] Database Impact Input
[%noheader,cols="3a,1a", role="table-no-borders" ]
|===
@@ -37,9 +37,11 @@ IMPORTANT: This transform supports error handling. You can add an error handlin
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ddl.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ddl.adoc
index f4effbaab1..fc62b4354e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ddl.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ddl.adoc
@@ -32,9 +32,11 @@ This transform generates https://en.wikipedia.org/wiki/Data_definition_language[
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/delay.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/delay.adoc
index 1266dd74ed..31b188e7bb 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/delay.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/delay.adoc
@@ -34,9 +34,11 @@ Use this transform if you deliberately want to slow down your pipeline.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/delete.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/delete.adoc
index 76041efddf..ff8dd9ba41 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/delete.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/delete.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Delete transform deletes rows of data from a database. This transform is similar to the update family of transforms in that it takes one or more key fields to determine the rows to delete.
+:description: The Delete transform deletes rows of data from a database.
= image:transforms/icons/delete.svg[Delete transform Icon, role="image-doc-icon"] Delete
@@ -34,9 +34,11 @@ This transform is similar to the update family of transforms in that it takes on
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/detectemptystream.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/detectemptystream.adoc
index fa9352cb63..3068d02b74 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/detectemptystream.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/detectemptystream.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Detect Empty Stream transform outputs one single empty row of data if the input stream is empty (ie when input stream does not contain any row). The output row will have the same field layout as the input row, but all field values will be empty (null).
+:description: The Detect Empty Stream transform outputs one single empty row of data if the input stream is empty.
= image:transforms/icons/detectemptystream.svg[Detect Empty Stream transform Icon, role="image-doc-icon"] Detect Empty Stream
@@ -35,9 +35,11 @@ If the input stream is not empty it will not output anything.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/detectlanguage.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/detectlanguage.adoc
index 5d5d4226c6..bad28d8dc4 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/detectlanguage.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/detectlanguage.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Detect Empty Stream transform outputs one single empty row of data if the input stream is empty (ie when input stream does not contain any row). The output row will have the same field layout as the input row, but all field values will be empty (null).
+:description: The Detect Language transform examine text to identify the language.
= image:transforms/icons/detectlanguage.svg[Detect Language transform Icon, role="image-doc-icon"] Detect Language
@@ -34,9 +34,11 @@ https://github.com/pemistahl/lingua?tab=readme-ov-file#3-which-languages-are-sup
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dimensionlookup.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dimensionlookup.adoc
index 86b06f7d29..e12ad4386b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dimensionlookup.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dimensionlookup.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Dimension Lookup/Update transform allows you to implement Ralph Kimball's slowly changing dimension for both types: Type I (update) and Type II (insert) together with some additional functions.
+:description: The Dimension Lookup/Update transform allows you to implement Ralph Kimball's slowly changing dimensions.
= image:transforms/icons/dimensionlookup.svg[Dimension lookup/update transform Icon, role="image-doc-icon"] Dimension lookup/update
@@ -34,9 +34,11 @@ This transform can be used to populate a dimension table or to look up values in
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dorisbulkloader.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dorisbulkloader.adoc
index 5e48ba265d..dec9c0a519 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dorisbulkloader.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dorisbulkloader.adoc
@@ -36,9 +36,11 @@ The Doris Bulk Loader allows you to insert data into Apache Doris at high speed
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dummy.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dummy.adoc
index 81d0115516..91c250450a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dummy.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dummy.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Dummy transform passes data without making any modifications. It functions as a placeholder for testing purposes or a way to combine multiple streams with the same field layout.
+:description: The Dummy transform passes data without making any modifications.
= image:transforms/icons/dummy.svg[Dummy transform Icon, role="image-doc-icon"] Dummy (do nothing)
@@ -34,9 +34,11 @@ It functions as a placeholder for testing purposes or a way to combine multiple
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dynamicsqlrow.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dynamicsqlrow.adoc
index aab67f3c1d..65a621f535 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dynamicsqlrow.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dynamicsqlrow.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Dynamic SQL Row transform allows you to execute a SQL statement that is defined in a database field. The lookup values returned by the transform are added as new fields onto the stream.
+:description: The Dynamic SQL Row transform allows you to execute a SQL statement that is defined in a field.
:openvar: ${
:closevar: }
@@ -37,9 +37,11 @@ The lookup values returned by the transform are added as new fields onto the str
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/edi2xml.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/edi2xml.adoc
index 6e06cb3040..d642d51dfd 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/edi2xml.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/edi2xml.adoc
@@ -34,9 +34,11 @@ The XML text is more accessible and allows for selective data extraction using X
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/emailinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/emailinput.adoc
index b3509ca072..c840927521 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/emailinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/emailinput.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Email Messages Input transform allows you to retrieve messages and their attachments from a mail server using the POP3, IMAP or MBOX standard protocols.
+:description: The Email Messages Input transform allows you to retrieve messages and their attachments from a mail server.
= image:transforms/icons/mailinput.svg[Email Messages Input transform Icon, role="image-doc-icon"] Email Messages Input
@@ -32,9 +32,11 @@ The Email Messages Input transform allows you to retrieve messages and their att
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/enhancedjsonoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/enhancedjsonoutput.adoc
index 1768438f93..7e32cfb88b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/enhancedjsonoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/enhancedjsonoutput.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Enhanced JSON Output transform allows you to generate JSON blocks based on input transform values. Output JSON will be available as a Javascript array or Javascript object depending on transform settings.
+:description: The Enhanced JSON Output transform allows you to generate JSON blocks based on input transform values.
= image:transforms/icons/JSO.svg[Enhanced JSON Output transform Icon, role="image-doc-icon"] Enhanced JSON Output
@@ -36,9 +36,11 @@ TIP: Because this transform loops over the fields defined as Group Key and seria
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/excelinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/excelinput.adoc
index c1fcbdd243..08f3ee86ab 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/excelinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/excelinput.adoc
@@ -36,9 +36,11 @@ When you read other file types like OpenOffice ODS and using special functions l
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/excelwriter.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/excelwriter.adoc
index dc8d709a3e..9279861007 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/excelwriter.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/excelwriter.adoc
@@ -42,9 +42,11 @@ The `.xls` files use a binary format which is better suited for simple content,
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
@@ -53,9 +55,22 @@ The `.xls` files use a binary format which is better suited for simple content,
[options="header"]
|===
|Format|Extension|Backend|Notes
-|Excel 97–2003|`.xls`|POI|Sheet password protection supported
-|Excel 2007+|`.xlsx`|POI|Streaming mode for large files; no sheet password protection
-|OpenDocument Spreadsheet|`.ods`|ODFDOM|LibreOffice Calc compatible; see <<ods-limitations>> below
+
+|Excel 97–2003
+|`.xls`
+|POI
+|Sheet password protection supported
+
+|Excel 2007+
+|`.xlsx`
+|POI
+|Streaming mode for large files; no sheet password protection
+
+|OpenDocument Spreadsheet
+|`.ods`
+|ODFDOM
+|LibreOffice Calc compatible; see <<ods-limitations>> below
+
|===
== Options
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execinfo.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execinfo.adoc
index 7057f71cd7..00308063d7 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execinfo.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execinfo.adoc
@@ -35,13 +35,14 @@ This means that this transform always needs input-rows in order to produce outpu
First use Generate rows to create your fields and values to use (children: Boolean = true) and (limit: Integer = 200). Then use a first transform to get the ids (Operation: Get execution IDs), and a second connected transform to delete them (Operation: Delete execution). Optionally you can also delete by other fields such as date and ID. It can take a while to query the Execution Information, so keep an eye on the Duration column.
| == Supported Engines
-[%noheader,cols="2,1a",frame=none,role="table-supported-engines"]
-
+[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
-!Hop Engine! image:check_mark.svg[Supported,24]
-!Spark! image:check_mark.svg[Maybe Supported,24]
-!Flink! image:check_mark.svg[Maybe Supported,24]
-!Dataflow! image:check_mark.svg[Maybe Supported,24]
+!Hop Engine! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execprocess.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execprocess.adoc
index aec3d8939c..f5bec93653 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execprocess.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execprocess.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Execute A Process transform executes a shell script (on the host that runs the pipeline). This transform is similar to the Shell workflow action, but can be used in a pipeline to execute for every row.
+:description: The Execute A Process transform executes a shell script (on the host that runs the pipeline).
= image:transforms/icons/execprocess.svg[Execute a process transform Icon, role="image-doc-icon"] Execute a process
@@ -34,9 +34,11 @@ The transform is similar to the xref:workflow/actions/shell.adoc[Shell] workflow
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execsql.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execsql.adoc
index c4e7006c5f..f0664ffefa 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execsql.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execsql.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Execute SQL Script transform runs a SQL script either once, during the initialization phase of the pipeline, or once for every input-row that the transform receives and inbound fields are passed on to its output.
+:description: The Execute SQL Script transform runs a SQL script.
:openvar: ${
:closevar: }
@@ -38,9 +38,11 @@ Only in this mode, input fields are passed on to the next hop.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execsqlrow.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execsqlrow.adoc
index 12df0dd038..386cf4a2fc 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execsqlrow.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/execsqlrow.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Execute Row SQL Script transform executes an SQL script for every input row the transform receives. The SQL to execute can be passed as pipeline fields or read from a file.
+:description: The Execute Row SQL Script transform executes an SQL script for every input row the transform receives.
= image:transforms/icons/execsqlrow.svg[Execute row SQL script transform Icon, role="image-doc-icon"] Execute row SQL script
@@ -34,9 +34,11 @@ The SQL to execute can be passed as pipeline fields or read from a file.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/exectests.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/exectests.adoc
index 4ea5bd68ba..4eee9dbdc2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/exectests.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/exectests.adoc
@@ -32,9 +32,11 @@ The Execute Unit Tests transform fetches and executes the available xref:pipelin
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:question_mark.svg[Uncertain, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fake.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fake.adoc
index 7a4e978697..032e11cfa2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fake.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fake.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Fake Data tranform allows you to generate fake data using the Java Faker library. It can be used to generate pretty data for development, testing or showcasing a project.
+:description: The Fake Data tranform allows you to generate fake data using the Java Faker library.
= image:transforms/icons/fake.svg[Fake data transform Icon, role="image-doc-icon"] Fake data
@@ -36,9 +36,11 @@ For instance, we could generate some random Pokémon data.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fileexists.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fileexists.adoc
index 0214598fcb..df64de2205 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fileexists.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fileexists.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The File Exists transforms verifies the existence of a file. The filenames to check are read from pipeline fields. The transform outputs a boolean flag field, indicating whether a file exists or doesn't.
+:description: The File Exists transforms verifies the existence of a file.
= image:transforms/icons/fileexists.svg[File exists transform Icon, role="image-doc-icon"] File exists
@@ -34,9 +34,11 @@ The transform outputs a boolean flag field, indicating whether a file exists or
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filemetadata.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filemetadata.adoc
index 460302eba9..4641e677d5 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filemetadata.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filemetadata.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The File Metadata transform scans a file to determine its metadata structure or layout. Use this transforms in situations where you need to read a structured text file (e.g. CSV, TSV) when you don't know the exact layout in advance.
+:description: The File Metadata transform scans a file to determine its metadata structure or layout.
= image:transforms/icons/filemetadata.svg[File Metadata transform Icon, role="image-doc-icon"] File Metadata
@@ -41,9 +41,11 @@ Using 0 rows for 'limit scanned rows' is a way to make sure the entire file is s
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filesfromresult.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filesfromresult.adoc
index 398a1de470..329f1125ca 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filesfromresult.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filesfromresult.adoc
@@ -34,9 +34,11 @@ Every time a file gets processed, used or created in a pipeline or a workflow, t
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filestoresult.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filestoresult.adoc
index a169feae93..cb82c3ae29 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filestoresult.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filestoresult.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Files To Result transform allows you to set filenames in the internal result files of a pipeline, for use by subsequent workflow actions. Subsequent workflow actions can then use this information
+:description: The Files To Result transform allows you to set filenames in the internal result files of a pipeline, for use by subsequent workflow actions.
= image:transforms/icons/filestoresult.svg[Files to result transform Icon, role="image-doc-icon"] Files to result
@@ -40,9 +40,11 @@ WARNING: the Files To Result is available for historical reasons. Check the xref
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filterrows.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filterrows.adoc
index 24e5b27c05..488a04e987 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filterrows.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filterrows.adoc
@@ -33,9 +33,11 @@ The Filter Rows transform allows you to filter rows based on conditions and comp
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/formula.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/formula.adoc
index 15d23d9519..119e167d3b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/formula.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/formula.adoc
@@ -45,9 +45,11 @@ With a formula you are not limited to comparing to three fields like in the Calc
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fuzzymatch.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fuzzymatch.adoc
index 358b82bc7b..76f19e414b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fuzzymatch.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fuzzymatch.adoc
@@ -34,9 +34,11 @@ This transform returns matching values as a separated list as specified by user-
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/generaterandomvalue.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/generaterandomvalue.adoc
index fe3594c6ce..efc1d1fa7e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/generaterandomvalue.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/generaterandomvalue.adoc
@@ -32,9 +32,11 @@ The "Generate Random Value" transform generates random numbers, integers, string
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getdatafromxml.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getdatafromxml.adoc
index e0252d528b..39e73f1ea0 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getdatafromxml.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getdatafromxml.adoc
@@ -42,9 +42,11 @@ Samples (Samples project):
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
@@ -80,33 +82,62 @@ If a file is required and it is not found, an error is generated;otherwise, the
[options="header", cols="1,3"]
|===
|Option|Description
-2+|Settings
-|Loop XPath| For every matching entry in the XML file(s) or data, one row of data is added to the output.
+2+|Settings
+
+|Loop XPath
+| For every matching entry in the XML file(s) or data, one row of data is added to the output.
This is the main specification used to flatten the XML file(s).
You can use the "Get XPath nodes" button to search for the possible repeating nodes in the XML document, however, if the XML document is large, this can take a while.
-|Encoding|The XML filename encoding, in case none is specified in the XML documents.
-|Namespace aware?|Enable if the XML document requires namespaces to be considered while parsing.
-|Ignore comments?|Enable to ignore all comments in the XML document while parsing.
-|Validate XML?|Enable to validate the XML prior to parsing.
+
+|Encoding
+|The XML filename encoding, in case none is specified in the XML documents.
+
+|Namespace aware?
+|Enable if the XML document requires namespaces to be considered while parsing.
+
+|Ignore comments?
+|Enable to ignore all comments in the XML document while parsing.
+
+|Validate XML?
+|Enable to validate the XML prior to parsing.
Use a token when you want to replace dynamically in a Xpath field value.
A token is between @_ and - (@_fieldname-).
Please see the Example 1 to see how it works.
-|Use token|Enable to use a token for validating the XML.
-|Ignore empty file|Enable to ignore any files with no content. These are not valid XML documents.
-|Do not raise an error if no files|Enable to do nothing if no files are found. Otherwise, an error is returned.
-|Limit|Specify a maximum number of rows to return. Zero (0) returns all rows.
-|Prune path to handle large files|Specifies a path, similar to the Loop XPath, used to process chunks of data from the XML file. Each matching value defines a chunk of data that is read and processed. Use the prune path to speed up processing of large files.
+
+|Use token
+|Enable to use a token for validating the XML.
+
+|Ignore empty file
+|Enable to ignore any files with no content. These are not valid XML documents.
+
+|Do not raise an error if no files
+|Enable to do nothing if no files are found. Otherwise, an error is returned.
+
+|Limit
+|Specify a maximum number of rows to return. Zero (0) returns all rows.
+
+|Prune path to handle large files
+|Specifies a path, similar to the Loop XPath, used to process chunks of data from the XML file. Each matching value defines a chunk of data that is read and processed. Use the prune path to speed up processing of large files.
You can also use this parameter to avoid multiple HTTP URL requests.
You can also do this using the xref:pipeline/transforms/xmlinputstream.adoc[XML Input Stream (StAX)] transform.
2+|Additional fields
-|Include filename in output?|Allows you to specify a field name to include the file name (String) in the output of this transform.
-|Filename fieldname|The field to read the file name value from.
-|Rownum in output?|Allows you to specify a field name to include the row number (Integer) in the output of this transform.
-|Rownum fieldname|The field to read the row number value from.
+|Include filename in output?
+|Allows you to specify a field name to include the file name (String) in the output of this transform.
+
+|Filename fieldname
+|The field to read the file name value from.
+
+|Rownum in output?
+|Allows you to specify a field name to include the row number (Integer) in the output of this transform.
+
+|Rownum fieldname
+|The field to read the row number value from.
+
+2+|Add to result filename
-2+|Add to result filename|
-|Add files to result filename|Adds the XML filenames read to the result of this pipeline. A unique list is being kept in memory that can be used in the next workflow action in a workflow, for example in another pipeline.
+|Add files to result filename
+|Adds the XML filenames read to the result of this pipeline. A unique list is being kept in memory that can be used in the next workflow action in a workflow, for example in another pipeline.
|===
@@ -136,7 +167,7 @@ You can also do this using the xref:pipeline/transforms/xmlinputstream.adoc[XML
|Select fields from snippet|Click to populate the table with fields corresponding to a Loop Xpath and an XML document that must be provided in the popup dialog.
|===
-===Additional output fields tab
+=== Additional output fields tab
[options="header", cols="1,3"]
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getfilenames.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getfilenames.adoc
index 254ab9eb11..829d596d5a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getfilenames.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getfilenames.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Get File Names transform allows you to get information associated with file names on the file system. The information about the retrieved files is added as rows onto the stream.
+:description: The Get File Names transform allows you to get information associated with file names on the file system.
= image:transforms/icons/getfilenames.svg[Get filenames transform Icon, role="image-doc-icon"] Get filenames
@@ -32,12 +32,14 @@ The information about the retrieved files is added as rows onto the stream.
| == Supported Engines
-[%noheader,cols="2,1a",frame=none,role="table-supported-engines"]
+[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
-!Hop Engine! image:check_mark.svg[Supported,24]
-!Spark! image:check_mark.svg[Supported,24]
-!Flink! image:check_mark.svg[Supported,24]
-!Dataflow! image:check_mark.svg[Supported,24]
+!Hop Engine! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getfilesrowcount.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getfilesrowcount.adoc
index 90826bd501..c515bd24af 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getfilesrowcount.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getfilesrowcount.adoc
@@ -32,9 +32,11 @@ The Get Files Row Count transform counts the number of rows in a file or set of
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getrecordsfromstream.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getrecordsfromstream.adoc
index 73de6c24b4..395228758b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getrecordsfromstream.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getrecordsfromstream.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Get Records From Stream transform returns records that were previously generated by another pipeline in a workflow. The records were passed to this transform using either the Copy rows to result transform or the Workflow Executor transform.
+:description: The Get Records From Stream transform returns records that were previously generated by another pipeline in a workflow.
= image:transforms/icons/recordsfromstream.svg[Get records from stream transform Icon, role="image-doc-icon"] Get records from stream
@@ -38,9 +38,11 @@ NOTE: Deprecated use RowsFromResult instead
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getrowsfromresult.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getrowsfromresult.adoc
index afafb225d4..ee4fe8ed88 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getrowsfromresult.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getrowsfromresult.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Get Rows from Result transform returns rows that were previously generated by another pipeline in a workflow. The rows were passed on to this transform using either the Copy rows to result transform or the Pipeline Executor transform.
+:description: The Get Rows from Result transform returns rows that were previously generated by another pipeline in a workflow.
= image:transforms/icons/rowsfromresult.svg[Get Rows from Result transform Icon, role="image-doc-icon"] Get Rows from Result
@@ -31,9 +31,11 @@ The Get Rows from Result transform returns rows that were previously generated b
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getsubfolders.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getsubfolders.adoc
index 56683f40a5..3ea77fa946 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getsubfolders.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getsubfolders.adoc
@@ -35,9 +35,11 @@ Each subfolder produces one output row with metadata fields describing the folde
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getsystemdata.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getsystemdata.adoc
index d8b65edfe1..6d8dd2db30 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getsystemdata.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getsystemdata.adoc
@@ -37,9 +37,11 @@ It can also accept any number of input streams, aggregate any fields defined by
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/gettablenames.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/gettablenames.adoc
index 69919de935..318cecc90e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/gettablenames.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/gettablenames.adoc
@@ -31,9 +31,11 @@ The Get Table Names transform gets table names from a database connection.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getvariable.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getvariable.adoc
index 8c951b3eab..39098ae21e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getvariable.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getvariable.adoc
@@ -34,9 +34,11 @@ TIP: If you need to refer to a previous pipeline’s data row(s) fields, then u
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-analytics.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-analytics.adoc
index 2f8871ab73..30108fe057 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-analytics.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-analytics.adoc
@@ -34,9 +34,11 @@ The https://ga-dev-tools.google/ga4/query-explorer/[GA4 Query Explorer] provides
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-sheets-input.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-sheets-input.adoc
index 8a7532468c..5275372556 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-sheets-input.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-sheets-input.adoc
@@ -34,9 +34,11 @@ This transform requires a Google service account (JSON file) and a Google Cloud
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-sheets-output.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-sheets-output.adoc
index 453d716989..f5bff3ccc7 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-sheets-output.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/google-sheets-output.adoc
@@ -34,9 +34,11 @@ This transform requires a Google service account (JSON file) and a Google Cloud
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/groupby.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/groupby.adoc
index 2b139e181b..b7e2e83b59 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/groupby.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/groupby.adoc
@@ -44,9 +44,11 @@ You can use the xref:pipeline/transforms/memgroupby.adoc[Memory Group By] transf
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/html2text.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/html2text.adoc
index b41dc7df78..55681cad9f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/html2text.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/html2text.adoc
@@ -31,8 +31,10 @@ The HTML 2 Text Transform allows you to parse an HTML page and convert the conte
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/http.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/http.adoc
index 491b627ce4..552e66ac63 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/http.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/http.adoc
@@ -38,9 +38,11 @@ The result is stored in a String field with the specified name.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/httppost.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/httppost.adoc
index 05280d4b81..19afcba56c 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/httppost.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/httppost.adoc
@@ -32,9 +32,11 @@ The HTTP Post transform uses an HTTP POST command to submit form data via a URL.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/identifylastrow.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/identifylastrow.adoc
index 736ee42ac5..4f76ba0ac4 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/identifylastrow.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/identifylastrow.adoc
@@ -32,9 +32,11 @@ The Identify Last Row In A Stream pipeline transform generates a Boolean field f
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ifnull.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ifnull.adoc
index 4a8c3fd63b..fa2996d1ea 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ifnull.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ifnull.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The If Null transform replaces nulls by a given value either by processing the complete row with all fields, by processing the complete row but only for specific field types (Number, String, Date etc.) or by processing the complete row but only for specific fields by name
+:description: The If Null transform replaces nulls by a given value .
= image:transforms/icons/ifnull.svg[If Null transform Icon, role="image-doc-icon"] If Null
@@ -37,8 +37,10 @@ The If Null transform replaces nulls by a given value either by:
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/injector.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/injector.adoc
index 93dd1a9de9..27b0256884 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/injector.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/injector.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Injector transform is used for special purpose pipelines where you want to 'inject' rows into the pipeline using the Hop API and Java, or from streaming input transforms like the Kafka Consumer transform.
+:description: The Injector transform is used for special purpose pipelines where you want to 'inject' rows into a pipeline.
= image:transforms/icons/injector.svg[Injector transform Icon, role="image-doc-icon"] Injector
@@ -34,9 +34,11 @@ Among other things you can build 'headless' pipelines with it: pipelines that ha
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/insertupdate.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/insertupdate.adoc
index c206798e1f..1da68aac89 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/insertupdate.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/insertupdate.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Insert/Update transform first looks up a row in a table using one or more lookup keys. If a table row for the specified keys can't be found, a new row is inserted.
+:description: The Insert/Update transform implements database upsert/merge.
= image:transforms/icons/insertupdate.svg[Insert / Update transform Icon, role="image-doc-icon"] Insert / Update
@@ -36,9 +36,11 @@ If they are not all the same, the row in the table is updated.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/javafilter.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/javafilter.adoc
index 6ac57c5c95..08b5fed414 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/javafilter.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/javafilter.adoc
@@ -44,9 +44,11 @@ else
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/javascript.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/javascript.adoc
index 54516c718e..18dc9564cf 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/javascript.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/javascript.adoc
@@ -35,9 +35,11 @@ TIP: Variables won’t be usable or testable until you create fields. Click the
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jdbcmetadata.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jdbcmetadata.adoc
index a53793fd17..edcb12d196 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jdbcmetadata.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jdbcmetadata.adoc
@@ -33,9 +33,11 @@ The JDBC Metadata transform allows you to fetch metadata from a database connect
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/joinrows.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/joinrows.adoc
index b850a07368..9f9dfcf3c0 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/joinrows.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/joinrows.adoc
@@ -31,9 +31,11 @@ The Join Rows (cartesian product) transform allows you to combine/join multiple
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsoninput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsoninput.adoc
index f0202cb5d8..728cb6aa03 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsoninput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsoninput.adoc
@@ -54,9 +54,11 @@ Check the `json-input-nested-elements.hpl` sample in the Samples project for an
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
@@ -199,7 +201,9 @@ The Additional output fields tab contains the following options to specify addit
While processing input JSON files, if a JSON record has one or more field containing null values, by default the null values will be present in the transform output.
For example if we have a JSON file like this
-```json
+
+[source,json]
+----
{
"persons" : [
{
@@ -220,7 +224,7 @@ For example if we have a JSON file like this
}
]
}
-```
+----
When extracting the fields id and Name using the following field definition:
[options="header", cols="1a, 3a"]
@@ -232,30 +236,33 @@ When extracting the fields id and Name using the following field definition:
given the default behavior, the output will be
-```
+[source]
+----
id;Name
1;Name 1
2;Name 2
3;null
4;Name 4
-```
+----
Now let's only select the `name` field and see what happens
-```
+[source]
+----
Name
Name 1
Name 2
Name 4
-```
+----
You will notice that you only have 3 rowsets returned in this case ( the null line is omited from the result)
To change Hop's behavior regarding null values in JSON files, so that null values will not be considered in JSON output, you change the `HOP_JSON_INCLUDE_NULLS` configuration variable and set it's value to N
-```
+[source]
+----
HOP_JSON_INPUT_INCLUDE_NULLS = N
-```
+----
After restaring Hop, when we run the pipeline once again you will have 3 rows resulting because the the null values will be omitted.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsonnormalizeinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsonnormalizeinput.adoc
index 98a6060468..e65b45e457 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsonnormalizeinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsonnormalizeinput.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The JSON Normalize Input transform reads JSON (files, URLs, or fields), selects an array of records with JsonPath, flattens nested objects into columns, and outputs one row per array element.
+:description: The JSON Normalize Input transform.
:openvar: ${
:closevar: }
@@ -82,7 +82,11 @@ Pipeline unit tests in `integration-tests/json/` (`0008`–`0011`) exercise thes
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Apache Spark, Flink, Dataflow! Not verified for this transform; validate in your environment before relying on distributed execution.
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsonoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsonoutput.adoc
index cdb9f423e2..fa790c2a53 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsonoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsonoutput.adoc
@@ -34,9 +34,11 @@ Output JSON will be available as a javascript array or a javascript object depen
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/kafkaconsumer.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/kafkaconsumer.adoc
index a388360e51..828413dc06 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/kafkaconsumer.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/kafkaconsumer.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Kafka Consumer transform pulls data from Kafka. It runs a sub-pipeline that executes according to message batch size or duration, and can optionally stop when idle.
+:description: The Kafka Consumer transform pulls data from a Kafka topic.
= image:transforms/icons/KafkaConsumerInput.svg[Kafka Consumer transform Icon, role="image-doc-icon"] Kafka Consumer
@@ -51,9 +51,11 @@ For example, you can run the parent pipeline on a timed schedule, drain a topic
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/kafkaproducer.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/kafkaproducer.adoc
index e1be10a71d..fd62a55fd9 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/kafkaproducer.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/kafkaproducer.adoc
@@ -34,9 +34,11 @@ A Kafka Producer transform publishes a stream of records to one Kafka topic.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/languagemodelchat.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/languagemodelchat.adoc
index a6b49ce7be..def3268275 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/languagemodelchat.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/languagemodelchat.adoc
@@ -31,12 +31,14 @@ Additionally, the transform supports the application of some prompt engineering
| == Supported Engines
-[%noheader,cols="2,1a",frame=none,role="table-supported-engines"]
+[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
-!Hop Engine! image:check_mark.svg[Supported,24]
-!Spark! image:question_mark.svg[Maybe Supported,24]
-!Flink! image:question_mark.svg[Maybe Supported,24]
-!Dataflow! image:question_mark.svg[Maybe Supported,24]
+!Hop Engine! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ldapinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ldapinput.adoc
index 81e45af0b0..ece712583b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ldapinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ldapinput.adoc
@@ -34,9 +34,11 @@ The following sections describe the available options for the LDAP input transfo
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ldapoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ldapoutput.adoc
index 268dd90067..fe77533783 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ldapoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/ldapoutput.adoc
@@ -34,9 +34,11 @@ The following sections describe the available options for the LDAP Output transf
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/loadfileinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/loadfileinput.adoc
index 9e75f36b20..badc41e223 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/loadfileinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/loadfileinput.adoc
@@ -32,9 +32,11 @@ The Load File Content In Memory transform loads the content of a file in memory.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mail.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mail.adoc
index 5904b55a69..2cdb367a2d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mail.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mail.adoc
@@ -32,9 +32,11 @@ The Mail transform uses an SMTP server to send an email using data from the pipe
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mapping-input.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mapping-input.adoc
index 4e20691214..a05ee8f487 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mapping-input.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mapping-input.adoc
@@ -34,9 +34,11 @@ The data of this input will be provided by the parent pipeline (the pipeline tha
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mapping-output.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mapping-output.adoc
index af59fd12dc..f4338b814e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mapping-output.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mapping-output.adoc
@@ -34,8 +34,10 @@ The data entering this transform will be streamed to the parent pipeline.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/memgroupby.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/memgroupby.adoc
index 515bf64add..df782843f9 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/memgroupby.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/memgroupby.adoc
@@ -37,9 +37,11 @@ TIP: When the number of rows is too large to fit into memory, use a combination
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mergejoin.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mergejoin.adoc
index 95ede4e4e6..76350240e9 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mergejoin.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mergejoin.adoc
@@ -36,9 +36,11 @@ Join options include INNER, LEFT OUTER, RIGHT OUTER, and FULL OUTER.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mergerows.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mergerows.adoc
index 7dd38f3390..ff90515e2f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mergerows.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mergerows.adoc
@@ -32,9 +32,11 @@ The Merge Rows (Diff) transform compares and merges rows between two data stream
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metadata-input.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metadata-input.adoc
index 29775ff451..c8a20d9638 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metadata-input.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metadata-input.adoc
@@ -34,9 +34,11 @@ It outputs all the metadata objects of all types unless one or more types are fi
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc
index 71545f44a3..f54a262e10 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc
@@ -36,9 +36,11 @@ Metadata injection allows Apache Hop users to provide the required metadata for
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metastructure.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metastructure.adoc
index bf69315681..84c6459a1d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metastructure.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metastructure.adoc
@@ -34,9 +34,11 @@ Before producing this output the transform reads and discards (or eats) all inpu
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/monetdbbulkloader.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/monetdbbulkloader.adoc
index f2747fd080..0caaa6dd31 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/monetdbbulkloader.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/monetdbbulkloader.adoc
@@ -32,9 +32,11 @@ The MonetDB Bulk Loader transform bulk loads data to MonetDB. This significantly
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodbdelete.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodbdelete.adoc
index 006a24e1ae..b7d5ae0061 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodbdelete.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodbdelete.adoc
@@ -34,9 +34,11 @@ For additional information about MongoDB, see the MongoDB http://www.mongodb.org
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodbinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodbinput.adoc
index 3a9d19e296..5482dd4678 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodbinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodbinput.adoc
@@ -34,9 +34,11 @@ For additional information about MongoDB, see the MongoDB http://www.mongodb.org
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodboutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodboutput.adoc
index 2ace28596b..9575b5e616 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodboutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mongodboutput.adoc
@@ -34,9 +34,11 @@ For additional information about MongoDB, see the MongoDB http://www.mongodb.org
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/multimerge.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/multimerge.adoc
index 7990bc6c24..747f651243 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/multimerge.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/multimerge.adoc
@@ -32,9 +32,11 @@ The Multiway Merge Join transform joins input data from multiple streams.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mysqlbulkloader.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mysqlbulkloader.adoc
index 04a94f6f96..82e577088c 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mysqlbulkloader.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/mysqlbulkloader.adoc
@@ -33,9 +33,11 @@ It will create a local file which will then be loaded using the `LOAD DATA` comm
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc
index 8b95068b86..2247932b3d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc
@@ -38,9 +38,11 @@ Both reading, writing or both are supported. You can also call procedures and ge
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-gencsv.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-gencsv.adoc
index ab264bb05d..06501575db 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-gencsv.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-gencsv.adoc
@@ -32,9 +32,11 @@ The Neo4j Generate CSV transform writes files for nodes and relationships to the
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-getloginfo.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-getloginfo.adoc
index 304c930233..22556b262f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-getloginfo.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-getloginfo.adoc
@@ -35,9 +35,11 @@ Check the xref:hop-gui/perspective-neo4j.adoc[Neo4j perspective] for more detail
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-graphoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-graphoutput.adoc
index 10ff9b0f67..b1d38bb430 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-graphoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-graphoutput.adoc
@@ -35,9 +35,11 @@ These will not merge on these nodes, nor will it create or update relationships
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-import.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-import.adoc
index a7f87a5888..c550d84555 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-import.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-import.adoc
@@ -44,9 +44,11 @@ when configured appropriately.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-output.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-output.adoc
index b05342151b..66c6edf750 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-output.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-output.adoc
@@ -34,9 +34,11 @@ The transform generates the required Cypher statements with accompanying paramet
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-split-graph.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-split-graph.adoc
index 7083aa03d2..f725f4721b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-split-graph.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-split-graph.adoc
@@ -32,9 +32,11 @@ The Neo4j Split Graph transforms splits the nodes and relationships of a graph d
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/nullif.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/nullif.adoc
index 0ea2118522..d78e5c00fe 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/nullif.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/nullif.adoc
@@ -35,8 +35,10 @@ You can add all fields from the input stream(s) using Get Fields.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/numberrange.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/numberrange.adoc
index dda29d7c0c..9fca7790d3 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/numberrange.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/numberrange.adoc
@@ -33,9 +33,11 @@ The Number Range transform creates groups numerical values into a number of pred
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/odata-input.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/odata-input.adoc
index 55783c0c7e..3c5eae164d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/odata-input.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/odata-input.adoc
@@ -34,9 +34,11 @@ This transform performs GET requests against OData Entity Sets, maps JSON proper
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/orabulkloader.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/orabulkloader.adoc
index 01c300e82b..64c3be90a6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/orabulkloader.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/orabulkloader.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Oracle Bulk Loader transform streams data from Hop to Oracle. It will write the data it receives to a proper load format and will then invoke Oracle SQL*Loader to transfer it to the specified table.
+:description: The Oracle Bulk Loader transform streams data from Hop to Oracle.
= image:transforms/icons/orabulkloader.svg[Oracle Bulk Loader transform Icon, role="image-doc-icon"] Oracle Bulk Loader
@@ -32,9 +32,11 @@ This transform allows you to bulk load data to an Oracle database. It will write
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-input.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-input.adoc
index d0b85a0a7c..e33657f69b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-input.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-input.adoc
@@ -34,9 +34,11 @@ For more information on this see: http://parquet.apache.org/[Apache Parquet].
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc
index 2d8aaac15c..2850e76854 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc
@@ -34,9 +34,11 @@ For more information on this see: http://parquet.apache.org/[Apache Parquet].
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pgpdecryptstream.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pgpdecryptstream.adoc
index 1c6d5c09b1..04ec0fa76d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pgpdecryptstream.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pgpdecryptstream.adoc
@@ -32,9 +32,11 @@ The PGP Decrypt Stream transform decrypts PGP encrypted text.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pgpencryptstream.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pgpencryptstream.adoc
index 7167878422..c35ca35132 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pgpencryptstream.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pgpencryptstream.adoc
@@ -32,9 +32,11 @@ The PGP Encrypt Stream transform encrypts text using PGP.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-data-probe.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-data-probe.adoc
index 3e84a2d8b1..c51b242dd5 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-data-probe.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-data-probe.adoc
@@ -34,9 +34,11 @@ Check the xref:metadata-types/pipeline-probe.adoc[Pipeline Probe] metadata type
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc
index 98498b79dd..e81c0b66ec 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Pipeline Executor transform allows you to execute a Hop pipeline from within a pipeline. It is similar to the Workflow Executor transform, but works with pipelines.
+:description: The Pipeline Executor transform allows you to execute a Hop pipeline from within a pipeline.
= image:transforms/icons/pipelineexecutor.svg[Pipeline Executor transform Icon, role="image-doc-icon"] Pipeline Executor
@@ -35,9 +35,11 @@ It is similar to the Workflow Executor transform, but works with pipelines.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-logging.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-logging.adoc
index 22e197aaee..f8455c34ea 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-logging.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-logging.adoc
@@ -37,9 +37,11 @@ The transform requires very little configuration, but provides a lot of informat
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/postgresbulkloader.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/postgresbulkloader.adoc
index d6eb17e449..517c665d49 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/postgresbulkloader.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/postgresbulkloader.adoc
@@ -34,9 +34,11 @@ TIP: boolean stream fields are serialized as `t` or `f`, which PostgreSQL `COPY`
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/processfiles.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/processfiles.adoc
index 823b98a3b0..c79c564d4d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/processfiles.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/processfiles.adoc
@@ -32,9 +32,11 @@ The Process Files transform copies, moves or deletes files by giving the source
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/propertyinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/propertyinput.adoc
index 1c80c1edb9..690ecce393 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/propertyinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/propertyinput.adoc
@@ -34,9 +34,11 @@ TIP: More information on this file format is available on https://en.wikipedia.o
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
@@ -47,19 +49,42 @@ TIP: More information on this file format is available on https://en.wikipedia.o
[options="header"]
|===
|Option|Description
-|Transform name|Name of the transform. This name has to be unique in a single pipeline.
-|Filename is defined in a field?|The file name to read is available in a field in the input stream.
-|Get filename from field|Specify the field to read the file name from.
-|File or directory|Specifies the location and/or name of the input text file.
-|Add|Click to add the file/directory/wildcard combination to the list of selected files (grid) below.|
-|Browse|Click to select the file to load from a file browser.
-|Regular expression|Specifies the regular expression you want to use to select the files in the directory specified in the previous option.
-|Exclude Regular|Specifies a regular expression to exclude files from the specified directory.
-|Selected Files|Contains a list of selected files (or wildcard selections) and a property specifying if that file is required or not.
+
+|Transform name
+|Name of the transform. This name has to be unique in a single pipeline.
+
+|Filename is defined in a field?
+|The file name to read is available in a field in the input stream.
+
+|Get filename from field
+|Specify the field to read the file name from.
+
+|File or directory
+|Specifies the location and/or name of the input text file.
+|Add|Click to add the file/directory/wildcard combination to the list of selected files (grid) below.
+
+|Browse
+|Click to select the file to load from a file browser.
+
+|Regular expression
+|Specifies the regular expression you want to use to select the files in the directory specified in the previous option.
+
+|Exclude Regular
+|Specifies a regular expression to exclude files from the specified directory.
+
+|Selected Files
+|Contains a list of selected files (or wildcard selections) and a property specifying if that file is required or not.
If a file is required and it is not found, an error is generated, otherwise, the file name is skipped.
-|Delete|Click to remove the selected file or wildcard selection from the list.
-|Edit|Click to modiy the selected file or wildcard.
-|Show filenames(s)...|Displays a list of all files that will be loaded based on the current selected file definitions
+
+|Delete
+|Click to remove the selected file or wildcard selection from the list.
+
+|Edit
+|Click to modiy the selected file or wildcard.
+
+|Show filenames(s)...
+|Displays a list of all files that will be loaded based on the current selected file definitions
+
|===
=== Content Tab
@@ -75,7 +100,7 @@ If a file is required and it is not found, an error is generated, otherwise, the
|Include filename in output?|Allows you to specify a field name to include the file name in the output of this transform.
|Rownum in output|Allows you to specify a field name to include the row number (Integer) in the output of this transform.
|Reset rownum per file?|Enable to reset the generated row number to 1 at the start of every individual file. Otherwise, the rownum counts all rows in all files.
-|Include section name in output?|Allows you specify a field name to include the section namemin the output of this transform.
+|Include section name in output?|Allows you specify a field name to include the section name in the output of this transform.
|Add filename to result?|Adds the names of the files read to the result of this pipeline.
A unique list is kept in memory that can be used in the next workflow action in a workflow, for example in another pipeline.
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/propertyoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/propertyoutput.adoc
index 8ebe2313ee..9763d71ba2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/propertyoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/propertyoutput.adoc
@@ -36,9 +36,11 @@ TIP: More information on this file format is available on https://en.wikipedia.o
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/redshift-bulkloader.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/redshift-bulkloader.adoc
index 3e2b9b7e8a..8f0c7ff1d0 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/redshift-bulkloader.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/redshift-bulkloader.adoc
@@ -34,9 +34,11 @@ TIP: make sure your target Redshift table has a layout that is compatible with P
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/regexeval.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/regexeval.adoc
index bfa2d78d74..672a0f8c00 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/regexeval.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/regexeval.adoc
@@ -38,9 +38,11 @@ You can use this transform to parse a complex string of text and create new fiel
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/repeatfields.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/repeatfields.adoc
index eb7b634890..c66e2547ad 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/repeatfields.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/repeatfields.adoc
@@ -49,9 +49,11 @@ parallel.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/replacestring.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/replacestring.adoc
index 8954b1c35f..44379a04d2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/replacestring.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/replacestring.adoc
@@ -39,9 +39,11 @@ You can also use xref::variables.adoc#_hexadecimal_values[hexadecimal replacemen
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/reservoirsampling.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/reservoirsampling.adoc
index caf2e3ff93..6d2feae988 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/reservoirsampling.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/reservoirsampling.adoc
@@ -38,9 +38,11 @@ The reservoir sampling transform uses link:https://en.wikipedia.org/wiki/Reservo
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rest.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rest.adoc
index 35c7a04765..583050091d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rest.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rest.adoc
@@ -45,9 +45,11 @@ The REST client Transform returns a "result" field (can change the name), and th
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowdenormaliser.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowdenormaliser.adoc
index 02da976ece..461f25aef7 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowdenormaliser.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowdenormaliser.adoc
@@ -34,9 +34,11 @@ Note: make sure to check the notes on this transform in the xref:pipeline/beam/g
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowflattener.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowflattener.adoc
index d4137fb1b7..497f1c76d2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowflattener.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowflattener.adoc
@@ -32,9 +32,11 @@ The Row Flattener transform allows you to flatten data sequentially.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowgenerator.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowgenerator.adoc
index 1b69466657..0842b66e31 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowgenerator.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rowgenerator.adoc
@@ -41,9 +41,11 @@ Examples:
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rownormaliser.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rownormaliser.adoc
index 9bf600c621..efa6b48b4e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rownormaliser.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rownormaliser.adoc
@@ -36,9 +36,11 @@ You can use this transform to normalize repeating groups of columns.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rulesaccumulator.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rulesaccumulator.adoc
index e31b92c520..4477f86599 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rulesaccumulator.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rulesaccumulator.adoc
@@ -34,9 +34,11 @@ https://www.drools.org/[Drools] is the present rule engine implementation and it
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rulesexecutor.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rulesexecutor.adoc
index 9ea2ea5937..e98194d0de 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rulesexecutor.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rulesexecutor.adoc
@@ -34,9 +34,11 @@ https://www.drools.org/[Drools] is the present rule engine implementation and it
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/runssh.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/runssh.adoc
index b17c67cb17..32c8803b6e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/runssh.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/runssh.adoc
@@ -34,9 +34,11 @@ You can pass text to stdout or stderr in the commands. This information can then
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforcedelete.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforcedelete.adoc
index 830f69d1b6..81464bf41f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforcedelete.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforcedelete.adoc
@@ -32,9 +32,11 @@ The Salesforce Delete transform deletes records directly from your Salesforce da
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceinput.adoc
index 35273caf42..ac0825985e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceinput.adoc
@@ -38,9 +38,11 @@ You can also use the following transforms for various ways to modify your Salesf
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceinsert.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceinsert.adoc
index 7995f35922..5001ffeb21 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceinsert.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceinsert.adoc
@@ -39,9 +39,11 @@ You can also use the following other transforms to modify your Salesforce databa
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceupdate.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceupdate.adoc
index d55b106904..5a8866e20b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceupdate.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceupdate.adoc
@@ -39,9 +39,11 @@ You can also use the following other transforms to modify your Salesforce databa
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceupsert.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceupsert.adoc
index f8de7a787f..6c71778c76 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceupsert.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/salesforceupsert.adoc
@@ -39,9 +39,11 @@ You can also use the following other transforms to modify your Salesforce databa
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/samplerows.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/samplerows.adoc
index db800f06ab..ba4aa081a5 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/samplerows.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/samplerows.adoc
@@ -32,9 +32,11 @@ The Sample Rows transform retains a sample set of rows.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sasinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sasinput.adoc
index 9caf597cd1..f7c829b7a8 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sasinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sasinput.adoc
@@ -34,9 +34,11 @@ The functionality is backed by the https://github.com/epam/parso[Parso java libr
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/schemamapping.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/schemamapping.adoc
index 2d39fd8054..24090fa073 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/schemamapping.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/schemamapping.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Schema Mapping transform maps fields from an incoming stream over a specified Schema Definition. The result of this transform is an output stream that conforms to the mapped Schema Definition.
+:description: The Schema Mapping transform maps fields from an incoming stream over a specified Schema Definition.
= image:transforms/icons/schemamapping.svg[Schema Mapping transform Icon, role="image-doc-icon"] Schema Mapping
@@ -32,9 +32,11 @@ The Schema Mapping transform maps fields from an incoming stream over a specifie
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/script.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/script.adoc
index ed72273540..a6cd5e159d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/script.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/script.adoc
@@ -41,9 +41,11 @@ On Java 17 and later, the JVM no longer includes an ECMAScript engine; Hop bundl
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/selectvalues.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/selectvalues.adoc
index 39ccccb786..ff6c85e05a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/selectvalues.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/selectvalues.adoc
@@ -38,9 +38,11 @@ These operations are organized into different categories:
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-de-from-file.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-de-from-file.adoc
index cd9b499cd9..6add72be2e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-de-from-file.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-de-from-file.adoc
@@ -32,9 +32,11 @@ The De-serialize From File transform reads rows of data from a binary Hop file c
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-to-file.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-to-file.adoc
index 1e4bf179da..c73ca4f127 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-to-file.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-to-file.adoc
@@ -36,8 +36,10 @@ The Serialize to file transform supports a write-once access pattern, and does n
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serverstatus.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serverstatus.adoc
index 5adf599cc2..207007c595 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serverstatus.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serverstatus.adoc
@@ -35,9 +35,11 @@ Every output field for which you specify a name will be added to the list of out
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvalueconstant.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvalueconstant.adoc
index 25eabdbbb4..1019f07de9 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvalueconstant.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvalueconstant.adoc
@@ -32,9 +32,11 @@ The Set Field Value To A Constant transform replaces the value of a field with a
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvaluefield.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvaluefield.adoc
index 57c99bf573..65339e5d36 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvaluefield.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvaluefield.adoc
@@ -32,9 +32,11 @@ The Set Field Value transform replaces the value of a field with the value of an
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvariable.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvariable.adoc
index 18256d9104..5abd78ce01 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvariable.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvariable.adoc
@@ -30,9 +30,11 @@ Set Variables allows you to set variables from fields. By clicking the Get Field
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/simple-mapping.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/simple-mapping.adoc
index d1f45c0eaa..606cf16ab6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/simple-mapping.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/simple-mapping.adoc
@@ -31,9 +31,11 @@ The Simple Mapping transform allows you to re-use a series of transforms in the
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/snowflakebulkloader.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/snowflakebulkloader.adoc
index ee35c19dd7..9f78bae37f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/snowflakebulkloader.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/snowflakebulkloader.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Snowflake Bulk Loader transform utilizes the Snowflake Copy command to load data as opposed to sending individual insert statements through the Table Output transform
+:description: The Snowflake Bulk Loader transform utilizes the Snowflake Copy command to load data.
= image:transforms/icons/snowflakebulkloader.svg[Snowflake Bulk Loader transform Icon, role="image-doc-icon"] Snowflake Bulk Loader
@@ -36,9 +36,11 @@ The Snowflake Bulk Loader transform utilizes the Snowflake Copy command to load
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/snowflakeid.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/snowflakeid.adoc
index f14ab5dde9..c464ce1986 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/snowflakeid.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/snowflakeid.adoc
@@ -40,9 +40,11 @@ Typical use cases:
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sort.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sort.adoc
index f23ec53a7b..fe1eedf6fb 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sort.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sort.adoc
@@ -36,9 +36,11 @@ TIP: You use can use multiple copies of the Sort Rows transform to speed up larg
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sortedmerge.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sortedmerge.adoc
index 2b7b98d78a..eaa2756a13 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sortedmerge.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sortedmerge.adoc
@@ -32,9 +32,11 @@ The Sorted Merge transform merges rows coming from multiple input transforms pro
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sortedschemamerge.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sortedschemamerge.adoc
index bfbd09c0e2..78b1f2f6a2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sortedschemamerge.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sortedschemamerge.adoc
@@ -40,9 +40,11 @@ IMPORTANT: Each input stream must already be sorted on the configured sort keys
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-input.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-input.adoc
new file mode 100644
index 0000000000..4791a6b7d4
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-input.adoc
@@ -0,0 +1,130 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Spark Lake Table Input reads Delta Lake or Apache Iceberg tables on the native Spark pipeline engine, with optional time travel and projection.
+
+= image:transforms/icons/spark-lake-table-input.svg[Spark Lake Table Input transform Icon, role="image-doc-icon"] Spark Lake Table Input
+
+[%noheader,cols="3a,1a", role="table-no-borders" ]
+|===
+|
+== Description
+
+*Spark Lake Table Input* reads an open table format table (https://delta.io/[Delta Lake] or https://iceberg.apache.org/[Apache Iceberg]) as a Spark SQL `Dataset` on the xref:pipeline/spark/getting-started-with-native-spark.adoc[native Spark pipeline engine].
+
+It supports **PATH** and **TABLE** identifiers, optional field projection, and **time travel** by version/snapshot or timestamp.
+
+This transform is **not** available on the local Hop engine or on Beam engines. Connector JARs ship with the native Spark plugin; see xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine].
+
+|
+== Supported Engines
+[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
+!===
+!Hop Engine! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:cross.svg[Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
+!===
+
+NOTE: *Spark* here means Hop’s **native** Spark pipeline engine only (not Beam Spark).
+|===
+
+== Options
+
+[options="header",cols="1,3"]
+|===
+|Option |Description
+
+|Transform name
+|Unique name of the transform in the pipeline.
+
+|Format
+|`delta` or `iceberg`.
+
+|Identifier mode
+|`PATH` — table root path/URI. +
+`TABLE` — catalog-qualified name (e.g. `lake.db.orders`).
+
+|Table path
+|Used when mode is PATH. Must be visible to Spark workers (local path on `local[*]`; shared URI on cluster).
+
+|Table identifier
+|Used when mode is TABLE. Multi-part name: `catalog.namespace.table`.
+
+|Spark Catalog metadata name
+|Optional Hop xref:metadata-types/spark-catalog.adoc[Spark Catalog] entry to register for TABLE mode (Iceberg Hadoop/REST, …). Leave empty if the catalog is already configured on the session.
+
+|Time travel type
+|`NONE` (current), `VERSION`, or `TIMESTAMP`.
+
+|Time travel version
+|Delta table version number or Iceberg snapshot id (when type is VERSION). Supports variables.
+
+|Time travel timestamp
+|Spark-parseable timestamp string when type is TIMESTAMP, e.g. `2024-01-15 10:30:00`.
+
+|Extra options
+|Optional Spark read options as `key=value` lines (advanced).
+
+|Fields
+|Optional projection. When empty, all columns are returned. When set, columns are selected/cast by name (same idea as Spark File Input with a schema).
+|===
+
+== Time travel
+
+[options="header",cols="1,2,2"]
+|===
+|Type |Delta |Iceberg
+
+|NONE
+|Current table version
+|Current snapshot
+
+|VERSION
+|Read option `versionAsOf`
+|SQL `VERSION AS OF <snapshot_id>`
+
+|TIMESTAMP
+|Read option `timestampAsOf`
+|SQL `TIMESTAMP AS OF TIMESTAMP '…'`
+|===
+
+Tip: After two Overwrite writes, Delta version `0` is typically the first commit. For Iceberg, inspect `{table}.snapshots` (or catalog metadata) for snapshot ids.
+
+== Metadata injection
+
+All options support xref:pipeline/metadata-injection.adoc[metadata injection] (ETL Metadata Injection).
+
+Common keys: `FORMAT`, `IDENTIFIER_MODE`, `TABLE_PATH`, `TABLE_IDENTIFIER`, `CATALOG_METADATA_NAME`, `TIME_TRAVEL_TYPE`, `TIME_TRAVEL_VERSION`, `TIME_TRAVEL_TIMESTAMP`, `EXTRA_OPTIONS`.
+
+Field projection group `FIELDS`: `NAME`, `TYPE`, `LENGTH`, `PRECISION`, `FORMAT_MASK`.
+
+Typical pattern: template pipeline with Lake Table Input; injecting pipeline supplies table paths / identifiers (and optional field list) per supplier or table.
+
+== Notes
+
+* Prefer Lake Table Input over *Spark File Input* for `delta` / `iceberg` so session extensions, `hop_iceberg` PATH catalog, and probes stay consistent.
+* See the xref:pipeline/spark/lakehouse.adoc[lakehouse guide] for packaging, PATH vs TABLE, and troubleshooting.
+
+== Related pages
+
+* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark Lake Table Output]
+* xref:metadata-types/spark-catalog.adoc[Spark Catalog]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-maintenance.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-maintenance.adoc
new file mode 100644
index 0000000000..82dbc8a23b
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-maintenance.adoc
@@ -0,0 +1,140 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Spark Lake Table Maintenance runs OPTIMIZE, VACUUM, expire snapshots, and related operations on Delta Lake or Apache Iceberg tables.
+
+= image:transforms/icons/spark-lake-table-maintenance.svg[Spark Lake Table Maintenance transform Icon, role="image-doc-icon"] Spark Lake Table Maintenance
+
+[%noheader,cols="3a,1a", role="table-no-borders" ]
+|===
+|
+== Description
+
+*Spark Lake Table Maintenance* runs operational SQL / procedures against a https://delta.io/[Delta Lake] or https://iceberg.apache.org/[Apache Iceberg] table on the xref:pipeline/spark/getting-started-with-native-spark.adoc[native Spark pipeline engine]: compaction (OPTIMIZE / rewrite data files), vacuum, expire snapshots, rewrite manifests, or conditional deletes.
+
+It is a **zero-input** action sink (no hop required). Destructive operations require an explicit acknowledgement.
+
+Connectors ship with the native Spark plugin. Session rules: xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine].
+
+|
+== Supported Engines
+[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
+!===
+!Hop Engine! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
+!===
+
+NOTE: *Spark* here means Hop’s **native** Spark pipeline engine only (not Beam Spark).
+|===
+
+== Options
+
+[options="header",cols="1,3"]
+|===
+|Option |Description
+
+|Transform name
+|Unique name of the transform in the pipeline.
+
+|Format
+|`delta` or `iceberg`.
+
+|Identifier mode
+|`PATH` or `TABLE` for the target table.
+
+|Table path / Table identifier / Spark Catalog
+|Same semantics as other Lake Table transforms.
+
+|Operation
+|See <<operations,Operations matrix>> below.
+
+|Retention hours
+|**Required** for `VACUUM` and `EXPIRE_SNAPSHOTS`. No silent default — you must set how long to retain history.
+
+|Retain last
+|Iceberg `expire_snapshots` `retain_last` (defaults to `1` when retention is set and this field is blank).
+
+|Where clause
+|Optional filter for `OPTIMIZE` / required for `DELETE_WHERE` (without the `WHERE` keyword), e.g. `id < 10`.
+
+|Z-Order columns
+|Optional comma-separated columns for Delta `OPTIMIZE … ZORDER BY (…)`.
+
+|Acknowledge destructive
+|Must be enabled for `VACUUM`, `EXPIRE_SNAPSHOTS`, and `DELETE_WHERE`. Confirms that data files or snapshots may be removed.
+|===
+
+[[operations]]
+== Operations matrix
+
+[options="header",cols="1,2,2"]
+|===
+|Operation |Delta |Iceberg
+
+|OPTIMIZE
+|`OPTIMIZE delta.\`path\` [ZORDER BY …]` (optional WHERE)
+|`CALL cat.system.rewrite_data_files(table => '…')`
+
+|VACUUM
+|`VACUUM … RETAIN n HOURS` (**n required**); destructive ack
+|Not used — use `EXPIRE_SNAPSHOTS`
+
+|EXPIRE_SNAPSHOTS
+|n/a
+|`CALL … expire_snapshots(…)` with retention / retain_last; destructive ack
+
+|REWRITE_MANIFESTS
+|n/a
+|`CALL … rewrite_manifests(…)`
+
+|DELETE_WHERE
+|`DELETE FROM … WHERE …` (**WHERE required**); destructive ack
+|Same SQL pattern; destructive ack
+|===
+
+Aliases accepted for operation: `COMPACT` / `REWRITE_DATA_FILES` → OPTIMIZE; `EXPIRE` → EXPIRE_SNAPSHOTS; `DELETE` → DELETE_WHERE.
+
+== Behaviour
+
+* SQL is executed as a Spark action during graph materialisation; an empty leaf Dataset is registered afterward.
+* Iceberg CALL procedures use the catalog name from TABLE mode (`lake` in `lake.db.t`) or `hop_iceberg` for PATH mode.
+* Metrics are **log-only**; expect little or no GUI row traffic on this sink.
+
+== Metadata injection
+
+Supports xref:pipeline/metadata-injection.adoc[metadata injection].
+
+Keys: `FORMAT`, `IDENTIFIER_MODE`, `TABLE_PATH`, `TABLE_IDENTIFIER`, `CATALOG_METADATA_NAME`, `OPERATION`, `RETENTION_HOURS`, `RETAIN_LAST`, `WHERE_CLAUSE`, `ZORDER_COLUMNS`, `ACKNOWLEDGE_DESTRUCTIVE`.
+
+Injecting pipelines can drive fleet-wide OPTIMIZE/VACUUM across many tables; still set retention and destructive acknowledgement deliberately.
+
+== Safety
+
+* There is **no** default retention for vacuum/expire — set hours deliberately for your recovery needs.
+* Destructive ops without acknowledgement fail validation.
+* Prefer OPTIMIZE / rewrite for routine compaction; use VACUUM / expire only when you understand snapshot and time-travel impact.
+
+== Related pages
+
+* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark Lake Table Output]
+* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark Lake Table Merge]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-merge.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-merge.adoc
new file mode 100644
index 0000000000..72713919c8
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-merge.adoc
@@ -0,0 +1,119 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Spark Lake Table Merge runs MERGE INTO against Delta Lake or Apache Iceberg tables on the native Spark pipeline engine.
+
+= image:transforms/icons/spark-lake-table-merge.svg[Spark Lake Table Merge transform Icon, role="image-doc-icon"] Spark Lake Table Merge
+
+[%noheader,cols="3a,1a", role="table-no-borders" ]
+|===
+|
+== Description
+
+*Spark Lake Table Merge* applies a Spark SQL `MERGE INTO` against a https://delta.io/[Delta Lake] or https://iceberg.apache.org/[Apache Iceberg] target table on the xref:pipeline/spark/getting-started-with-native-spark.adoc[native Spark pipeline engine].
+
+The **single main-stream upstream hop** is the merge **source**. The transform builds (or accepts) MERGE SQL, executes it as a Spark action, and registers an empty leaf Dataset.
+
+Connectors ship with the native Spark plugin. Session rules: xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine].
+
+|
+== Supported Engines
+[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
+!===
+!Hop Engine! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
+!===
+
+NOTE: *Spark* here means Hop’s **native** Spark pipeline engine only (not Beam Spark).
+|===
+
+== Options
+
+[options="header",cols="1,3"]
+|===
+|Option |Description
+
+|Transform name
+|Unique name of the transform in the pipeline.
+
+|Format
+|`delta` or `iceberg`.
+
+|Identifier mode
+|`PATH` or `TABLE` for the **target** table.
+
+|Table path / Table identifier / Spark Catalog
+|Same semantics as Lake Table Input/Output for the merge target.
+
+|Merge condition
+|`ON` clause body. Use aliases **`t`** (target) and **`s`** (source), e.g. `t.id = s.id`. Required unless raw SQL is set.
+
+|When matched
+|`UPDATE_ALL` — `WHEN MATCHED THEN UPDATE SET *` +
+`DELETE` — delete matched target rows +
+`NONE` — no matched clause
+
+|When not matched
+|`INSERT_ALL` — `WHEN NOT MATCHED THEN INSERT *` +
+`NONE` — no not-matched clause
+
+|When not matched by source
+|`NONE` (default) or `DELETE` (Delta; Iceberg support depends on engine/version). Removes target rows with no source match.
+
+|Raw MERGE SQL
+|Advanced override. When non-empty, replaces structured fields. Empty by default. Operator is trusted to supply valid SQL.
+|===
+
+== Behaviour
+
+* Source rows are registered as a temporary view; structured mode emits SQL similar to:
++
+----
+MERGE INTO delta.`/path/to/table` AS t
+USING hop_merge_src_… AS s
+ON t.id = s.id
+WHEN MATCHED THEN UPDATE SET *
+WHEN NOT MATCHED THEN INSERT *
+----
+* Target SQL ids: Delta PATH → `delta.\`path\``; Iceberg PATH → `hop_iceberg.\`uri\``; TABLE → multi-part identifier.
+* At least one action (matched and/or not-matched) is required.
+* Metrics are **log-only** (merge completion); GUI in/out row counts for the sink may stay at zero.
+
+== Metadata injection
+
+Supports xref:pipeline/metadata-injection.adoc[metadata injection].
+
+Keys: `FORMAT`, `IDENTIFIER_MODE`, `TABLE_PATH`, `TABLE_IDENTIFIER`, `CATALOG_METADATA_NAME`, `MERGE_CONDITION`, `MATCHED_ACTION`, `NOT_MATCHED_ACTION`, `NOT_MATCHED_BY_SOURCE_ACTION`, `RAW_MERGE_SQL`.
+
+Useful for multi-table CDC/upsert templates where merge keys and target tables vary per run.
+
+== Notes
+
+* Exactly **one** main-stream input hop.
+* Prefer structured fields for common upserts; use raw SQL only when you need full MERGE control.
+* Packaging and PATH vs TABLE: xref:pipeline/spark/lakehouse.adoc[lakehouse guide].
+
+== Related pages
+
+* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark Lake Table Output]
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark Lake Table Input]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-output.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-output.adoc
new file mode 100644
index 0000000000..a939b2d62b
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-output.adoc
@@ -0,0 +1,113 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:description: Spark Lake Table Output writes Delta Lake or Apache Iceberg tables on the native Spark pipeline engine with ACID commit semantics.
+
+= image:transforms/icons/spark-lake-table-output.svg[Spark Lake Table Output transform Icon, role="image-doc-icon"] Spark Lake Table Output
+
+[%noheader,cols="3a,1a", role="table-no-borders" ]
+|===
+|
+== Description
+
+*Spark Lake Table Output* writes the incoming Spark `Dataset` to a https://delta.io/[Delta Lake] or https://iceberg.apache.org/[Apache Iceberg] table using table-format commit semantics on the xref:pipeline/spark/getting-started-with-native-spark.adoc[native Spark pipeline engine].
+
+Unlike *Spark File Output* (classic file formats), the **default save mode is ErrorIfExists** so ACID tables are not overwritten by accident.
+
+This transform is **native Spark only**. Connectors ship with the native Spark plugin; see xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine].
+
+|
+== Supported Engines
+[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
+!===
+!Hop Engine! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:cross.svg[Not Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
+!===
+
+NOTE: *Spark* here means Hop’s **native** Spark pipeline engine only (not Beam Spark).
+|===
+
+== Options
+
+[options="header",cols="1,3"]
+|===
+|Option |Description
+
+|Transform name
+|Unique name of the transform in the pipeline.
+
+|Format
+|`delta` or `iceberg`.
+
+|Identifier mode
+|`PATH` or `TABLE` (see xref:pipeline/spark/lakehouse.adoc#path-vs-table[PATH vs TABLE]).
+
+|Table path
+|PATH mode: table root directory / URI.
+
+|Table identifier
+|TABLE mode: e.g. `lake.db.orders`.
+
+|Spark Catalog metadata name
+|Optional xref:metadata-types/spark-catalog.adoc[Spark Catalog] Hop metadata for TABLE mode.
+
+|Save mode
+|`ErrorIfExists` (**default**), `Append`, `Overwrite`, or `Ignore`. Choose Overwrite or Append explicitly when you intend to replace or extend data.
+
+|Partition by
+|Optional comma-separated partition column list.
+
+|Coalesce
+|Optional partition count before write (e.g. `1` for a single file layout — costs a shuffle).
+
+|Extra options
+|Optional Spark write options as `key=value` lines (advanced).
+|===
+
+== Behaviour
+
+* The write runs as a Spark **action** when the engine materialises the graph.
+* An empty leaf Dataset is registered so a later engine-wide `count()` does not re-write the table.
+* **Delta PATH:** `format("delta").mode(…).save(path)` (requires Delta extension + `DeltaCatalog` on the session — hop-run applies these when Delta transforms are present).
+* **Iceberg PATH:** `writeTo(hop_iceberg.\`uri\`).using("iceberg")` with the built-in `hop_iceberg` Hadoop catalog.
+* **Iceberg TABLE:** `writeTo(catalog.ns.table).using("iceberg")`.
+* **Delta TABLE (advanced):** `format("delta").saveAsTable(id)`.
+
+== Metadata injection
+
+Supports xref:pipeline/metadata-injection.adoc[metadata injection].
+
+Keys: `FORMAT`, `IDENTIFIER_MODE`, `TABLE_PATH`, `TABLE_IDENTIFIER`, `CATALOG_METADATA_NAME`, `SAVE_MODE`, `PARTITION_BY`, `COALESCE`, `EXTRA_OPTIONS`.
+
+Use this with a template write path when landing many tables (different paths/identifiers/save modes) from one injecting pipeline.
+
+== Notes
+
+* Requires exactly one main-stream upstream hop (the data to write).
+* For upserts, use xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark Lake Table Merge] instead of Overwrite.
+* Packaging, session conf, and troubleshooting: xref:pipeline/spark/lakehouse.adoc[lakehouse guide].
+
+== Related pages
+
+* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine]
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark Lake Table Input]
+* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark Lake Table Merge]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splitfields.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splitfields.adoc
index d3aafabfb7..62384e15ee 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splitfields.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splitfields.adoc
@@ -32,9 +32,11 @@ The Split Fields transform splits a field into multiple fields based on a specif
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splitfieldtorows.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splitfieldtorows.adoc
index 88c03ee916..140d42e20c 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splitfieldtorows.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splitfieldtorows.adoc
@@ -32,9 +32,11 @@ The Split Fields To Rows transform splits a row containing a delimited field int
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splunkinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splunkinput.adoc
index b8e317fa59..7b41943596 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splunkinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/splunkinput.adoc
@@ -34,9 +34,11 @@ Check the xref:metadata-types/splunk-connection.adoc[Splunk Connection docs] for
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sqlfileoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sqlfileoutput.adoc
index 1e1d3479fa..e312337751 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sqlfileoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sqlfileoutput.adoc
@@ -34,9 +34,11 @@ The SQL is generated in the dialect of the selected database.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sstable-output.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sstable-output.adoc
index 011bd3a5b3..8f45463b1a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sstable-output.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sstable-output.adoc
@@ -32,9 +32,11 @@ The Cassandra SSTable Output transform writes to a file system directory as a Ca
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/standardizephonenumber.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/standardizephonenumber.adoc
index e70259d20c..f07f15f0d9 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/standardizephonenumber.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/standardizephonenumber.adoc
@@ -34,9 +34,11 @@ The transform uses https://github.com/google/libphonenumber[Google libphonenumbe
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stanfordnlp.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stanfordnlp.adoc
index 742a84c47c..2ebe90a68a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stanfordnlp.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stanfordnlp.adoc
@@ -31,9 +31,11 @@ Stanford CoreNLP provides a set of natural language analysis tools which can tak
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/streamlookup.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/streamlookup.adoc
index 18f611c36f..624e1dd7bf 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/streamlookup.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/streamlookup.adoc
@@ -36,9 +36,11 @@ TIP: Since this transform loads the lookup data into memory, it can be an extrem
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/streamschemamerge.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/streamschemamerge.adoc
index f9fa84622e..032ff30581 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/streamschemamerge.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/streamschemamerge.adoc
@@ -35,9 +35,11 @@ If you want to make sure fields keep their original data type, make sure to conv
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stringcut.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stringcut.adoc
index 3da4961ad2..5cf5ab7318 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stringcut.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stringcut.adoc
@@ -32,9 +32,11 @@ The Strings Cut transform cuts a portion of a string (i.e., a substring). If the
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stringoperations.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stringoperations.adoc
index c4aef185b0..d89f1f5fc0 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stringoperations.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/stringoperations.adoc
@@ -40,9 +40,11 @@ The String Operations transform can perform the following string operations on a
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/switchcase.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/switchcase.adoc
index c37590e6c5..e1211434c5 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/switchcase.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/switchcase.adoc
@@ -34,9 +34,11 @@ In our case we route rows of data to one or more target transforms based on the
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/synchronizeaftermerge.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/synchronizeaftermerge.adoc
index 656f9b04d0..efe42e6f52 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/synchronizeaftermerge.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/synchronizeaftermerge.adoc
@@ -36,9 +36,11 @@ This flag column is then used by the Synchronize After Merge pipeline transform
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tablecompare.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tablecompare.adoc
index ca686f8689..4c9d08c5e7 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tablecompare.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tablecompare.adoc
@@ -37,9 +37,11 @@ TIP: You can click on the red error handling arrow to configure additional infor
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableexists.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableexists.adoc
index 9c78d0ca33..cbd34c74a1 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableexists.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableexists.adoc
@@ -32,9 +32,11 @@ The Table Exists transform verifies if a certain table exists in a database. The
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableinput.adoc
index 62a7d14b47..28fed70917 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableinput.adoc
@@ -40,9 +40,11 @@ These features make Table Input ideal for dynamic and reusable data retrieval lo
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableoutput.adoc
index 34d840a05a..86651745e8 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tableoutput.adoc
@@ -32,9 +32,11 @@ The Table Output transform inserts data into a relational database table.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/terafast.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/terafast.adoc
index a0e8ac6ddc..78842dfcfd 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/terafast.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/terafast.adoc
@@ -35,9 +35,11 @@ The Teradata Bulkloader transform supports fastloading data into a Teradata data
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textfileinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textfileinput.adoc
index 184586b210..f71f84eb24 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textfileinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textfileinput.adoc
@@ -34,9 +34,11 @@ The features of the transform allow you to read from a list of files or director
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textfileoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textfileoutput.adoc
index f70251cdb0..955eb9f5a9 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textfileoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textfileoutput.adoc
@@ -41,9 +41,11 @@ The `ignore manual fields` ignores any fields manually defined in the transform'
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tokenreplacement.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tokenreplacement.adoc
index e80f171612..f92e32630f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tokenreplacement.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tokenreplacement.adoc
@@ -46,9 +46,11 @@ When replacing tokens in a file it is a best practice to output to a file also t
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/uniquerows.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/uniquerows.adoc
index 10f37fe186..197b51476d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/uniquerows.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/uniquerows.adoc
@@ -34,9 +34,11 @@ For unsorted input streams, check the xref:pipeline/transforms/uniquerowsbyhashs
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/uniquerowsbyhashset.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/uniquerowsbyhashset.adoc
index f993dd4497..8c363da0d2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/uniquerowsbyhashset.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/uniquerowsbyhashset.adoc
@@ -34,9 +34,11 @@ This transform differs from the xref:pipeline/transforms/uniquerows.adoc[Unique
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:cross.svg[Not Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/update.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/update.adoc
index e506574900..8fa86985ad 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/update.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/update.adoc
@@ -36,9 +36,11 @@ If the row is found the row in the table is updated.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/userdefinedjavaclass.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/userdefinedjavaclass.adoc
index 22dbc32121..d7e153cd12 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/userdefinedjavaclass.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/userdefinedjavaclass.adoc
@@ -42,9 +42,11 @@ For this we use the https://janino-compiler.github.io/janino/[Janino^] project l
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/userdefinedjavaexpression.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/userdefinedjavaexpression.adoc
index ee4462e23c..bc0c3e9258 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/userdefinedjavaexpression.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/userdefinedjavaexpression.adoc
@@ -32,9 +32,11 @@ This transform allows you to enter Java expressions that are used to calculate n
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/validator.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/validator.adoc
index 11bbbc2b6f..c2aa039543 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/validator.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/validator.adoc
@@ -34,9 +34,11 @@ To get a list of all the validation errors you can define an xref:pipeline/error
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/valuemapper.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/valuemapper.adoc
index d136a6ab22..dda8793649 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/valuemapper.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/valuemapper.adoc
@@ -45,9 +45,11 @@ Source/Target: EN/English, FR/French, NL/Dutch, ES/Spanish, DE/German, ...
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/vcardinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/vcardinput.adoc
index 22a37a03fc..0b54c36148 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/vcardinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/vcardinput.adoc
@@ -38,9 +38,11 @@ TIP: More information on the vCard format is available on https://en.wikipedia.o
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/vcardoutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/vcardoutput.adoc
index 27cfda38fd..9e2a895ae3 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/vcardoutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/vcardoutput.adoc
@@ -34,9 +34,11 @@ You can output the generated vCard text as a field, write it to a file, or both.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/verticabulkloader.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/verticabulkloader.adoc
index d0dcd23a0a..0cc6829be2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/verticabulkloader.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/verticabulkloader.adoc
@@ -35,9 +35,11 @@ This is typically significantly faster than loading data through e.g. a Table Ou
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/webservices.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/webservices.adoc
index 55fab8a5ad..82122bcc8c 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/webservices.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/webservices.adoc
@@ -32,9 +32,11 @@ The Web Services Lookup transform performs a Web Services lookup using the Web S
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc
index 3dc3d9fd2e..33b1c7065c 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc
@@ -32,9 +32,11 @@ The Workflow Executor transform execute a Hop workflow from within a pipeline.
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-logging.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-logging.adoc
index 4de94984db..b5b98296fb 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-logging.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-logging.adoc
@@ -37,9 +37,11 @@ The transform requires very little configuration, but provides a lot of informat
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/writetolog.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/writetolog.adoc
index ebd79225ff..b3bdde87da 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/writetolog.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/writetolog.adoc
@@ -36,9 +36,11 @@ Typical use cases are logging specific values or custom logging messages to the
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmlinputstream.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmlinputstream.adoc
index 8ca11bedc4..3f3a85e736 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmlinputstream.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmlinputstream.adoc
@@ -32,9 +32,11 @@ The XML Input Stream (StAX) transform reads data from XML files using the Stream
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmljoin.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmljoin.adoc
index 62426c24ea..b0c79ec44e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmljoin.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmljoin.adoc
@@ -36,9 +36,11 @@ Only one row is produced after the join. This single row contains the fields of
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmloutput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmloutput.adoc
index b0347c4fd3..921eb24b96 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmloutput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmloutput.adoc
@@ -32,9 +32,11 @@ The XML Output transform allows you to write rows from any source to one or more
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:check_mark.svg[Supported, 24]
-!Flink! image:check_mark.svg[Supported, 24]
-!Dataflow! image:check_mark.svg[Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:check_mark.svg[Supported, 24]
+!Beam Flink! image:check_mark.svg[Supported, 24]
+!Beam Dataflow! image:check_mark.svg[Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmloutputadvanced.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmloutputadvanced.adoc
index 34dd775af7..ca5fcc0f86 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmloutputadvanced.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xmloutputadvanced.adoc
@@ -36,9 +36,11 @@ This transform complements the simpler `XML Output` transform. Use `XML Output`
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:cross.svg[Not Supported, 24]
-!Flink! image:cross.svg[Not Supported, 24]
-!Dataflow! image:cross.svg[Not Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:cross.svg[Not Supported, 24]
+!Beam Flink! image:cross.svg[Not Supported, 24]
+!Beam Dataflow! image:cross.svg[Not Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xsdvalidator.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xsdvalidator.adoc
index f612ce8a89..1e10b72fea 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xsdvalidator.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xsdvalidator.adoc
@@ -43,9 +43,11 @@ See also the xref:workflow/actions/xsdvalidator.adoc[XSD Validator workflow acti
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xslt.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xslt.adoc
index cdb6c04b08..8699038569 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xslt.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/xslt.adoc
@@ -34,9 +34,11 @@ XSLT is short for link:http://en.wikipedia.org/wiki/XSLT[Extensible Stylesheet L
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/yamlinput.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/yamlinput.adoc
index 9ee4d167ee..02f70bcb50 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/yamlinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/yamlinput.adoc
@@ -32,9 +32,11 @@ The YAML Input transform reads data from YAML structures, files or incoming fiel
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/zipfile.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/zipfile.adoc
index 2519101515..5f0435ce9e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/zipfile.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/zipfile.adoc
@@ -32,9 +32,11 @@ The Zip File transform creates a standard ZIP archive using the options you spec
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
-!Spark! image:question_mark.svg[Maybe Supported, 24]
-!Flink! image:question_mark.svg[Maybe Supported, 24]
-!Dataflow! image:question_mark.svg[Maybe Supported, 24]
+!Single Threaded! image:check_mark.svg[Supported, 24]
+!Native Spark! image:check_mark.svg[Supported, 24]
+!Beam Spark! image:question_mark.svg[Maybe Supported, 24]
+!Beam Flink! image:question_mark.svg[Maybe Supported, 24]
+!Beam Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc b/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc
index 91acb32492..5203122266 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc
@@ -29,6 +29,13 @@ Apache Hop makes fervent usage of VFS.
Beyond the standard VFS file system types, we have added a number which are present in the various technology stacks supported by Hop.
Like the standard file systems each has its own unique name scheme which you can use.
+[IMPORTANT]
+====
+**Native Spark** Dataset I/O (*Spark File Input/Output*, Lake Table PATH) does *not* use Hop VFS.
+Those transforms expect **Spark/Hadoop URIs** (for example `s3a://`, `hdfs://`, `file:///`), which are a different dialect from Hop schemes such as `s3://` or named MinIO connections.
+See xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems on native Spark].
+====
+
== Apache Hop VFS File Systems
The table below provides a quick overview of the VFS file systems supported by Apache Hop.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/databricks-job-run.adoc b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/databricks-job-run.adoc
new file mode 100644
index 0000000000..d2433b10f1
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/databricks-job-run.adoc
@@ -0,0 +1,181 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:imagesdir: ../../../assets/images/
+:page-pagination:
+:description: Databricks Job Run workflow action — run or submit Jobs API work and wait for completion.
+
+= Databricks Job Run
+
+== Description
+
+The **Databricks Job Run** action triggers a job on a Databricks workspace via the Jobs REST API and can wait until the run finishes. It stores Job ID, Run ID, and status in workflow variables for downstream actions.
+
+Use a xref:metadata-types/databricks-connection.adoc[Databricks Connection] (workspace host + personal access token). This is **not** the JDBC SQL warehouse connection.
+
+== Modes
+
+[options="header",cols="1,3"]
+|===
+|Mode |Behavior
+
+|Run existing job
+|Calls `jobs/run-now` with the given **Job ID**.
+
+|One-time submit
+|Calls `jobs/runs/submit` with a raw **Submit JSON** body (full Jobs API payload you provide).
+
+|Deploy & run
+|Uploads fat jar, pipeline `.hpl`, and exported metadata JSON to a Volume or DBFS base path (optionally a **Spark project package** zip and **environment config** file), then creates or updates a JAR job (`org.apache.hop.spark.run.MainSpark`) with classic existing / `new_cluster` or serverless compute and runs it. See xref:pipeline/spark/databricks.adoc[Native Spark on Databricks].
+|===
+
+== Wait modes
+
+[options="header",cols="1,3"]
+|===
+|Wait mode |Behavior
+
+|Wait for completion
+|Polls `jobs/runs/get` until the run is terminal (`SUCCESS`, `FAILED`, …) or **Timeout** is reached (0 = no limit).
+
+|Fire and forget
+|Returns as soon as the run is accepted; does not poll. Pair with xref:workflow/actions/databricks-job-wait.adoc[Databricks Job Wait] later in the workflow (using `${DatabricksRunId}`).
... 58133 lines suppressed ...