[GH] (geaflow/master): Workflow run "Java CI with Maven On JDK 8" is working again!

[GitBox <gi...@apache.org>]

The GitHub Actions job "Java CI with Maven On JDK 8" on geaflow.git/master has succeeded.
Run started by GitHub user Leomrlin (triggered by Leomrlin).

Head commit for run:
7847fa2514d5286fc7c12ce11b8a1b05387249e6 / E2ern1ty <11...@qq.com>
[ISSUE-817] Add geaflow-dsl-connector-doris (Stream Load sink + partitioned source) (#819)

* [ISSUE-817] Add geaflow-dsl-connector-doris (Stream Load sink + partitioned source)

- Sink writes to Doris via Stream Load: buffer rows in write(), flush a whole
  batch on row/byte threshold or window finish(), giving much higher throughput
  than row-by-row JDBC INSERT. CSV payload by default, JSON supported.
- Source reads through the MySQL protocol with parallel partitioned reads.
- SPI registration + parent/aggregation pom wiring + runtime dependency.
- CN + EN docs and examples.
- Unit tests and a Testcontainers integration test with a throughput benchmark
  (self-skips when Docker/host-networking is unavailable).

* [ISSUE-817] Make Doris integration test opt-in to keep CI green

The Doris Testcontainers integration test previously ran whenever Docker was
available on a Linux host, which is exactly the case on GitHub CI runners, so
it would try to pull the multi-GB Doris image and start a container during a
normal 'mvn test'. Make it opt-in: it only runs with -Ddoris.it.enabled=true
or when an external Doris is provided via -Ddoris.it.fenodes/-Ddoris.it.jdbcUrl,
and otherwise skips. Docs updated accordingly.

* [ISSUE-817] Address review: serialization safety, idempotent close, FE failover

Address the maintainer's review points on the Doris connector:
- serialization: default to json (Gson with serializeNulls + disableHtmlEscaping)
  so newlines, quotes, backslashes, unicode, null and empty strings round-trip
  correctly; document csv as a no-escape plain-split option.
- sink flush: flushBuffer stays safe on an empty buffer and is idempotent on
  finish(); close() now flushes the remaining buffer and is safe on repeated
  calls and when the client is not initialized.
- partition logic: clamp partition.num to at least 1 instead of throwing; the
  first partition already covers rows whose partition column is entirely NULL.
- label: kept globally unique via UUID.
- FE failover: DorisStreamLoad now takes the full FE list and rotates to the
  next FE on a failed request (retries at least once per FE).
- integration tests: add table-not-found, FE-failover and full data-type
  round-trip fidelity cases (still opt-in). Add unit tests for special-char /
  null / empty-string serialization and multi-FE url building.

* [ISSUE-817] Do not bundle the GPL MySQL driver; minor cleanup

- Make mysql-connector-java a test-scope dependency instead of compile, so the
  GPL-licensed driver is not shipped in the distribution (matching
  geaflow-dsl-connector-jdbc, which bundles no driver). The source only uses the
  standard java.sql API and loads the driver by name, so users must provide a
  MySQL-protocol driver on the runtime classpath; documented in CN + EN docs.
- Remove the unused COLON constant.

Verified: checkstyle 0 violations, RAT 0 unapproved (14 files), unit tests
23 run / 0 failed / 7 skipped (integration tests are opt-in).

* [ISSUE-817] Add real benchmark numbers verified against a local Doris

Verified the connector end to end against a real single-node
apache/doris:3.0.8-all: the sink (Stream Load), the partitioned source, FE
failover, table-not-found and full data-type round-trip (newlines, quotes,
backslashes, chinese, emoji) integration tests all pass. Benchmark: Stream Load
loaded 500 rows in ~91ms (~5500 rows/s) vs ~11335ms (~44 rows/s) for row-by-row
JDBC INSERT, ~125x faster; 2000 rows via Stream Load took only ~97ms. Recorded
the reference numbers in the CN + EN docs.

* [ISSUE-817] Address PR review: idempotent label, partition strategies, timeouts

- Label/idempotency: generate one label per batch and reuse it across all
  retries and FE failover attempts (Doris loads a label at most once); treat
  'Label Already Exists' on retry as success to avoid duplicate writes; log the
  label on failures/retries. Documented the write semantics.
- Partition strategies: add source.partition.mode (range/custom). 'custom' takes
  semicolon-separated WHERE predicates (source.partition.clauses), one partition
  per predicate, supporting non-numeric/skewed columns and arbitrary conditions.
  Documented the performance impact of unindexed/skewed partition columns and
  recommended indexed, evenly-distributed columns.
- Timeouts: read/write timeout is configurable and its default is raised to 60s
  to cover large-batch uploads; documented tuning guidance.
- Tests: add DorisTableSourceTest covering single/range/custom partition listing.

Report URL: https://github.com/apache/geaflow/actions/runs/30186794575

With regards,
GitHub Actions via GitBox


---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@geaflow.apache.org
For additional commands, e-mail: builds-help@geaflow.apache.org