You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JunRuiLee (via GitHub)" <gi...@apache.org> on 2026/06/01 09:31:25 UTC
[PR] [spark] Support ON_ERROR = CONTINUE / SKIP_FILE in COPY INTO [paimon]
JunRuiLee opened a new pull request, #8062:
URL: https://github.com/apache/paimon/pull/8062
## Motivation
COPY INTO previously only supported `ON_ERROR = ABORT_STATEMENT`: any parse or
cast error aborted the entire command. In production data-loading pipelines a
single malformed row or file would then fail the whole batch, which is often
too strict. This adds two error-tolerant modes:
- `CONTINUE` — skip bad rows and load the rest (row-level tolerance).
- `SKIP_FILE` — skip any file that contains an error, all-or-nothing per file.
`ABORT_STATEMENT` remains the default, so existing behavior is unchanged.
## Changes
- Grammar: `ON_ERROR` now accepts `CONTINUE` and `SKIP_FILE` in addition to
`ABORT_STATEMENT`.
- Result schema gains two columns:
- `errors_seen` (BIGINT) — number of error rows per file.
- `first_error` (STRING) — first error message, NULL when the file is clean.
- `status` now also reports `PARTIALLY_LOADED` and `LOAD_FAILED`.
- Error detection runs once per batch; both modes write in a single commit.
Load history is recorded so error-tolerant runs stay idempotent under
`FORCE = FALSE`.
- Refactor: `CopyIntoTableExec` is split into focused helpers
(`CopyIntoHelper`, `CopyIntoCastValidator`, `CopyIntoDataFrameBuilder`,
`CopyIntoErrorHandler`, `CopyIntoResultBuilder`), shared across CSV/JSON/Parquet.
- Docs updated in `sql-write.md`, including the CSV column-count-mismatch caveat
under `CONTINUE`.
Supported for CSV, JSON, and Parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support ON_ERROR = CONTINUE / SKIP_FILE in COPY INTO [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8062:
URL: https://github.com/apache/paimon/pull/8062#issuecomment-4603964091
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support ON_ERROR = CONTINUE / SKIP_FILE in COPY INTO [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8062:
URL: https://github.com/apache/paimon/pull/8062#issuecomment-4601022162
Thanks for the work here. I have one concern about the SQL dialect alignment before approving.
This syntax looks closer to a Snowflake-style `COPY INTO` subset than Databricks syntax. Databricks `COPY INTO` uses `FILEFORMAT = ...` plus `FORMAT_OPTIONS (...)` / `COPY_OPTIONS (...)`, and the documented copy options are things like `force` and `mergeSchema`; it does not expose `ON_ERROR = ABORT_STATEMENT | CONTINUE | SKIP_FILE` in this top-level form.
If the intent is to follow Databricks syntax, I think this PR is not aligned yet. If the intent is to introduce a Snowflake-style Paimon extension, please clarify that in the PR/docs and document that this is only a subset: `ABORT_STATEMENT`, `CONTINUE`, and `SKIP_FILE` are supported, while Snowflake variants such as `SKIP_FILE_<num>` / `SKIP_FILE_<num>%` and other `COPY INTO` options are not.
One more compatibility point: this changes the import result schema from 4 columns to 6 columns for all modes by adding `errors_seen` and `first_error`. Please confirm this result-schema compatibility impact is intentional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support ON_ERROR = CONTINUE / SKIP_FILE in COPY INTO [paimon]
Posted by "JunRuiLee (via GitHub)" <gi...@apache.org>.
JunRuiLee commented on PR #8062:
URL: https://github.com/apache/paimon/pull/8062#issuecomment-4598512977
Hi @JingsongLi,
Thanks for the review! I've addressed all comments, PTAL.
1. **CSV column-count mismatch**: Updated docs to match the implementation - mismatched rows are treated as errors and skipped with `ON_ERROR = CONTINUE`.
2. **nonReservedIdentifierTokenTypes**: Added `CONTINUE` and `SKIP_FILE` tokens with parser tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support ON_ERROR = CONTINUE / SKIP_FILE in COPY INTO [paimon]
Posted by "JunRuiLee (via GitHub)" <gi...@apache.org>.
JunRuiLee commented on PR #8062:
URL: https://github.com/apache/paimon/pull/8062#issuecomment-4601631447
Hi @JingsongLi , I've updated the docs that document that Paimon's COPY INTO follows Snowflake syntax (not Databricks) and supports only a subset. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support ON_ERROR = CONTINUE / SKIP_FILE in COPY INTO [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8062:
URL: https://github.com/apache/paimon/pull/8062#discussion_r3338311057
##########
docs/docs/spark/sql-write.md:
##########
@@ -597,9 +599,9 @@ By default (`FORCE = FALSE`), COPY INTO tracks which files have been successfull
### Limitations
+- **CSV column-count mismatch with ON_ERROR=CONTINUE**: Paimon follows Spark's PERMISSIVE mode behavior — rows with fewer columns are padded with NULL, and extra columns are silently ignored. These rows are loaded successfully and not counted as errors.
Review Comment:
The behavior described here conflicts with the implementation and the new tests below. `runTextImport` reads CSV with `_corrupt_record` in PERMISSIVE mode and `detectTextErrors` filters `corruptCol.isNotNull` as parse errors, while `CopyIntoOnErrorTest` expects fewer/extra columns to become `PARTIALLY_LOADED` with `errors_seen = 1`. So either this limitation should say these rows are skipped/count as errors, or the implementation/tests need to be changed to match the documented Spark PERMISSIVE behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support ON_ERROR = CONTINUE / SKIP_FILE in COPY INTO [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8062:
URL: https://github.com/apache/paimon/pull/8062#issuecomment-4603768377
This currently breaks `paimon-spark-ut` test compilation on the PR head. I ran:
```shell
mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -DfailIfNoTests=false \
-DwildcardSuites=org.apache.paimon.spark.execution.CopyIntoCastValidatorTest -Dtest=none test
```
and the Scala test compile fails before running the suite:
```text
CopyIntoCastValidatorTest.scala:29: not found: type CopyIntoCastValidator
CopyIntoDataFrameBuilderTest.scala:32: not found: type CopyIntoDataFrameBuilder
CopyIntoHelperTest.scala:28: not found: value CopyIntoHelper
```
The production `paimon-spark-common` module compiles, but these new helper-level tests were added under `paimon-spark-ut`. That module depends on the Spark-version common artifact (`${paimon-sparkx-common}`, defaulting to `paimon-spark3-common_2.12`), while the new helpers are defined in `paimon-spark-common` as `private[execution]` classes/objects. As a result the tests cannot resolve them from this module/classpath. Could you please move these helper-level tests to a module/classpath where `paimon-spark-common` sources are directly visible, or cover the same behavior through the existing Spark-version COPY INTO suites?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support ON_ERROR = CONTINUE / SKIP_FILE in COPY INTO [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8062:
URL: https://github.com/apache/paimon/pull/8062
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org