You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "shashank (Jira)" <ji...@apache.org> on 2026/10/21 07:59:00 UTC
[jira] [Created] (CAMEL-25904) camel-beanio - BeanIOSplitter reads a file with the default charset of the JVM and ignores its encoding, so a file consumed with charset=ISO-8859-1 is split with its non-ASCII characters replaced
shashank created CAMEL-25904:
--------------------------------
Summary: camel-beanio - BeanIOSplitter reads a file with the default charset of the JVM and ignores its encoding, so a file consumed with charset=ISO-8859-1 is split with its non-ASCII characters replaced
Key: CAMEL-25904
URL: https://issues.apache.org/jira/browse/CAMEL-25904
Project: Camel
Issue Type: Bug
Components: camel-beanio
Reporter: shashank
Assignee: shashank
{{BeanIOSplitter.evaluate}} (main 74ee085f7d64, lines 79-107) creates the BeanIO reader like this:
{code:java}
if (body instanceof WrappedFile wrappedfile) {
body = wrappedfile.getFile();
}
if (body instanceof File file) {
beanReader = factory.createReader(getStreamName(), file);
}
if (beanReader == null) {
Reader reader = msg.getMandatoryBody(Reader.class);
beanReader = factory.createReader(getStreamName(), reader);
}
{code}
BeanIO's {{StreamFactory.createReader(String, File)}} (beanio 3.2.1) opens the file with a {{FileReader}}, that is with the default charset of the JVM. Other bodies (byte array, stream) are converted to a {{Reader}} with the charset of the exchange, else the default charset. The {{encoding}} option of the splitter ({{setEncoding}}, kept in its {{BeanIOConfiguration}}) is never read.
The usual route for big files is a file consumer and the streaming splitter, and the file body is always a {{File}}:
{code:java}
from("file:in?charset=ISO-8859-1")
.split(beanIOSplitter).streaming()
.to("direct:record");
{code}
The {{charset}} of the file consumer is ignored, and so is {{beanIOSplitter.setEncoding(StandardCharsets.ISO_8859_1)}}: an ISO-8859-1 or windows-1252 file (the typical encoding of the legacy fixed-length and CSV files BeanIO is used for) is decoded as UTF-8 on Java 18+, and every non-ASCII character of the records becomes U+FFFD ({{Jörg Müller}} -> {{J?rg M?ller}}). The split completes without an error. CAMEL-12284 fixed the same for {{BeanIODataFormat}} (its encoding option did not work); the splitter was added later (CAMEL-9843) and has never used it.
*Fix*: {{BeanIOSplitter}} decodes a file, stream or byte array body itself, with the charset of the exchange ({{CamelCharsetName}} header or property, as set by the {{charset}} option of the file consumer), else with its {{encoding}} (default: the default charset of the JVM, as before). A {{String}} or {{Reader}} body is read as before. The reader is closed if BeanIO cannot create its reader (an unknown stream name). Upgrade note in the 4.24 guide.
*Tests*: new {{BeanIOSplitterEncodingTest}} (4 tests, ISO-8859-1 data with ö, ü, é, ß). On main, 3 fail with {{expected: <[Jörg Müller Développeur, Renée Straße Architect]> but was: <[J?rg M?ller D?veloppeur, Ren?e Stra?e Architect]>}} (U+FFFD shown as ?): {{fileIsReadWithTheEncodingOfTheSplitter}} (a {{File}} body, {{setEncoding(ISO_8859_1)}}), {{fileFromTheFileConsumerIsReadWithItsCharset}} (a real file consumer with {{charset=ISO-8859-1}}), {{bytesAreReadWithTheEncodingOfTheSplitter}}. {{bytesAreReadWithTheCharsetOfTheExchange}} passes on main and with the fix (control: that path already used the exchange charset). camel-beanio unit tests: 25 run, 0 failures.
*How it was found*: a sweep of the data formats and their streaming splitters for options that one code path does not use (here the {{File}} branch hands the file to a library method without a charset). The JUnit tests above are the evidence.
*Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x (the same {{createReader(getStreamName(), file)}}; read, not run).
Not changed, possible follow-up: {{BeanIOIterator}} reads the first record in its constructor, before {{evaluate}} sets the error handler on the reader, so the {{ignoreInvalidRecords}} / {{ignoreUnexpectedRecords}} / custom {{BeanReaderErrorHandler}} options do not apply to the first record (an invalid first record fails the split, loudly).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)