You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by "Sergey Zolotaryov (JIRA)" <ji...@apache.org> on 2010/03/15 12:17:44 UTC
[jira] Created: (CAMEL-2551) File component does not correctly
handle PipedInputStream in message body.
File component does not correctly handle PipedInputStream in message body.
--------------------------------------------------------------------------
Key: CAMEL-2551
URL: https://issues.apache.org/activemq/browse/CAMEL-2551
Project: Apache Camel
Issue Type: Bug
Components: camel-core
Affects Versions: 2.2.0
Environment: does not matter
Reporter: Sergey Zolotaryov
Streams that do not have their contents length at immediate disposal, like PipedInputStream, are not processed correctly by the file component.
\\
{code}
private void writeFileByStream(InputStream in, File target) throws IOException {
FileChannel out = null;
try {
out = prepareOutputFileChannel(target, out);
if (LOG.isTraceEnabled()) {
LOG.trace("Using InputStream to transfer from: " + in + " to: " + out);
}
int size = endpoint.getBufferSize();
byte[] buffer = new byte[size];
ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
while (true) {
int count = in.read(buffer);
if (count <= 0) {
break;
} else if (count < size) {
byteBuffer = ByteBuffer.wrap(buffer, 0, count);
out.write(byteBuffer);
break;
} else {
out.write(byteBuffer);
byteBuffer.clear();
}
}
} finally {
ObjectHelper.close(in, target.getName(), LOG);
ObjectHelper.close(out, target.getName(), LOG);
}
}
{code}
The code
{code}
} else if (count < size) {
byteBuffer = ByteBuffer.wrap(buffer, 0, count);
out.write(byteBuffer);
break;
} else {
{code}
does not take into account that bytes read can be less than the size of the buffer passed into the InputStream.read method and stream can still have more content. The only indication that EOF was reached is -1 returned from the read method according to Java API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-2551) File component does not correctly
handle PipedInputStream in message body.
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58259#action_58259 ]
Claus Ibsen commented on CAMEL-2551:
------------------------------------
Sergey well spotted.
Btw which use case do you use a PipedInputStream with?
> File component does not correctly handle PipedInputStream in message body.
> --------------------------------------------------------------------------
>
> Key: CAMEL-2551
> URL: https://issues.apache.org/activemq/browse/CAMEL-2551
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.2.0
> Environment: does not matter
> Reporter: Sergey Zolotaryov
> Assignee: Willem Jiang
> Attachments: patch.txt
>
>
> Streams that do not have their contents length at immediate disposal, like PipedInputStream, are not processed correctly by the file component.
> \\
> {code}
> private void writeFileByStream(InputStream in, File target) throws IOException {
> FileChannel out = null;
> try {
> out = prepareOutputFileChannel(target, out);
> if (LOG.isTraceEnabled()) {
> LOG.trace("Using InputStream to transfer from: " + in + " to: " + out);
> }
> int size = endpoint.getBufferSize();
> byte[] buffer = new byte[size];
> ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
> while (true) {
> int count = in.read(buffer);
> if (count <= 0) {
> break;
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> out.write(byteBuffer);
> byteBuffer.clear();
> }
> }
> } finally {
> ObjectHelper.close(in, target.getName(), LOG);
> ObjectHelper.close(out, target.getName(), LOG);
> }
> }
> {code}
> The code
> {code}
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> {code}
> does not take into account that bytes read can be less than the size of the buffer passed into the InputStream.read method and stream can still have more content. The only indication that EOF was reached is -1 returned from the read method according to Java API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CAMEL-2551) File component does not correctly
handle PipedInputStream in message body.
Posted by "Willem Jiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Willem Jiang reassigned CAMEL-2551:
-----------------------------------
Assignee: Willem Jiang
> File component does not correctly handle PipedInputStream in message body.
> --------------------------------------------------------------------------
>
> Key: CAMEL-2551
> URL: https://issues.apache.org/activemq/browse/CAMEL-2551
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.2.0
> Environment: does not matter
> Reporter: Sergey Zolotaryov
> Assignee: Willem Jiang
> Attachments: patch.txt
>
>
> Streams that do not have their contents length at immediate disposal, like PipedInputStream, are not processed correctly by the file component.
> \\
> {code}
> private void writeFileByStream(InputStream in, File target) throws IOException {
> FileChannel out = null;
> try {
> out = prepareOutputFileChannel(target, out);
> if (LOG.isTraceEnabled()) {
> LOG.trace("Using InputStream to transfer from: " + in + " to: " + out);
> }
> int size = endpoint.getBufferSize();
> byte[] buffer = new byte[size];
> ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
> while (true) {
> int count = in.read(buffer);
> if (count <= 0) {
> break;
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> out.write(byteBuffer);
> byteBuffer.clear();
> }
> }
> } finally {
> ObjectHelper.close(in, target.getName(), LOG);
> ObjectHelper.close(out, target.getName(), LOG);
> }
> }
> {code}
> The code
> {code}
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> {code}
> does not take into account that bytes read can be less than the size of the buffer passed into the InputStream.read method and stream can still have more content. The only indication that EOF was reached is -1 returned from the read method according to Java API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CAMEL-2551) File component does not correctly
handle PipedInputStream in message body.
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-2551.
--------------------------------
Resolution: Fixed
Fix Version/s: 2.3.0
> File component does not correctly handle PipedInputStream in message body.
> --------------------------------------------------------------------------
>
> Key: CAMEL-2551
> URL: https://issues.apache.org/activemq/browse/CAMEL-2551
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.2.0
> Environment: does not matter
> Reporter: Sergey Zolotaryov
> Assignee: Willem Jiang
> Fix For: 2.3.0
>
> Attachments: patch.txt
>
>
> Streams that do not have their contents length at immediate disposal, like PipedInputStream, are not processed correctly by the file component.
> \\
> {code}
> private void writeFileByStream(InputStream in, File target) throws IOException {
> FileChannel out = null;
> try {
> out = prepareOutputFileChannel(target, out);
> if (LOG.isTraceEnabled()) {
> LOG.trace("Using InputStream to transfer from: " + in + " to: " + out);
> }
> int size = endpoint.getBufferSize();
> byte[] buffer = new byte[size];
> ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
> while (true) {
> int count = in.read(buffer);
> if (count <= 0) {
> break;
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> out.write(byteBuffer);
> byteBuffer.clear();
> }
> }
> } finally {
> ObjectHelper.close(in, target.getName(), LOG);
> ObjectHelper.close(out, target.getName(), LOG);
> }
> }
> {code}
> The code
> {code}
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> {code}
> does not take into account that bytes read can be less than the size of the buffer passed into the InputStream.read method and stream can still have more content. The only indication that EOF was reached is -1 returned from the read method according to Java API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-2551) File component does not correctly
handle PipedInputStream in message body.
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58267#action_58267 ]
Claus Ibsen commented on CAMEL-2551:
------------------------------------
Sergey
Thanks for taking the time to explain the use case. I was just wondering where the PipedInputStream came into the picture.
But its nice with some many different use cases people use Camel with.
Willem have already committed your patch this morning:
http://svn.apache.org/viewvc?rev=923583&view=rev
> File component does not correctly handle PipedInputStream in message body.
> --------------------------------------------------------------------------
>
> Key: CAMEL-2551
> URL: https://issues.apache.org/activemq/browse/CAMEL-2551
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.2.0
> Environment: does not matter
> Reporter: Sergey Zolotaryov
> Assignee: Willem Jiang
> Fix For: 2.3.0
>
> Attachments: patch.txt
>
>
> Streams that do not have their contents length at immediate disposal, like PipedInputStream, are not processed correctly by the file component.
> \\
> {code}
> private void writeFileByStream(InputStream in, File target) throws IOException {
> FileChannel out = null;
> try {
> out = prepareOutputFileChannel(target, out);
> if (LOG.isTraceEnabled()) {
> LOG.trace("Using InputStream to transfer from: " + in + " to: " + out);
> }
> int size = endpoint.getBufferSize();
> byte[] buffer = new byte[size];
> ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
> while (true) {
> int count = in.read(buffer);
> if (count <= 0) {
> break;
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> out.write(byteBuffer);
> byteBuffer.clear();
> }
> }
> } finally {
> ObjectHelper.close(in, target.getName(), LOG);
> ObjectHelper.close(out, target.getName(), LOG);
> }
> }
> {code}
> The code
> {code}
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> {code}
> does not take into account that bytes read can be less than the size of the buffer passed into the InputStream.read method and stream can still have more content. The only indication that EOF was reached is -1 returned from the read method according to Java API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-2551) File component does not correctly
handle PipedInputStream in message body.
Posted by "Sergey Zolotaryov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58246#action_58246 ]
Sergey Zolotaryov commented on CAMEL-2551:
------------------------------------------
Current workaround we have to use is the following:
\\
{code}
InputStream is = ....
final InputStream camelBugWorkaround = new InputStream()
{
@Override
public int read(final byte[] b, int off, int len) throws IOException
{
final int expectedMaxBytes = len;
int read = 0;
int totalRead = 0;
while ((read = is.read(b, off, len)) != -1)
{
if (read < len)
{
off += read;
len -= read;
}
totalRead += read;
if (totalRead == expectedMaxBytes)
{
break;
}
}
return totalRead == 0 ? -1 : totalRead;
}
@Override
public int read() throws IOException
{
return is.read();
}
@Override
public void close() throws IOException
{
is.close();
}
@Override
public int available() throws IOException
{
return is.available();
}
@Override
public long skip(final long n) throws IOException
{
return is.skip(n);
}
};
exchange.getOut().setBody(camelBugWorkaround);
{code}
Where _is_ is the PipedInputStream.
> File component does not correctly handle PipedInputStream in message body.
> --------------------------------------------------------------------------
>
> Key: CAMEL-2551
> URL: https://issues.apache.org/activemq/browse/CAMEL-2551
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.2.0
> Environment: does not matter
> Reporter: Sergey Zolotaryov
> Attachments: patch.txt
>
>
> Streams that do not have their contents length at immediate disposal, like PipedInputStream, are not processed correctly by the file component.
> \\
> {code}
> private void writeFileByStream(InputStream in, File target) throws IOException {
> FileChannel out = null;
> try {
> out = prepareOutputFileChannel(target, out);
> if (LOG.isTraceEnabled()) {
> LOG.trace("Using InputStream to transfer from: " + in + " to: " + out);
> }
> int size = endpoint.getBufferSize();
> byte[] buffer = new byte[size];
> ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
> while (true) {
> int count = in.read(buffer);
> if (count <= 0) {
> break;
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> out.write(byteBuffer);
> byteBuffer.clear();
> }
> }
> } finally {
> ObjectHelper.close(in, target.getName(), LOG);
> ObjectHelper.close(out, target.getName(), LOG);
> }
> }
> {code}
> The code
> {code}
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> {code}
> does not take into account that bytes read can be less than the size of the buffer passed into the InputStream.read method and stream can still have more content. The only indication that EOF was reached is -1 returned from the read method according to Java API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-2551) File component does not correctly
handle PipedInputStream in message body.
Posted by "Sergey Zolotaryov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58266#action_58266 ]
Sergey Zolotaryov commented on CAMEL-2551:
------------------------------------------
Hi Claus
We use Camel for a downstream distribution product and one of the requirements is streaming of large amounts of data. The use case is bulk output of data - I put a db cursor into an Iterable, forward to a home made velocity component (out of the box velocity component writes to a String, which is not acceptable for us) - and it puts PipedInputStream as message body. A separate thread feeds the PipedOutputStream to velocity engine. Then when this stream is passed on to file component we get InputStream closed exception, because of the bug mentioned. For now I have to wrap PipedInputStream with a buffering stream, but if you include the fix, we will remove the workaround.
> File component does not correctly handle PipedInputStream in message body.
> --------------------------------------------------------------------------
>
> Key: CAMEL-2551
> URL: https://issues.apache.org/activemq/browse/CAMEL-2551
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.2.0
> Environment: does not matter
> Reporter: Sergey Zolotaryov
> Assignee: Willem Jiang
> Attachments: patch.txt
>
>
> Streams that do not have their contents length at immediate disposal, like PipedInputStream, are not processed correctly by the file component.
> \\
> {code}
> private void writeFileByStream(InputStream in, File target) throws IOException {
> FileChannel out = null;
> try {
> out = prepareOutputFileChannel(target, out);
> if (LOG.isTraceEnabled()) {
> LOG.trace("Using InputStream to transfer from: " + in + " to: " + out);
> }
> int size = endpoint.getBufferSize();
> byte[] buffer = new byte[size];
> ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
> while (true) {
> int count = in.read(buffer);
> if (count <= 0) {
> break;
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> out.write(byteBuffer);
> byteBuffer.clear();
> }
> }
> } finally {
> ObjectHelper.close(in, target.getName(), LOG);
> ObjectHelper.close(out, target.getName(), LOG);
> }
> }
> {code}
> The code
> {code}
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> {code}
> does not take into account that bytes read can be less than the size of the buffer passed into the InputStream.read method and stream can still have more content. The only indication that EOF was reached is -1 returned from the read method according to Java API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CAMEL-2551) File component does not correctly
handle PipedInputStream in message body.
Posted by "Sergey Zolotaryov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sergey Zolotaryov updated CAMEL-2551:
-------------------------------------
Attachment: patch.txt
This is the patch which works ok for our project.
> File component does not correctly handle PipedInputStream in message body.
> --------------------------------------------------------------------------
>
> Key: CAMEL-2551
> URL: https://issues.apache.org/activemq/browse/CAMEL-2551
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.2.0
> Environment: does not matter
> Reporter: Sergey Zolotaryov
> Attachments: patch.txt
>
>
> Streams that do not have their contents length at immediate disposal, like PipedInputStream, are not processed correctly by the file component.
> \\
> {code}
> private void writeFileByStream(InputStream in, File target) throws IOException {
> FileChannel out = null;
> try {
> out = prepareOutputFileChannel(target, out);
> if (LOG.isTraceEnabled()) {
> LOG.trace("Using InputStream to transfer from: " + in + " to: " + out);
> }
> int size = endpoint.getBufferSize();
> byte[] buffer = new byte[size];
> ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
> while (true) {
> int count = in.read(buffer);
> if (count <= 0) {
> break;
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> out.write(byteBuffer);
> byteBuffer.clear();
> }
> }
> } finally {
> ObjectHelper.close(in, target.getName(), LOG);
> ObjectHelper.close(out, target.getName(), LOG);
> }
> }
> {code}
> The code
> {code}
> } else if (count < size) {
> byteBuffer = ByteBuffer.wrap(buffer, 0, count);
> out.write(byteBuffer);
> break;
> } else {
> {code}
> does not take into account that bytes read can be less than the size of the buffer passed into the InputStream.read method and stream can still have more content. The only indication that EOF was reached is -1 returned from the read method according to Java API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.