You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/06/14 07:53:06 UTC
[GitHub] [camel-kafka-connector] ChhatralaNirmal opened a new issue #1208: CamelFileSink Connector appendChars = \n not working
ChhatralaNirmal opened a new issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208
Hello, We are using CamelFileSink connector to read the messages from Kafka and dump into file but we want each messages to be written in next line in file. We have been trying to use appendChars option with double slash \\n but it doesn't work.
Here is our file sink url :
file:/user/app/folder?appendChars=\\n&autoCreate=true&fileExist=Append&fileName=messages-${date:now:yyyyMMdd}.txt&forceWrites=true&charset=utf-8
Here is what the error we get :
Caused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: file:/user/app/folder?appendChars=\n&autoCreate=true&fileExist=Append&fileName=messages-${date:now:yyyyMMdd}.txt&forceWrites=true&charset=utf-8 due to: Illegal character in query at index 34: file:/user/app/folder?appendChars=\n&autoCreate=true&fileExist=Append&fileName=messages-$%7Bdate:now:yyyyMMdd%7D.txt&forceWrites=true&charset=utf-8\n\tat org.apache.camel.support.EndpointHelper.normalizeEndpointUri(EndpointHelper.java:156)\n\tat org.apache.camel.support.NormalizedUri.newNormalizedUri(NormalizedUri.java:46)\n\tat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] ChhatralaNirmal commented on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
ChhatralaNirmal commented on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860575886
We did that too but no change in result. same error.
"camel.sink.endpoint.appendChars": "\\n",
\nCaused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: file:/user/app/folder?appendChars=\n&autoCreate=true&charset=UTF-8&fileExist=Append&fileName=messages-${date:now:yyyyMMdd}.txt&forceWrites=true due to: Illegal character in query at index 34: file:/user/app/folder?appendChars=\n&autoCreate=true&charset=UTF-8&fileExist=Append&fileName=messages-$%7Bdate:now:yyyyMMdd%7D.txt&forceWrites=true\n\
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] ChhatralaNirmal commented on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
ChhatralaNirmal commented on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860560072
We tried using the properties instead of file sink url and the result is same. It gives us the same error.
Here is the error messages after defining the properties :
\nCaused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: file:/user/app/folder?appendChars=\n&autoCreate=true&charset=UTF-8&fileExist=Append&fileName=messages-${date:now:yyyyMMdd}.txt&forceWrites=true due to: Illegal character in query at index 34: file:/user/app/folder?appendChars=\n&autoCreate=true&charset=UTF-8&fileExist=Append&fileName=messages-$%7Bdate:now:yyyyMMdd%7D.txt&forceWrites=true\n\tat
Here is our full connect config :
{
"name": "camel-file-sink-connector",
"connector.class": "org.apache.camel.kafkaconnector.file.CamelFileSinkConnector",
"tasks.max": "1",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"topics": "camel-file-sink-kafka-topic",
"camel.sink.contentLogLevel": "DEBUG",
"camel.idempotency.enabled": "false",
"camel.sink.path.directoryName": "/user/app/folder",
"camel.sink.endpoint.charset": "UTF-8",
"camel.sink.endpoint.fileName": "messages-${date:now:yyyyMMdd}.txt",
"camel.sink.endpoint.appendChars": "\n",
"camel.sink.endpoint.fileExist": "Append",
"camel.sink.endpoint.forceWrites": "true",
"camel.sink.endpoint.autoCreate": "true"
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske closed issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske closed issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske edited a comment on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske edited a comment on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860786247
> @orpiske
>
> Is the "correct" value "\n" ?
>
> What would be the fix in the code?
Long story short is this: when using special chars for `appendChars` on camel-file, it requires us to escape the parameters (i.e.: `\\n` for appending new lines). However, it seems to me that because the way the configuration works on Kafka connect, the escaping doesn't work. Instead it requires this to be sent encoded as `%0A` from our config.
Therefore, I'm wondering if we should somehow fix this in CKC code or just update our documentation to instruct users to use `%0A` and others for special chars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske commented on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske commented on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860786247
> @orpiske
>
> Is the "correct" value "\n" ?
>
> What would be the fix in the code?
Long story short is this: `appendChars' on camel-file requires `\\n`. However, it seems to me that because the way the configuration works on Kafka connect, it requires this to be sent encoded as `%0A` from our config.
Therefore, I'm wondering if we should somehow fix this in CKC code or just update our documentation to instruct users to use `%0A` and others for special chars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske edited a comment on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske edited a comment on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860570807
Can you try escaping the new line char, please? Such as ```\\n```.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] valdar commented on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
valdar commented on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860784052
@orpiske
Is the "correct" value "\\n" ?
What would be the fix in the code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske commented on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske commented on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860578118
Thanks for double checking. I'll investigate this today (or, at worst, in a few days).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] ChhatralaNirmal commented on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
ChhatralaNirmal commented on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860776313
@orpiske - thank you so much for this advise. It worked perfectly fine and we were able to get the messages on next line.
Should we expect the documentation be updated with this so it could help many others , or will it get fixed to use "\\n" as you pointed out it is not encoded properly ?
Once again, thanks a lot for digging out and pointing us to use the right value instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske edited a comment on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske edited a comment on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860786247
> @orpiske
>
> Is the "correct" value "\n" ?
>
> What would be the fix in the code?
Long story short is this: when using special chars for `appendChars` on camel-file, it requires us to escape the parameters (i.e.: `\\n` for appending new lines). However, it seems to me that because the way the configuration works on Kafka connect, the escaping doesn't work. Instead it requires this to be sent encoded as `%0A` from our config so it can be properly interpreted as a newline.
Therefore, I'm wondering if we should somehow fix this in CKC code or just update our documentation to instruct users to use `%0A` and others for special chars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske edited a comment on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske edited a comment on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860786247
> @orpiske
>
> Is the "correct" value "\n" ?
>
> What would be the fix in the code?
Long story short is this: when using special chars for `appendChars` on camel-file, it requires us to escape the parameters `\\n`. However, it seems to me that because the way the configuration works on Kafka connect, the escaping doesn't work. Instead it requires this to be sent encoded as `%0A` from our config.
Therefore, I'm wondering if we should somehow fix this in CKC code or just update our documentation to instruct users to use `%0A` and others for special chars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske commented on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske commented on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860570807
Can you try escaping the new line char, please? Such as '\\n'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske commented on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske commented on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860777427
> @orpiske - thank you so much for this advise. It worked perfectly fine and we were able to get the messages on next line.
> Should we expect the documentation be updated with this so it could help many others , or will it get fixed to use "\n" as you pointed out it is not encoded properly ?
> Once again, thanks a lot for digging out and pointing us to use the right value instead.
I am glad it worked.
@oscerd / @valdar I am a bit conflicted about this one: I'm wondering if we should fix the docs or the code. Any ideas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske edited a comment on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske edited a comment on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860786247
> @orpiske
>
> Is the "correct" value "\n" ?
>
> What would be the fix in the code?
Long story short is this: `appendChars` on camel-file requires `\\n`. However, it seems to me that because the way the configuration works on Kafka connect, it requires this to be sent encoded as `%0A` from our config.
Therefore, I'm wondering if we should somehow fix this in CKC code or just update our documentation to instruct users to use `%0A` and others for special chars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske commented on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske commented on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860459310
Hello,
Please, can you try defining the URLs parameters as properties and let us know the result?
Such as:
```
camel.sink.path.directoryName=/user/app/folder/
camel.sink.endpoint.appendChars=\n
camel.sink.endpoint.autoCreate=true
...
```
The full list of properties is here: https://camel.apache.org/camel-kafka-connector/latest/reference/connectors/camel-file-kafka-sink-connector.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske commented on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske commented on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-866984625
I documented this in multiple places so that users have a greater chance to find out the information. Therefore, I am closing this one. Please refer to PR #1220 for details about the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske commented on issue #1208: CamelFileSink Connector appendChars = \n not working
Posted by GitBox <gi...@apache.org>.
orpiske commented on issue #1208:
URL: https://github.com/apache/camel-kafka-connector/issues/1208#issuecomment-860726614
@ChhatralaNirmal I've run a test with the append char as `%0A` instead of `\\n` as it seems it is not encoded correctly when that is read from the settings. Are you able to validate that on your end, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org