You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Arundhati Bhende <ar...@prudential.com> on 2020/12/25 19:58:48 UTC
Camel S3 Source Connector - value.converter
I am trying to test The Camel S3 Source connector using basic example from this link https://dzone.com/articles/reading-aws-s3-file-content-to-kafka-topic. Below is the file I am uploading to S3.
testfile.txt
add one line
add another line
test the connector
After the connector picks up and processes the file, If I use this for value converter
value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
When I run the consumer, it shows the message processed but the output shows blank.
If I use the String converter as value converter
org.apache.kafka.connect.storage.StringConverter
I get the output in the form of
com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
From the example in the above link, the author doesn't seem to have to do any actions to see the string.
So, why would the S3ObjectConverter show a blank output for me? I can understand the String representation for the object really shows the memory address.
What do I need to look at to fix the output.
Also, is it possible to change the converter to JSON or Avro converter in the configuration file or I need to add either different properties to the configuration or do some processing afterwards?
Thank You
Re: Camel S3 Source Connector - value.converter
Posted by Andrea Cosentino <an...@gmail.com>.
Currently I think so.
Il giorno dom 27 dic 2020 alle ore 17:32 Arundhati Bhende <
arundhati.bhende@prudential.com> ha scritto:
> Thanks. So my only option is the second one I mentioned earlier - write
> an application to read 1 message from this topic and do processing on that
> message and create individual messages in a different topic?
>
> On 12/27/20, 11:13 AM, "Andrea Cosentino" <an...@gmail.com> wrote:
>
> I don't think it is feasible to do that at transformers level, we need
> to
> introduce that at camel level eventually.
>
> Il giorno dom 27 dic 2020 alle ore 15:36 Arundhati Bhende <
> arundhati.bhende@prudential.com> ha scritto:
>
> > In order to overcome - "There is no splitting support actually.
> So it
> > will be something like 1 file
> > - 1 message."
> >
> > Is there a way - like using transformer ( I have never used a
> transformer,
> > but wondering if it will help here ) - to split the message into
> multiple
> > messages? If it is possible, is there an example I could look at to
> define
> > how I can use it?
> >
> > -----Original Message-----
> > From: Andrea Cosentino <an...@gmail.com>
> > Sent: Saturday, December 26, 2020 10:19 AM
> > To: users@camel.apache.org
> > Subject: Re: Camel S3 Source Connector - value.converter
> >
> > You can use any of the dataformat provided by camel or converters
> from
> > kafka connect. So even avro or json.
> >
> > There is no splitting support actually. So it will be something like
> 1 file
> > - 1 message.
> >
> >
> >
> > Il sab 26 dic 2020, 16:08 Arundhati Bhende <
> > arundhati.bhende@prudential.com> ha scritto:
> >
> > > Thank you.
> > >
> > > I will test with setting the autocloseBody to false. In that case,
> > > will I be able to use any valid converter ? What I mean is
> currently
> > > I am testing with a simple text file and hence was trying with
> > > StringConverter, but in reality, those maybe JSON on Avro
> formatted
> > > messages in the files, so will I be able to use those formats as
> > converters?
> > >
> > > Other part of the question - after I get a simple example working
> - a
> > > single file will contain multiple records, one record per line -
> do I
> > > need to set-up any other property to convert each line to a
> separate
> > > message to the topic?
> > >
> > > One doubt / question - to use the camel-aws2-s3 connector, our aws
> set-up
> > > must be at version 2, right? Because, currently it is at v1 and
> do not
> > > know, if we have immediate option of moving to v2
> > >
> > >
> > >
> > > On 12/26/20, 10:00 AM, "Andrea Cosentino" <an...@gmail.com>
> wrote:
> > >
> > > Yes, the approach without url is correct
> > > Autoclosebody will close s3object after consuming the
> payload, so
> > > you want
> > > to get something it must be equal to false, while using the
> > converter.
> > >
> > > My suggestion by the way is using the camel-aws2-s3 connector,
> > > based on ask
> > > v2
> > >
> > > Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> > > arundhati.bhende@prudential.com>
> > > ha scritto:
> > >
> > > > This is the full configuration that I have used. I am been
> > > experimenting
> > > > with changing the value.converter between StringConverter and
> > > > S3ObjectConverter - otherwise everything is same.
> > > >
> > > >
> > > >
> > > > DATA=$( cat << EOF
> > > > {
> > > > "connector.class":
> > > >
> "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > > > "key.converter":
> > > > "org.apache.kafka.connect.storage.StringConverter",
> > > > "value.converter":
> > > > "org.apache.kafka.connect.storage.StringConverter",
> > > > "camel.source.maxPollDuration": "10000",
> > > > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > > > "camel.source.path.bucketNameOrArn": "
> push-json-poc",
> > > > "camel.component.aws-s3.region": "US_EAST_1",
> > > > "tasks.max": "1",
> > > > "camel.source.endpoint.useIAMCredentials": "true",
> > > > "camel.source.endpoint.autocloseBody": "true"
> > > > }
> > > > EOF
> > > > )
> > > > # "value.converter":
> > > >
> > "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> > > >
> > > >
> > > > This is official one from the link in the other email.
> > > > name=CamelAWSS3SourceConnector
> > > >
> > > >
> > >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > > >
> key.converter=org.apache.kafka.connect.storage.StringConverter
> > > >
> > > >
> > >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> > > >
> > > > camel.source.maxPollDuration=10000
> > > >
> > > > topics=mytopic
> > > >
> > > >
> camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> > > >
> > > > camel.component.aws-s3.access-key=xxxx
> > > > camel.component.aws-s3.secret-key=yyyy
> > > > camel.component.aws-s3.region=EU_WEST_1
> > > >
> > > >
> > > >
> > > >
> > > > I can see two differences -
> > > > 1. How the AWS credentials are set-up
> > > > 2. In my configuration
> > > > I am using camel.source.path.bucketNameOrArn instead
> of
> > > > camel.source.url
> > > > Because I was getting error that bucketNameOrArn
> must be
> > > defined
> > > >
> > > > In my configuration - autocloseBody is set true to
> whereas
> > > in the
> > > > default configuration it is set to false
> > > >
> > > > How does autocloseBody work? What impact does it
> have when
> > > > setting it to true vs. false?
> > > >
> > > >
> > > > Thank you
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Andrea Cosentino <an...@gmail.com>
> > > > Sent: Saturday, December 26, 2020 3:59 AM
> > > > To: users@camel.apache.org
> > > > Subject: Re: Camel S3 Source Connector - value.converter
> > > >
> > > > Can you show your full configuration?
> > > >
> > > > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > > > arundhati.bhende@prudential.com> ha scritto:
> > > >
> > > > > I am trying to test The Camel S3 Source connector using
> basic
> > > example
> > > > > from this link
> > > > >
> > > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C2e13a12f628e4014a5ab08d8aa824c14%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637446823884910393%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UQd42qkez%2Fz0NblRLpqFlDru8Nj5duluzGQ8i3uVe%2Bg%3D&reserved=0
> > > > .
> > > > > Below is the file I am uploading to S3.
> > > > >
> > > > > testfile.txt
> > > > > add one line
> > > > > add another line
> > > > > test the connector
> > > > >
> > > > >
> > > > > After the connector picks up and processes the file, If I
> use
> > > this for
> > > > > value converter
> > > > >
> > > > >
> > > > >
> > >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > > > > ectConverter
> > > > >
> > > > > When I run the consumer, it shows the message processed but
> > > the output
> > > > > shows blank.
> > > > >
> > > > > If I use the String converter as value converter
> > > > >
> > > > > org.apache.kafka.connect.storage.StringConverter
> > > > >
> > > > > I get the output in the form of
> > > > >
> > > > >
> com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > > > >
> com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> > > > >
> > > > > From the example in the above link, the author doesn't
> seem to
> > > have to
> > > > > do any actions to see the string.
> > > > >
> > > > > So, why would the S3ObjectConverter show a blank output for
> > > me? I can
> > > > > understand the String representation for the object really
> > > shows the
> > > > > memory address.
> > > > >
> > > > > What do I need to look at to fix the output.
> > > > >
> > > > > Also, is it possible to change the converter to JSON or
> Avro
> > > converter
> > > > > in the configuration file or I need to add either different
> > > properties
> > > > > to the configuration or do some processing afterwards?
> > > > >
> > > > > Thank You
> > > > >
> > > > >
> > > >
> > >
> > >
> >
>
>
Re: Camel S3 Source Connector - value.converter
Posted by Arundhati Bhende <ar...@prudential.com>.
Thanks. So my only option is the second one I mentioned earlier - write an application to read 1 message from this topic and do processing on that message and create individual messages in a different topic?
On 12/27/20, 11:13 AM, "Andrea Cosentino" <an...@gmail.com> wrote:
I don't think it is feasible to do that at transformers level, we need to
introduce that at camel level eventually.
Il giorno dom 27 dic 2020 alle ore 15:36 Arundhati Bhende <
arundhati.bhende@prudential.com> ha scritto:
> In order to overcome - "There is no splitting support actually. So it
> will be something like 1 file
> - 1 message."
>
> Is there a way - like using transformer ( I have never used a transformer,
> but wondering if it will help here ) - to split the message into multiple
> messages? If it is possible, is there an example I could look at to define
> how I can use it?
>
> -----Original Message-----
> From: Andrea Cosentino <an...@gmail.com>
> Sent: Saturday, December 26, 2020 10:19 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> You can use any of the dataformat provided by camel or converters from
> kafka connect. So even avro or json.
>
> There is no splitting support actually. So it will be something like 1 file
> - 1 message.
>
>
>
> Il sab 26 dic 2020, 16:08 Arundhati Bhende <
> arundhati.bhende@prudential.com> ha scritto:
>
> > Thank you.
> >
> > I will test with setting the autocloseBody to false. In that case,
> > will I be able to use any valid converter ? What I mean is currently
> > I am testing with a simple text file and hence was trying with
> > StringConverter, but in reality, those maybe JSON on Avro formatted
> > messages in the files, so will I be able to use those formats as
> converters?
> >
> > Other part of the question - after I get a simple example working - a
> > single file will contain multiple records, one record per line - do I
> > need to set-up any other property to convert each line to a separate
> > message to the topic?
> >
> > One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> > must be at version 2, right? Because, currently it is at v1 and do not
> > know, if we have immediate option of moving to v2
> >
> >
> >
> > On 12/26/20, 10:00 AM, "Andrea Cosentino" <an...@gmail.com> wrote:
> >
> > Yes, the approach without url is correct
> > Autoclosebody will close s3object after consuming the payload, so
> > you want
> > to get something it must be equal to false, while using the
> converter.
> >
> > My suggestion by the way is using the camel-aws2-s3 connector,
> > based on ask
> > v2
> >
> > Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> > arundhati.bhende@prudential.com>
> > ha scritto:
> >
> > > This is the full configuration that I have used. I am been
> > experimenting
> > > with changing the value.converter between StringConverter and
> > > S3ObjectConverter - otherwise everything is same.
> > >
> > >
> > >
> > > DATA=$( cat << EOF
> > > {
> > > "connector.class":
> > > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > > "key.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "value.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "camel.source.maxPollDuration": "10000",
> > > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > > "camel.component.aws-s3.region": "US_EAST_1",
> > > "tasks.max": "1",
> > > "camel.source.endpoint.useIAMCredentials": "true",
> > > "camel.source.endpoint.autocloseBody": "true"
> > > }
> > > EOF
> > > )
> > > # "value.converter":
> > >
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> > >
> > >
> > > This is official one from the link in the other email.
> > > name=CamelAWSS3SourceConnector
> > >
> > >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > > key.converter=org.apache.kafka.connect.storage.StringConverter
> > >
> > >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> > >
> > > camel.source.maxPollDuration=10000
> > >
> > > topics=mytopic
> > >
> > > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> > >
> > > camel.component.aws-s3.access-key=xxxx
> > > camel.component.aws-s3.secret-key=yyyy
> > > camel.component.aws-s3.region=EU_WEST_1
> > >
> > >
> > >
> > >
> > > I can see two differences -
> > > 1. How the AWS credentials are set-up
> > > 2. In my configuration
> > > I am using camel.source.path.bucketNameOrArn instead of
> > > camel.source.url
> > > Because I was getting error that bucketNameOrArn must be
> > defined
> > >
> > > In my configuration - autocloseBody is set true to whereas
> > in the
> > > default configuration it is set to false
> > >
> > > How does autocloseBody work? What impact does it have when
> > > setting it to true vs. false?
> > >
> > >
> > > Thank you
> > >
> > >
> > > -----Original Message-----
> > > From: Andrea Cosentino <an...@gmail.com>
> > > Sent: Saturday, December 26, 2020 3:59 AM
> > > To: users@camel.apache.org
> > > Subject: Re: Camel S3 Source Connector - value.converter
> > >
> > > Can you show your full configuration?
> > >
> > > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > > arundhati.bhende@prudential.com> ha scritto:
> > >
> > > > I am trying to test The Camel S3 Source connector using basic
> > example
> > > > from this link
> > > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C2e13a12f628e4014a5ab08d8aa824c14%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637446823884910393%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UQd42qkez%2Fz0NblRLpqFlDru8Nj5duluzGQ8i3uVe%2Bg%3D&reserved=0
> > > .
> > > > Below is the file I am uploading to S3.
> > > >
> > > > testfile.txt
> > > > add one line
> > > > add another line
> > > > test the connector
> > > >
> > > >
> > > > After the connector picks up and processes the file, If I use
> > this for
> > > > value converter
> > > >
> > > >
> > > >
> > value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > > > ectConverter
> > > >
> > > > When I run the consumer, it shows the message processed but
> > the output
> > > > shows blank.
> > > >
> > > > If I use the String converter as value converter
> > > >
> > > > org.apache.kafka.connect.storage.StringConverter
> > > >
> > > > I get the output in the form of
> > > >
> > > > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > > > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> > > >
> > > > From the example in the above link, the author doesn't seem to
> > have to
> > > > do any actions to see the string.
> > > >
> > > > So, why would the S3ObjectConverter show a blank output for
> > me? I can
> > > > understand the String representation for the object really
> > shows the
> > > > memory address.
> > > >
> > > > What do I need to look at to fix the output.
> > > >
> > > > Also, is it possible to change the converter to JSON or Avro
> > converter
> > > > in the configuration file or I need to add either different
> > properties
> > > > to the configuration or do some processing afterwards?
> > > >
> > > > Thank You
> > > >
> > > >
> > >
> >
> >
>
Re: Camel S3 Source Connector - value.converter
Posted by Andrea Cosentino <an...@gmail.com>.
I don't think it is feasible to do that at transformers level, we need to
introduce that at camel level eventually.
Il giorno dom 27 dic 2020 alle ore 15:36 Arundhati Bhende <
arundhati.bhende@prudential.com> ha scritto:
> In order to overcome - "There is no splitting support actually. So it
> will be something like 1 file
> - 1 message."
>
> Is there a way - like using transformer ( I have never used a transformer,
> but wondering if it will help here ) - to split the message into multiple
> messages? If it is possible, is there an example I could look at to define
> how I can use it?
>
> -----Original Message-----
> From: Andrea Cosentino <an...@gmail.com>
> Sent: Saturday, December 26, 2020 10:19 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> You can use any of the dataformat provided by camel or converters from
> kafka connect. So even avro or json.
>
> There is no splitting support actually. So it will be something like 1 file
> - 1 message.
>
>
>
> Il sab 26 dic 2020, 16:08 Arundhati Bhende <
> arundhati.bhende@prudential.com> ha scritto:
>
> > Thank you.
> >
> > I will test with setting the autocloseBody to false. In that case,
> > will I be able to use any valid converter ? What I mean is currently
> > I am testing with a simple text file and hence was trying with
> > StringConverter, but in reality, those maybe JSON on Avro formatted
> > messages in the files, so will I be able to use those formats as
> converters?
> >
> > Other part of the question - after I get a simple example working - a
> > single file will contain multiple records, one record per line - do I
> > need to set-up any other property to convert each line to a separate
> > message to the topic?
> >
> > One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> > must be at version 2, right? Because, currently it is at v1 and do not
> > know, if we have immediate option of moving to v2
> >
> >
> >
> > On 12/26/20, 10:00 AM, "Andrea Cosentino" <an...@gmail.com> wrote:
> >
> > Yes, the approach without url is correct
> > Autoclosebody will close s3object after consuming the payload, so
> > you want
> > to get something it must be equal to false, while using the
> converter.
> >
> > My suggestion by the way is using the camel-aws2-s3 connector,
> > based on ask
> > v2
> >
> > Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> > arundhati.bhende@prudential.com>
> > ha scritto:
> >
> > > This is the full configuration that I have used. I am been
> > experimenting
> > > with changing the value.converter between StringConverter and
> > > S3ObjectConverter - otherwise everything is same.
> > >
> > >
> > >
> > > DATA=$( cat << EOF
> > > {
> > > "connector.class":
> > > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > > "key.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "value.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "camel.source.maxPollDuration": "10000",
> > > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > > "camel.component.aws-s3.region": "US_EAST_1",
> > > "tasks.max": "1",
> > > "camel.source.endpoint.useIAMCredentials": "true",
> > > "camel.source.endpoint.autocloseBody": "true"
> > > }
> > > EOF
> > > )
> > > # "value.converter":
> > >
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> > >
> > >
> > > This is official one from the link in the other email.
> > > name=CamelAWSS3SourceConnector
> > >
> > >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > > key.converter=org.apache.kafka.connect.storage.StringConverter
> > >
> > >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> > >
> > > camel.source.maxPollDuration=10000
> > >
> > > topics=mytopic
> > >
> > > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> > >
> > > camel.component.aws-s3.access-key=xxxx
> > > camel.component.aws-s3.secret-key=yyyy
> > > camel.component.aws-s3.region=EU_WEST_1
> > >
> > >
> > >
> > >
> > > I can see two differences -
> > > 1. How the AWS credentials are set-up
> > > 2. In my configuration
> > > I am using camel.source.path.bucketNameOrArn instead of
> > > camel.source.url
> > > Because I was getting error that bucketNameOrArn must be
> > defined
> > >
> > > In my configuration - autocloseBody is set true to whereas
> > in the
> > > default configuration it is set to false
> > >
> > > How does autocloseBody work? What impact does it have when
> > > setting it to true vs. false?
> > >
> > >
> > > Thank you
> > >
> > >
> > > -----Original Message-----
> > > From: Andrea Cosentino <an...@gmail.com>
> > > Sent: Saturday, December 26, 2020 3:59 AM
> > > To: users@camel.apache.org
> > > Subject: Re: Camel S3 Source Connector - value.converter
> > >
> > > Can you show your full configuration?
> > >
> > > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > > arundhati.bhende@prudential.com> ha scritto:
> > >
> > > > I am trying to test The Camel S3 Source connector using basic
> > example
> > > > from this link
> > > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C885aea82555a4e4e1b2d08d8a9b1aa84%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445927846361197%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yr9urZgFayjVnAF72kDV87GPr1TMKzPQSj%2BrKyvD0wo%3D&reserved=0
> > > .
> > > > Below is the file I am uploading to S3.
> > > >
> > > > testfile.txt
> > > > add one line
> > > > add another line
> > > > test the connector
> > > >
> > > >
> > > > After the connector picks up and processes the file, If I use
> > this for
> > > > value converter
> > > >
> > > >
> > > >
> > value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > > > ectConverter
> > > >
> > > > When I run the consumer, it shows the message processed but
> > the output
> > > > shows blank.
> > > >
> > > > If I use the String converter as value converter
> > > >
> > > > org.apache.kafka.connect.storage.StringConverter
> > > >
> > > > I get the output in the form of
> > > >
> > > > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > > > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> > > >
> > > > From the example in the above link, the author doesn't seem to
> > have to
> > > > do any actions to see the string.
> > > >
> > > > So, why would the S3ObjectConverter show a blank output for
> > me? I can
> > > > understand the String representation for the object really
> > shows the
> > > > memory address.
> > > >
> > > > What do I need to look at to fix the output.
> > > >
> > > > Also, is it possible to change the converter to JSON or Avro
> > converter
> > > > in the configuration file or I need to add either different
> > properties
> > > > to the configuration or do some processing afterwards?
> > > >
> > > > Thank You
> > > >
> > > >
> > >
> >
> >
>
RE: Camel S3 Source Connector - value.converter
Posted by Arundhati Bhende <ar...@prudential.com>.
In order to overcome - "There is no splitting support actually. So it will be something like 1 file
- 1 message."
Is there a way - like using transformer ( I have never used a transformer, but wondering if it will help here ) - to split the message into multiple messages? If it is possible, is there an example I could look at to define how I can use it?
-----Original Message-----
From: Andrea Cosentino <an...@gmail.com>
Sent: Saturday, December 26, 2020 10:19 AM
To: users@camel.apache.org
Subject: Re: Camel S3 Source Connector - value.converter
You can use any of the dataformat provided by camel or converters from kafka connect. So even avro or json.
There is no splitting support actually. So it will be something like 1 file
- 1 message.
Il sab 26 dic 2020, 16:08 Arundhati Bhende <ar...@prudential.com> ha scritto:
> Thank you.
>
> I will test with setting the autocloseBody to false. In that case,
> will I be able to use any valid converter ? What I mean is currently
> I am testing with a simple text file and hence was trying with
> StringConverter, but in reality, those maybe JSON on Avro formatted
> messages in the files, so will I be able to use those formats as converters?
>
> Other part of the question - after I get a simple example working - a
> single file will contain multiple records, one record per line - do I
> need to set-up any other property to convert each line to a separate
> message to the topic?
>
> One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> must be at version 2, right? Because, currently it is at v1 and do not
> know, if we have immediate option of moving to v2
>
>
>
> On 12/26/20, 10:00 AM, "Andrea Cosentino" <an...@gmail.com> wrote:
>
> Yes, the approach without url is correct
> Autoclosebody will close s3object after consuming the payload, so
> you want
> to get something it must be equal to false, while using the converter.
>
> My suggestion by the way is using the camel-aws2-s3 connector,
> based on ask
> v2
>
> Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> arundhati.bhende@prudential.com>
> ha scritto:
>
> > This is the full configuration that I have used. I am been
> experimenting
> > with changing the value.converter between StringConverter and
> > S3ObjectConverter - otherwise everything is same.
> >
> >
> >
> > DATA=$( cat << EOF
> > {
> > "connector.class":
> > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > "key.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "value.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "camel.source.maxPollDuration": "10000",
> > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > "camel.component.aws-s3.region": "US_EAST_1",
> > "tasks.max": "1",
> > "camel.source.endpoint.useIAMCredentials": "true",
> > "camel.source.endpoint.autocloseBody": "true"
> > }
> > EOF
> > )
> > # "value.converter":
> > "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> >
> >
> > This is official one from the link in the other email.
> > name=CamelAWSS3SourceConnector
> >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > key.converter=org.apache.kafka.connect.storage.StringConverter
> >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> >
> > camel.source.maxPollDuration=10000
> >
> > topics=mytopic
> >
> > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> >
> > camel.component.aws-s3.access-key=xxxx
> > camel.component.aws-s3.secret-key=yyyy
> > camel.component.aws-s3.region=EU_WEST_1
> >
> >
> >
> >
> > I can see two differences -
> > 1. How the AWS credentials are set-up
> > 2. In my configuration
> > I am using camel.source.path.bucketNameOrArn instead of
> > camel.source.url
> > Because I was getting error that bucketNameOrArn must be
> defined
> >
> > In my configuration - autocloseBody is set true to whereas
> in the
> > default configuration it is set to false
> >
> > How does autocloseBody work? What impact does it have when
> > setting it to true vs. false?
> >
> >
> > Thank you
> >
> >
> > -----Original Message-----
> > From: Andrea Cosentino <an...@gmail.com>
> > Sent: Saturday, December 26, 2020 3:59 AM
> > To: users@camel.apache.org
> > Subject: Re: Camel S3 Source Connector - value.converter
> >
> > Can you show your full configuration?
> >
> > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > arundhati.bhende@prudential.com> ha scritto:
> >
> > > I am trying to test The Camel S3 Source connector using basic
> example
> > > from this link
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C885aea82555a4e4e1b2d08d8a9b1aa84%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445927846361197%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yr9urZgFayjVnAF72kDV87GPr1TMKzPQSj%2BrKyvD0wo%3D&reserved=0
> > .
> > > Below is the file I am uploading to S3.
> > >
> > > testfile.txt
> > > add one line
> > > add another line
> > > test the connector
> > >
> > >
> > > After the connector picks up and processes the file, If I use
> this for
> > > value converter
> > >
> > >
> > >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > > ectConverter
> > >
> > > When I run the consumer, it shows the message processed but
> the output
> > > shows blank.
> > >
> > > If I use the String converter as value converter
> > >
> > > org.apache.kafka.connect.storage.StringConverter
> > >
> > > I get the output in the form of
> > >
> > > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> > >
> > > From the example in the above link, the author doesn't seem to
> have to
> > > do any actions to see the string.
> > >
> > > So, why would the S3ObjectConverter show a blank output for
> me? I can
> > > understand the String representation for the object really
> shows the
> > > memory address.
> > >
> > > What do I need to look at to fix the output.
> > >
> > > Also, is it possible to change the converter to JSON or Avro
> converter
> > > in the configuration file or I need to add either different
> properties
> > > to the configuration or do some processing afterwards?
> > >
> > > Thank You
> > >
> > >
> >
>
>
Re: Camel S3 Source Connector - value.converter
Posted by Andrea Cosentino <an...@gmail.com>.
Yes
Il sab 26 dic 2020, 16:25 Arundhati Bhende <ar...@prudential.com>
ha scritto:
> Thanks again. Will test with these combinations of different value
> converters after I am able to get the avro format files.
>
> In the previous response, you had mentioned it is better to use aws2-s3
> version of connector - So, in order to do so, to use the camel-aws2-s3
> connector, our aws set-up must be at version 2, right? Because, currently
> it is at v1 and do not know, if we have immediate option of moving to v2
>
> -----Original Message-----
> From: Andrea Cosentino <an...@gmail.com>
> Sent: Saturday, December 26, 2020 10:19 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> You can use any of the dataformat provided by camel or converters from
> kafka connect. So even avro or json.
>
> There is no splitting support actually. So it will be something like 1 file
> - 1 message.
>
>
>
> Il sab 26 dic 2020, 16:08 Arundhati Bhende <
> arundhati.bhende@prudential.com> ha scritto:
>
> > Thank you.
> >
> > I will test with setting the autocloseBody to false. In that case,
> > will I be able to use any valid converter ? What I mean is currently
> > I am testing with a simple text file and hence was trying with
> > StringConverter, but in reality, those maybe JSON on Avro formatted
> > messages in the files, so will I be able to use those formats as
> converters?
> >
> > Other part of the question - after I get a simple example working - a
> > single file will contain multiple records, one record per line - do I
> > need to set-up any other property to convert each line to a separate
> > message to the topic?
> >
> > One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> > must be at version 2, right? Because, currently it is at v1 and do not
> > know, if we have immediate option of moving to v2
> >
> >
> >
> > On 12/26/20, 10:00 AM, "Andrea Cosentino" <an...@gmail.com> wrote:
> >
> > Yes, the approach without url is correct
> > Autoclosebody will close s3object after consuming the payload, so
> > you want
> > to get something it must be equal to false, while using the
> converter.
> >
> > My suggestion by the way is using the camel-aws2-s3 connector,
> > based on ask
> > v2
> >
> > Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> > arundhati.bhende@prudential.com>
> > ha scritto:
> >
> > > This is the full configuration that I have used. I am been
> > experimenting
> > > with changing the value.converter between StringConverter and
> > > S3ObjectConverter - otherwise everything is same.
> > >
> > >
> > >
> > > DATA=$( cat << EOF
> > > {
> > > "connector.class":
> > > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > > "key.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "value.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "camel.source.maxPollDuration": "10000",
> > > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > > "camel.component.aws-s3.region": "US_EAST_1",
> > > "tasks.max": "1",
> > > "camel.source.endpoint.useIAMCredentials": "true",
> > > "camel.source.endpoint.autocloseBody": "true"
> > > }
> > > EOF
> > > )
> > > # "value.converter":
> > >
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> > >
> > >
> > > This is official one from the link in the other email.
> > > name=CamelAWSS3SourceConnector
> > >
> > >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > > key.converter=org.apache.kafka.connect.storage.StringConverter
> > >
> > >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> > >
> > > camel.source.maxPollDuration=10000
> > >
> > > topics=mytopic
> > >
> > > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> > >
> > > camel.component.aws-s3.access-key=xxxx
> > > camel.component.aws-s3.secret-key=yyyy
> > > camel.component.aws-s3.region=EU_WEST_1
> > >
> > >
> > >
> > >
> > > I can see two differences -
> > > 1. How the AWS credentials are set-up
> > > 2. In my configuration
> > > I am using camel.source.path.bucketNameOrArn instead of
> > > camel.source.url
> > > Because I was getting error that bucketNameOrArn must be
> > defined
> > >
> > > In my configuration - autocloseBody is set true to whereas
> > in the
> > > default configuration it is set to false
> > >
> > > How does autocloseBody work? What impact does it have when
> > > setting it to true vs. false?
> > >
> > >
> > > Thank you
> > >
> > >
> > > -----Original Message-----
> > > From: Andrea Cosentino <an...@gmail.com>
> > > Sent: Saturday, December 26, 2020 3:59 AM
> > > To: users@camel.apache.org
> > > Subject: Re: Camel S3 Source Connector - value.converter
> > >
> > > Can you show your full configuration?
> > >
> > > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > > arundhati.bhende@prudential.com> ha scritto:
> > >
> > > > I am trying to test The Camel S3 Source connector using basic
> > example
> > > > from this link
> > > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C885aea82555a4e4e1b2d08d8a9b1aa84%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445927846361197%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yr9urZgFayjVnAF72kDV87GPr1TMKzPQSj%2BrKyvD0wo%3D&reserved=0
> > > .
> > > > Below is the file I am uploading to S3.
> > > >
> > > > testfile.txt
> > > > add one line
> > > > add another line
> > > > test the connector
> > > >
> > > >
> > > > After the connector picks up and processes the file, If I use
> > this for
> > > > value converter
> > > >
> > > >
> > > >
> > value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > > > ectConverter
> > > >
> > > > When I run the consumer, it shows the message processed but
> > the output
> > > > shows blank.
> > > >
> > > > If I use the String converter as value converter
> > > >
> > > > org.apache.kafka.connect.storage.StringConverter
> > > >
> > > > I get the output in the form of
> > > >
> > > > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > > > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> > > >
> > > > From the example in the above link, the author doesn't seem to
> > have to
> > > > do any actions to see the string.
> > > >
> > > > So, why would the S3ObjectConverter show a blank output for
> > me? I can
> > > > understand the String representation for the object really
> > shows the
> > > > memory address.
> > > >
> > > > What do I need to look at to fix the output.
> > > >
> > > > Also, is it possible to change the converter to JSON or Avro
> > converter
> > > > in the configuration file or I need to add either different
> > properties
> > > > to the configuration or do some processing afterwards?
> > > >
> > > > Thank You
> > > >
> > > >
> > >
> >
> >
>
RE: Camel S3 Source Connector - value.converter
Posted by Arundhati Bhende <ar...@prudential.com>.
Thanks again. Will test with these combinations of different value converters after I am able to get the avro format files.
In the previous response, you had mentioned it is better to use aws2-s3 version of connector - So, in order to do so, to use the camel-aws2-s3 connector, our aws set-up must be at version 2, right? Because, currently it is at v1 and do not know, if we have immediate option of moving to v2
-----Original Message-----
From: Andrea Cosentino <an...@gmail.com>
Sent: Saturday, December 26, 2020 10:19 AM
To: users@camel.apache.org
Subject: Re: Camel S3 Source Connector - value.converter
You can use any of the dataformat provided by camel or converters from kafka connect. So even avro or json.
There is no splitting support actually. So it will be something like 1 file
- 1 message.
Il sab 26 dic 2020, 16:08 Arundhati Bhende <ar...@prudential.com> ha scritto:
> Thank you.
>
> I will test with setting the autocloseBody to false. In that case,
> will I be able to use any valid converter ? What I mean is currently
> I am testing with a simple text file and hence was trying with
> StringConverter, but in reality, those maybe JSON on Avro formatted
> messages in the files, so will I be able to use those formats as converters?
>
> Other part of the question - after I get a simple example working - a
> single file will contain multiple records, one record per line - do I
> need to set-up any other property to convert each line to a separate
> message to the topic?
>
> One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> must be at version 2, right? Because, currently it is at v1 and do not
> know, if we have immediate option of moving to v2
>
>
>
> On 12/26/20, 10:00 AM, "Andrea Cosentino" <an...@gmail.com> wrote:
>
> Yes, the approach without url is correct
> Autoclosebody will close s3object after consuming the payload, so
> you want
> to get something it must be equal to false, while using the converter.
>
> My suggestion by the way is using the camel-aws2-s3 connector,
> based on ask
> v2
>
> Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> arundhati.bhende@prudential.com>
> ha scritto:
>
> > This is the full configuration that I have used. I am been
> experimenting
> > with changing the value.converter between StringConverter and
> > S3ObjectConverter - otherwise everything is same.
> >
> >
> >
> > DATA=$( cat << EOF
> > {
> > "connector.class":
> > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > "key.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "value.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "camel.source.maxPollDuration": "10000",
> > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > "camel.component.aws-s3.region": "US_EAST_1",
> > "tasks.max": "1",
> > "camel.source.endpoint.useIAMCredentials": "true",
> > "camel.source.endpoint.autocloseBody": "true"
> > }
> > EOF
> > )
> > # "value.converter":
> > "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> >
> >
> > This is official one from the link in the other email.
> > name=CamelAWSS3SourceConnector
> >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > key.converter=org.apache.kafka.connect.storage.StringConverter
> >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> >
> > camel.source.maxPollDuration=10000
> >
> > topics=mytopic
> >
> > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> >
> > camel.component.aws-s3.access-key=xxxx
> > camel.component.aws-s3.secret-key=yyyy
> > camel.component.aws-s3.region=EU_WEST_1
> >
> >
> >
> >
> > I can see two differences -
> > 1. How the AWS credentials are set-up
> > 2. In my configuration
> > I am using camel.source.path.bucketNameOrArn instead of
> > camel.source.url
> > Because I was getting error that bucketNameOrArn must be
> defined
> >
> > In my configuration - autocloseBody is set true to whereas
> in the
> > default configuration it is set to false
> >
> > How does autocloseBody work? What impact does it have when
> > setting it to true vs. false?
> >
> >
> > Thank you
> >
> >
> > -----Original Message-----
> > From: Andrea Cosentino <an...@gmail.com>
> > Sent: Saturday, December 26, 2020 3:59 AM
> > To: users@camel.apache.org
> > Subject: Re: Camel S3 Source Connector - value.converter
> >
> > Can you show your full configuration?
> >
> > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > arundhati.bhende@prudential.com> ha scritto:
> >
> > > I am trying to test The Camel S3 Source connector using basic
> example
> > > from this link
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C885aea82555a4e4e1b2d08d8a9b1aa84%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445927846361197%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yr9urZgFayjVnAF72kDV87GPr1TMKzPQSj%2BrKyvD0wo%3D&reserved=0
> > .
> > > Below is the file I am uploading to S3.
> > >
> > > testfile.txt
> > > add one line
> > > add another line
> > > test the connector
> > >
> > >
> > > After the connector picks up and processes the file, If I use
> this for
> > > value converter
> > >
> > >
> > >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > > ectConverter
> > >
> > > When I run the consumer, it shows the message processed but
> the output
> > > shows blank.
> > >
> > > If I use the String converter as value converter
> > >
> > > org.apache.kafka.connect.storage.StringConverter
> > >
> > > I get the output in the form of
> > >
> > > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> > >
> > > From the example in the above link, the author doesn't seem to
> have to
> > > do any actions to see the string.
> > >
> > > So, why would the S3ObjectConverter show a blank output for
> me? I can
> > > understand the String representation for the object really
> shows the
> > > memory address.
> > >
> > > What do I need to look at to fix the output.
> > >
> > > Also, is it possible to change the converter to JSON or Avro
> converter
> > > in the configuration file or I need to add either different
> properties
> > > to the configuration or do some processing afterwards?
> > >
> > > Thank You
> > >
> > >
> >
>
>
Re: Camel S3 Source Connector - value.converter
Posted by Andrea Cosentino <an...@gmail.com>.
You can use any of the dataformat provided by camel or converters from
kafka connect. So even avro or json.
There is no splitting support actually. So it will be something like 1 file
- 1 message.
Il sab 26 dic 2020, 16:08 Arundhati Bhende <ar...@prudential.com>
ha scritto:
> Thank you.
>
> I will test with setting the autocloseBody to false. In that case, will I
> be able to use any valid converter ? What I mean is currently I am testing
> with a simple text file and hence was trying with StringConverter, but in
> reality, those maybe JSON on Avro formatted messages in the files, so will
> I be able to use those formats as converters?
>
> Other part of the question - after I get a simple example working - a
> single file will contain multiple records, one record per line - do I need
> to set-up any other property to convert each line to a separate message to
> the topic?
>
> One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> must be at version 2, right? Because, currently it is at v1 and do not
> know, if we have immediate option of moving to v2
>
>
>
> On 12/26/20, 10:00 AM, "Andrea Cosentino" <an...@gmail.com> wrote:
>
> Yes, the approach without url is correct
> Autoclosebody will close s3object after consuming the payload, so you
> want
> to get something it must be equal to false, while using the converter.
>
> My suggestion by the way is using the camel-aws2-s3 connector, based
> on ask
> v2
>
> Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> arundhati.bhende@prudential.com>
> ha scritto:
>
> > This is the full configuration that I have used. I am been
> experimenting
> > with changing the value.converter between StringConverter and
> > S3ObjectConverter - otherwise everything is same.
> >
> >
> >
> > DATA=$( cat << EOF
> > {
> > "connector.class":
> > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > "key.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "value.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "camel.source.maxPollDuration": "10000",
> > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > "camel.component.aws-s3.region": "US_EAST_1",
> > "tasks.max": "1",
> > "camel.source.endpoint.useIAMCredentials": "true",
> > "camel.source.endpoint.autocloseBody": "true"
> > }
> > EOF
> > )
> > # "value.converter":
> > "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> >
> >
> > This is official one from the link in the other email.
> > name=CamelAWSS3SourceConnector
> >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > key.converter=org.apache.kafka.connect.storage.StringConverter
> >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> >
> > camel.source.maxPollDuration=10000
> >
> > topics=mytopic
> >
> > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> >
> > camel.component.aws-s3.access-key=xxxx
> > camel.component.aws-s3.secret-key=yyyy
> > camel.component.aws-s3.region=EU_WEST_1
> >
> >
> >
> >
> > I can see two differences -
> > 1. How the AWS credentials are set-up
> > 2. In my configuration
> > I am using camel.source.path.bucketNameOrArn instead of
> > camel.source.url
> > Because I was getting error that bucketNameOrArn must be
> defined
> >
> > In my configuration - autocloseBody is set true to whereas
> in the
> > default configuration it is set to false
> >
> > How does autocloseBody work? What impact does it have when
> > setting it to true vs. false?
> >
> >
> > Thank you
> >
> >
> > -----Original Message-----
> > From: Andrea Cosentino <an...@gmail.com>
> > Sent: Saturday, December 26, 2020 3:59 AM
> > To: users@camel.apache.org
> > Subject: Re: Camel S3 Source Connector - value.converter
> >
> > Can you show your full configuration?
> >
> > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > arundhati.bhende@prudential.com> ha scritto:
> >
> > > I am trying to test The Camel S3 Source connector using basic
> example
> > > from this link
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7Cf7b74faf6add4a4f39f808d8a9aeed82%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445916065455084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2ISA5ZwtgVUwF%2BvydKewwZbMB5YVr9HNem1f6r4lDB8%3D&reserved=0
> > .
> > > Below is the file I am uploading to S3.
> > >
> > > testfile.txt
> > > add one line
> > > add another line
> > > test the connector
> > >
> > >
> > > After the connector picks up and processes the file, If I use this
> for
> > > value converter
> > >
> > >
> > >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > > ectConverter
> > >
> > > When I run the consumer, it shows the message processed but the
> output
> > > shows blank.
> > >
> > > If I use the String converter as value converter
> > >
> > > org.apache.kafka.connect.storage.StringConverter
> > >
> > > I get the output in the form of
> > >
> > > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> > >
> > > From the example in the above link, the author doesn't seem to
> have to
> > > do any actions to see the string.
> > >
> > > So, why would the S3ObjectConverter show a blank output for me? I
> can
> > > understand the String representation for the object really shows
> the
> > > memory address.
> > >
> > > What do I need to look at to fix the output.
> > >
> > > Also, is it possible to change the converter to JSON or Avro
> converter
> > > in the configuration file or I need to add either different
> properties
> > > to the configuration or do some processing afterwards?
> > >
> > > Thank You
> > >
> > >
> >
>
>
Re: Camel S3 Source Connector - value.converter
Posted by Arundhati Bhende <ar...@prudential.com>.
Thank you.
I will test with setting the autocloseBody to false. In that case, will I be able to use any valid converter ? What I mean is currently I am testing with a simple text file and hence was trying with StringConverter, but in reality, those maybe JSON on Avro formatted messages in the files, so will I be able to use those formats as converters?
Other part of the question - after I get a simple example working - a single file will contain multiple records, one record per line - do I need to set-up any other property to convert each line to a separate message to the topic?
One doubt / question - to use the camel-aws2-s3 connector, our aws set-up must be at version 2, right? Because, currently it is at v1 and do not know, if we have immediate option of moving to v2
On 12/26/20, 10:00 AM, "Andrea Cosentino" <an...@gmail.com> wrote:
Yes, the approach without url is correct
Autoclosebody will close s3object after consuming the payload, so you want
to get something it must be equal to false, while using the converter.
My suggestion by the way is using the camel-aws2-s3 connector, based on ask
v2
Il sab 26 dic 2020, 15:53 Arundhati Bhende <ar...@prudential.com>
ha scritto:
> This is the full configuration that I have used. I am been experimenting
> with changing the value.converter between StringConverter and
> S3ObjectConverter - otherwise everything is same.
>
>
>
> DATA=$( cat << EOF
> {
> "connector.class":
> "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> "key.converter":
> "org.apache.kafka.connect.storage.StringConverter",
> "value.converter":
> "org.apache.kafka.connect.storage.StringConverter",
> "camel.source.maxPollDuration": "10000",
> "topics": "TEST-S3-SOURCE-DZONE-POC",
> "camel.source.path.bucketNameOrArn": " push-json-poc",
> "camel.component.aws-s3.region": "US_EAST_1",
> "tasks.max": "1",
> "camel.source.endpoint.useIAMCredentials": "true",
> "camel.source.endpoint.autocloseBody": "true"
> }
> EOF
> )
> # "value.converter":
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
>
>
> This is official one from the link in the other email.
> name=CamelAWSS3SourceConnector
>
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> key.converter=org.apache.kafka.connect.storage.StringConverter
>
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
>
> camel.source.maxPollDuration=10000
>
> topics=mytopic
>
> camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
>
> camel.component.aws-s3.access-key=xxxx
> camel.component.aws-s3.secret-key=yyyy
> camel.component.aws-s3.region=EU_WEST_1
>
>
>
>
> I can see two differences -
> 1. How the AWS credentials are set-up
> 2. In my configuration
> I am using camel.source.path.bucketNameOrArn instead of
> camel.source.url
> Because I was getting error that bucketNameOrArn must be defined
>
> In my configuration - autocloseBody is set true to whereas in the
> default configuration it is set to false
>
> How does autocloseBody work? What impact does it have when
> setting it to true vs. false?
>
>
> Thank you
>
>
> -----Original Message-----
> From: Andrea Cosentino <an...@gmail.com>
> Sent: Saturday, December 26, 2020 3:59 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> Can you show your full configuration?
>
> Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> arundhati.bhende@prudential.com> ha scritto:
>
> > I am trying to test The Camel S3 Source connector using basic example
> > from this link
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7Cf7b74faf6add4a4f39f808d8a9aeed82%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445916065455084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2ISA5ZwtgVUwF%2BvydKewwZbMB5YVr9HNem1f6r4lDB8%3D&reserved=0
> .
> > Below is the file I am uploading to S3.
> >
> > testfile.txt
> > add one line
> > add another line
> > test the connector
> >
> >
> > After the connector picks up and processes the file, If I use this for
> > value converter
> >
> >
> > value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > ectConverter
> >
> > When I run the consumer, it shows the message processed but the output
> > shows blank.
> >
> > If I use the String converter as value converter
> >
> > org.apache.kafka.connect.storage.StringConverter
> >
> > I get the output in the form of
> >
> > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> >
> > From the example in the above link, the author doesn't seem to have to
> > do any actions to see the string.
> >
> > So, why would the S3ObjectConverter show a blank output for me? I can
> > understand the String representation for the object really shows the
> > memory address.
> >
> > What do I need to look at to fix the output.
> >
> > Also, is it possible to change the converter to JSON or Avro converter
> > in the configuration file or I need to add either different properties
> > to the configuration or do some processing afterwards?
> >
> > Thank You
> >
> >
>
Re: Camel S3 Source Connector - value.converter
Posted by Andrea Cosentino <an...@gmail.com>.
Yes, the approach without url is correct
Autoclosebody will close s3object after consuming the payload, so you want
to get something it must be equal to false, while using the converter.
My suggestion by the way is using the camel-aws2-s3 connector, based on ask
v2
Il sab 26 dic 2020, 15:53 Arundhati Bhende <ar...@prudential.com>
ha scritto:
> This is the full configuration that I have used. I am been experimenting
> with changing the value.converter between StringConverter and
> S3ObjectConverter - otherwise everything is same.
>
>
>
> DATA=$( cat << EOF
> {
> "connector.class":
> "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> "key.converter":
> "org.apache.kafka.connect.storage.StringConverter",
> "value.converter":
> "org.apache.kafka.connect.storage.StringConverter",
> "camel.source.maxPollDuration": "10000",
> "topics": "TEST-S3-SOURCE-DZONE-POC",
> "camel.source.path.bucketNameOrArn": " push-json-poc",
> "camel.component.aws-s3.region": "US_EAST_1",
> "tasks.max": "1",
> "camel.source.endpoint.useIAMCredentials": "true",
> "camel.source.endpoint.autocloseBody": "true"
> }
> EOF
> )
> # "value.converter":
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
>
>
> This is official one from the link in the other email.
> name=CamelAWSS3SourceConnector
>
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> key.converter=org.apache.kafka.connect.storage.StringConverter
>
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
>
> camel.source.maxPollDuration=10000
>
> topics=mytopic
>
> camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
>
> camel.component.aws-s3.access-key=xxxx
> camel.component.aws-s3.secret-key=yyyy
> camel.component.aws-s3.region=EU_WEST_1
>
>
>
>
> I can see two differences -
> 1. How the AWS credentials are set-up
> 2. In my configuration
> I am using camel.source.path.bucketNameOrArn instead of
> camel.source.url
> Because I was getting error that bucketNameOrArn must be defined
>
> In my configuration - autocloseBody is set true to whereas in the
> default configuration it is set to false
>
> How does autocloseBody work? What impact does it have when
> setting it to true vs. false?
>
>
> Thank you
>
>
> -----Original Message-----
> From: Andrea Cosentino <an...@gmail.com>
> Sent: Saturday, December 26, 2020 3:59 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> Can you show your full configuration?
>
> Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> arundhati.bhende@prudential.com> ha scritto:
>
> > I am trying to test The Camel S3 Source connector using basic example
> > from this link
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C2fe91ef852ff491ba73608d8a97c923a%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445700040770003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=84qab73sSiJnvgsTZGVcraOjp8tEMuLmNyeFOrYHZbM%3D&reserved=0
> .
> > Below is the file I am uploading to S3.
> >
> > testfile.txt
> > add one line
> > add another line
> > test the connector
> >
> >
> > After the connector picks up and processes the file, If I use this for
> > value converter
> >
> >
> > value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > ectConverter
> >
> > When I run the consumer, it shows the message processed but the output
> > shows blank.
> >
> > If I use the String converter as value converter
> >
> > org.apache.kafka.connect.storage.StringConverter
> >
> > I get the output in the form of
> >
> > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> >
> > From the example in the above link, the author doesn't seem to have to
> > do any actions to see the string.
> >
> > So, why would the S3ObjectConverter show a blank output for me? I can
> > understand the String representation for the object really shows the
> > memory address.
> >
> > What do I need to look at to fix the output.
> >
> > Also, is it possible to change the converter to JSON or Avro converter
> > in the configuration file or I need to add either different properties
> > to the configuration or do some processing afterwards?
> >
> > Thank You
> >
> >
>
RE: Camel S3 Source Connector - value.converter
Posted by Arundhati Bhende <ar...@prudential.com>.
This is the full configuration that I have used. I am been experimenting with changing the value.converter between StringConverter and S3ObjectConverter - otherwise everything is same.
DATA=$( cat << EOF
{
"connector.class": "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"camel.source.maxPollDuration": "10000",
"topics": "TEST-S3-SOURCE-DZONE-POC",
"camel.source.path.bucketNameOrArn": " push-json-poc",
"camel.component.aws-s3.region": "US_EAST_1",
"tasks.max": "1",
"camel.source.endpoint.useIAMCredentials": "true",
"camel.source.endpoint.autocloseBody": "true"
}
EOF
)
# "value.converter": "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
This is official one from the link in the other email.
name=CamelAWSS3SourceConnector
connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
camel.source.maxPollDuration=10000
topics=mytopic
camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
camel.component.aws-s3.access-key=xxxx
camel.component.aws-s3.secret-key=yyyy
camel.component.aws-s3.region=EU_WEST_1
I can see two differences -
1. How the AWS credentials are set-up
2. In my configuration
I am using camel.source.path.bucketNameOrArn instead of camel.source.url
Because I was getting error that bucketNameOrArn must be defined
In my configuration - autocloseBody is set true to whereas in the default configuration it is set to false
How does autocloseBody work? What impact does it have when setting it to true vs. false?
Thank you
-----Original Message-----
From: Andrea Cosentino <an...@gmail.com>
Sent: Saturday, December 26, 2020 3:59 AM
To: users@camel.apache.org
Subject: Re: Camel S3 Source Connector - value.converter
Can you show your full configuration?
Il sab 26 dic 2020, 07:40 Arundhati Bhende <ar...@prudential.com> ha scritto:
> I am trying to test The Camel S3 Source connector using basic example
> from this link
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C2fe91ef852ff491ba73608d8a97c923a%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445700040770003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=84qab73sSiJnvgsTZGVcraOjp8tEMuLmNyeFOrYHZbM%3D&reserved=0.
> Below is the file I am uploading to S3.
>
> testfile.txt
> add one line
> add another line
> test the connector
>
>
> After the connector picks up and processes the file, If I use this for
> value converter
>
>
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> ectConverter
>
> When I run the consumer, it shows the message processed but the output
> shows blank.
>
> If I use the String converter as value converter
>
> org.apache.kafka.connect.storage.StringConverter
>
> I get the output in the form of
>
> com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
>
> From the example in the above link, the author doesn't seem to have to
> do any actions to see the string.
>
> So, why would the S3ObjectConverter show a blank output for me? I can
> understand the String representation for the object really shows the
> memory address.
>
> What do I need to look at to fix the output.
>
> Also, is it possible to change the converter to JSON or Avro converter
> in the configuration file or I need to add either different properties
> to the configuration or do some processing afterwards?
>
> Thank You
>
>
Re: Camel S3 Source Connector - value.converter
Posted by Andrea Cosentino <an...@gmail.com>.
As example use the official one
https://github.com/apache/camel-kafka-connector-examples/blob/master/aws-s3-to-jms/config/CamelAWSS3SourceConnector.properties
Il sab 26 dic 2020, 09:59 Andrea Cosentino <an...@gmail.com> ha scritto:
> Can you show your full configuration?
>
> Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> arundhati.bhende@prudential.com> ha scritto:
>
>> I am trying to test The Camel S3 Source connector using basic example
>> from this link
>> https://dzone.com/articles/reading-aws-s3-file-content-to-kafka-topic.
>> Below is the file I am uploading to S3.
>>
>> testfile.txt
>> add one line
>> add another line
>> test the connector
>>
>>
>> After the connector picks up and processes the file, If I use this for
>> value converter
>>
>>
>> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
>>
>> When I run the consumer, it shows the message processed but the output
>> shows blank.
>>
>> If I use the String converter as value converter
>>
>> org.apache.kafka.connect.storage.StringConverter
>>
>> I get the output in the form of
>>
>> com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
>> com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
>>
>> From the example in the above link, the author doesn't seem to have to do
>> any actions to see the string.
>>
>> So, why would the S3ObjectConverter show a blank output for me? I can
>> understand the String representation for the object really shows the memory
>> address.
>>
>> What do I need to look at to fix the output.
>>
>> Also, is it possible to change the converter to JSON or Avro converter in
>> the configuration file or I need to add either different properties to the
>> configuration or do some processing afterwards?
>>
>> Thank You
>>
>>
Re: Camel S3 Source Connector - value.converter
Posted by Andrea Cosentino <an...@gmail.com>.
Can you show your full configuration?
Il sab 26 dic 2020, 07:40 Arundhati Bhende <ar...@prudential.com>
ha scritto:
> I am trying to test The Camel S3 Source connector using basic example from
> this link
> https://dzone.com/articles/reading-aws-s3-file-content-to-kafka-topic.
> Below is the file I am uploading to S3.
>
> testfile.txt
> add one line
> add another line
> test the connector
>
>
> After the connector picks up and processes the file, If I use this for
> value converter
>
>
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
>
> When I run the consumer, it shows the message processed but the output
> shows blank.
>
> If I use the String converter as value converter
>
> org.apache.kafka.connect.storage.StringConverter
>
> I get the output in the form of
>
> com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
>
> From the example in the above link, the author doesn't seem to have to do
> any actions to see the string.
>
> So, why would the S3ObjectConverter show a blank output for me? I can
> understand the String representation for the object really shows the memory
> address.
>
> What do I need to look at to fix the output.
>
> Also, is it possible to change the converter to JSON or Avro converter in
> the configuration file or I need to add either different properties to the
> configuration or do some processing afterwards?
>
> Thank You
>
>