You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by andrew davidson <an...@santacruzanalytics.com> on 2022/03/29 22:48:36 UTC
Newbie looking for a connector I can configure on my mac
I found the quick start https://kafka.apache.org/quickstart example very helpful. It made it really easy to understand how download, start up, create topic, push some data through the Kafka. I did not find https://kafka.apache.org/quickstart#quickstart_kafkaconnect useful.
I am looking for something very simple to learning how to configure and use connectors using Apache Kafka distribution, not Confluent. I can run on my mac or Linux server. Being a newbie I want to keep things super simple. I do not want to have to debug firewalls, ACL, …
I do not have a data base, access to twitter, …
I thought maybe something some sort source/sink using the local file system?
Any suggestions?
Kind regards
Andy
p.s. I have read a lot of documentation most of it is very high level. Can anyone recommend a “hand on” tutorial?
Connector quick start demo: Re: Newbie looking for a connector I can configure on my mac
Posted by andrew davidson <an...@santacruzanalytics.com>.
Hi Liam
Many thanks. I was able to get the file stream connector demo working. See the attached file for detail.
Andy
On 3/29/22, 5:27 PM, "Liam Clarke-Hutchinson" <lc...@redhat.com> wrote:
Hi Andrew,
So if you've downloaded Apache Kafka, you can run a standalone connect
instance using the bin/connect-standalone.sh script mentioned. And while a
lot of the connector documentation is on the Confluent website, you can
still use them with FOSS Kafka so long as you're in line with the Confluent
Community Licence (basically, IIRC, you can use them for free, but not to
run a SAAS or similar that competes with Confluent, but IANAL).
I agree that there's not much useful documentation for your use case. I
will look into writing a tutorial for your use case, would you be happy to
give me feedback on it as I go?
The most important configuration initially is the plugin.path, where your
standalone KC process will look for those JARs. You can see an example
properties file for standalone Connect under the config/ dir in the Kafka
you downloaded. Note that it has the plugin path commented out initially.
So, Kafka ships with a connector that exposes a file source and file sink,
which is good for testing out KC and getting used to it. You can either
build it from source, or download it from here:
https://mvnrepository.com/artifact/org.apache.kafka/connect-file - choose
the version that matches the version of Kafka you've downloaded, and then
you can download the JAR under files up the top. This documentation from
Confluent is useful:
https://docs.confluent.io/platform/current/connect/filestream_connector.html
Note that if you don't provide a file property, (this isn't documented
either(!)) it will use standard input for the file source, and standard
output for the file sink. You can see example configs for this connector
reading from a file or console under that same config/ directory, and ditto
for writing.
These connectors might also be useful for playing with KC, and are all free
and downloadable:
https://www.confluent.io/hub/confluentinc/kafka-connect-datagen <-
generates a stream of test data
https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter <-
disregard, I saw you mentioned not having Twitter
https://www.confluent.io/hub/C0urante/kafka-connect-reddit <- I haven't
used this, but could be interesting?
I hope this helps you get started, and please let me know if I can help
with anything else :)
Cheers,
Liam Clarke
On Wed, 30 Mar 2022 at 11:54, andrew davidson <an...@santacruzanalytics.com>
wrote:
> I found the quick start https://kafka.apache.org/quickstart example very
> helpful. It made it really easy to understand how download, start up,
> create topic, push some data through the Kafka. I did not find
> https://kafka.apache.org/quickstart#quickstart_kafkaconnect useful.
>
> I am looking for something very simple to learning how to configure and
> use connectors using Apache Kafka distribution, not Confluent. I can run on
> my mac or Linux server. Being a newbie I want to keep things super simple.
> I do not want to have to debug firewalls, ACL, …
>
> I do not have a data base, access to twitter, …
>
> I thought maybe something some sort source/sink using the local file
> system?
>
> Any suggestions?
>
> Kind regards
>
> Andy
>
> p.s. I have read a lot of documentation most of it is very high level. Can
> anyone recommend a “hand on” tutorial?
>
>
>
Re: Newbie looking for a connector I can configure on my mac
Posted by Liam Clarke-Hutchinson <lc...@redhat.com>.
Thanks Andy, and glad I could help somewhat :) I'll start writing something
up over the next week, and will share the link here for feedback :)
Just a caveat, the Filestream Connector could be considered a security risk
in a production environment given its direct access to the filesystem. Not
that you're looking to use this in prod, but just wanted to be sure I
mentioned it.
On Thu, 31 Mar 2022 at 09:24, andrew davidson <an...@santacruzanalytics.com>
wrote:
> Thanks Liam.
>
> What is 'FOSS Kafka'? google did not find any useful definitions
>
> A tutorial would be great! I would be happy to provide feed back
>
>
>
> https://docs.confluent.io/platform/current/connect/filestream_connector.html
> looks really helpful.
>
> Your list of connectors is very helpful. I wonder if your email would be a
> good thing to add to the kafka documentation quick start guide or FAQ?
>
> Kind regards
>
> Andy
>
> On 3/29/22, 5:27 PM, "Liam Clarke-Hutchinson" <lc...@redhat.com>
> wrote:
>
> Hi Andrew,
>
> So if you've downloaded Apache Kafka, you can run a standalone connect
> instance using the bin/connect-standalone.sh script mentioned. And
> while a
> lot of the connector documentation is on the Confluent website, you can
> still use them with FOSS Kafka so long as you're in line with the
> Confluent
> Community Licence (basically, IIRC, you can use them for free, but not
> to
> run a SAAS or similar that competes with Confluent, but IANAL).
>
> I agree that there's not much useful documentation for your use case. I
> will look into writing a tutorial for your use case, would you be
> happy to
> give me feedback on it as I go?
>
> The most important configuration initially is the plugin.path, where
> your
> standalone KC process will look for those JARs. You can see an example
> properties file for standalone Connect under the config/ dir in the
> Kafka
> you downloaded. Note that it has the plugin path commented out
> initially.
>
> So, Kafka ships with a connector that exposes a file source and file
> sink,
> which is good for testing out KC and getting used to it. You can either
> build it from source, or download it from here:
> https://mvnrepository.com/artifact/org.apache.kafka/connect-file -
> choose
> the version that matches the version of Kafka you've downloaded, and
> then
> you can download the JAR under files up the top. This documentation
> from
> Confluent is useful:
>
> https://docs.confluent.io/platform/current/connect/filestream_connector.html
>
> Note that if you don't provide a file property, (this isn't documented
> either(!)) it will use standard input for the file source, and standard
> output for the file sink. You can see example configs for this
> connector
> reading from a file or console under that same config/ directory, and
> ditto
> for writing.
>
> These connectors might also be useful for playing with KC, and are all
> free
> and downloadable:
> https://www.confluent.io/hub/confluentinc/kafka-connect-datagen <-
> generates a stream of test data
> https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter <-
> disregard, I saw you mentioned not having Twitter
> https://www.confluent.io/hub/C0urante/kafka-connect-reddit <- I
> haven't
> used this, but could be interesting?
>
>
> I hope this helps you get started, and please let me know if I can help
> with anything else :)
>
> Cheers,
>
> Liam Clarke
>
>
>
> On Wed, 30 Mar 2022 at 11:54, andrew davidson <
> andy@santacruzanalytics.com>
> wrote:
>
> > I found the quick start https://kafka.apache.org/quickstart example
> very
> > helpful. It made it really easy to understand how download, start up,
> > create topic, push some data through the Kafka. I did not find
> > https://kafka.apache.org/quickstart#quickstart_kafkaconnect useful.
> >
> > I am looking for something very simple to learning how to configure
> and
> > use connectors using Apache Kafka distribution, not Confluent. I can
> run on
> > my mac or Linux server. Being a newbie I want to keep things super
> simple.
> > I do not want to have to debug firewalls, ACL, …
> >
> > I do not have a data base, access to twitter, …
> >
> > I thought maybe something some sort source/sink using the local file
> > system?
> >
> > Any suggestions?
> >
> > Kind regards
> >
> > Andy
> >
> > p.s. I have read a lot of documentation most of it is very high
> level. Can
> > anyone recommend a “hand on” tutorial?
> >
> >
> >
>
Re: Newbie looking for a connector I can configure on my mac
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
FOSS == Free Open Source Software
From: andrew davidson <an...@santacruzanalytics.com>
Date: Wednesday, March 30, 2022 at 3:16 PM
To: users@kafka.apache.org <us...@kafka.apache.org>
Subject: [EXTERNAL] Re: Newbie looking for a connector I can configure on my mac
Thanks Liam.
What is 'FOSS Kafka'? google did not find any useful definitions
A tutorial would be great! I would be happy to provide feed back
https://urldefense.com/v3/__https://docs.confluent.io/platform/current/connect/filestream_connector.html__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUe9wex7w$<https://urldefense.com/v3/__https:/docs.confluent.io/platform/current/connect/filestream_connector.html__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUe9wex7w$> looks really helpful.
Your list of connectors is very helpful. I wonder if your email would be a good thing to add to the kafka documentation quick start guide or FAQ?
Kind regards
Andy
On 3/29/22, 5:27 PM, "Liam Clarke-Hutchinson" <lc...@redhat.com> wrote:
Hi Andrew,
So if you've downloaded Apache Kafka, you can run a standalone connect
instance using the bin/connect-standalone.sh script mentioned. And while a
lot of the connector documentation is on the Confluent website, you can
still use them with FOSS Kafka so long as you're in line with the Confluent
Community Licence (basically, IIRC, you can use them for free, but not to
run a SAAS or similar that competes with Confluent, but IANAL).
I agree that there's not much useful documentation for your use case. I
will look into writing a tutorial for your use case, would you be happy to
give me feedback on it as I go?
The most important configuration initially is the plugin.path, where your
standalone KC process will look for those JARs. You can see an example
properties file for standalone Connect under the config/ dir in the Kafka
you downloaded. Note that it has the plugin path commented out initially.
So, Kafka ships with a connector that exposes a file source and file sink,
which is good for testing out KC and getting used to it. You can either
build it from source, or download it from here:
https://urldefense.com/v3/__https://mvnrepository.com/artifact/org.apache.kafka/connect-file__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCW7qkQA0Q$<https://urldefense.com/v3/__https:/mvnrepository.com/artifact/org.apache.kafka/connect-file__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCW7qkQA0Q$> - choose
the version that matches the version of Kafka you've downloaded, and then
you can download the JAR under files up the top. This documentation from
Confluent is useful:
https://urldefense.com/v3/__https://docs.confluent.io/platform/current/connect/filestream_connector.html__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUe9wex7w$<https://urldefense.com/v3/__https:/docs.confluent.io/platform/current/connect/filestream_connector.html__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUe9wex7w$>
Note that if you don't provide a file property, (this isn't documented
either(!)) it will use standard input for the file source, and standard
output for the file sink. You can see example configs for this connector
reading from a file or console under that same config/ directory, and ditto
for writing.
These connectors might also be useful for playing with KC, and are all free
and downloadable:
https://urldefense.com/v3/__https://www.confluent.io/hub/confluentinc/kafka-connect-datagen__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCVucIx6Ng$<https://urldefense.com/v3/__https:/www.confluent.io/hub/confluentinc/kafka-connect-datagen__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCVucIx6Ng$> <-
generates a stream of test data
https://urldefense.com/v3/__https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCXfwK4JoQ$<https://urldefense.com/v3/__https:/www.confluent.io/hub/jcustenborder/kafka-connect-twitter__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCXfwK4JoQ$> <-
disregard, I saw you mentioned not having Twitter
https://urldefense.com/v3/__https://www.confluent.io/hub/C0urante/kafka-connect-reddit__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUgRY66HQ$<https://urldefense.com/v3/__https:/www.confluent.io/hub/C0urante/kafka-connect-reddit__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUgRY66HQ$> <- I haven't
used this, but could be interesting?
I hope this helps you get started, and please let me know if I can help
with anything else :)
Cheers,
Liam Clarke
On Wed, 30 Mar 2022 at 11:54, andrew davidson <an...@santacruzanalytics.com>
wrote:
> I found the quick start https://urldefense.com/v3/__https://kafka.apache.org/quickstart__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCVqUYeB3g$<https://urldefense.com/v3/__https:/kafka.apache.org/quickstart__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCVqUYeB3g$> example very
> helpful. It made it really easy to understand how download, start up,
> create topic, push some data through the Kafka. I did not find
> https://urldefense.com/v3/__https://kafka.apache.org/quickstart*quickstart_kafkaconnect__;Iw!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCXxevE4wA$<https://urldefense.com/v3/__https:/kafka.apache.org/quickstart*quickstart_kafkaconnect__;Iw!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCXxevE4wA$> useful.
>
> I am looking for something very simple to learning how to configure and
> use connectors using Apache Kafka distribution, not Confluent. I can run on
> my mac or Linux server. Being a newbie I want to keep things super simple.
> I do not want to have to debug firewalls, ACL, …
>
> I do not have a data base, access to twitter, …
>
> I thought maybe something some sort source/sink using the local file
> system?
>
> Any suggestions?
>
> Kind regards
>
> Andy
>
> p.s. I have read a lot of documentation most of it is very high level. Can
> anyone recommend a “hand on” tutorial?
>
>
>
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Newbie looking for a connector I can configure on my mac
Posted by andrew davidson <an...@santacruzanalytics.com>.
Thanks Liam.
What is 'FOSS Kafka'? google did not find any useful definitions
A tutorial would be great! I would be happy to provide feed back
https://docs.confluent.io/platform/current/connect/filestream_connector.html looks really helpful.
Your list of connectors is very helpful. I wonder if your email would be a good thing to add to the kafka documentation quick start guide or FAQ?
Kind regards
Andy
On 3/29/22, 5:27 PM, "Liam Clarke-Hutchinson" <lc...@redhat.com> wrote:
Hi Andrew,
So if you've downloaded Apache Kafka, you can run a standalone connect
instance using the bin/connect-standalone.sh script mentioned. And while a
lot of the connector documentation is on the Confluent website, you can
still use them with FOSS Kafka so long as you're in line with the Confluent
Community Licence (basically, IIRC, you can use them for free, but not to
run a SAAS or similar that competes with Confluent, but IANAL).
I agree that there's not much useful documentation for your use case. I
will look into writing a tutorial for your use case, would you be happy to
give me feedback on it as I go?
The most important configuration initially is the plugin.path, where your
standalone KC process will look for those JARs. You can see an example
properties file for standalone Connect under the config/ dir in the Kafka
you downloaded. Note that it has the plugin path commented out initially.
So, Kafka ships with a connector that exposes a file source and file sink,
which is good for testing out KC and getting used to it. You can either
build it from source, or download it from here:
https://mvnrepository.com/artifact/org.apache.kafka/connect-file - choose
the version that matches the version of Kafka you've downloaded, and then
you can download the JAR under files up the top. This documentation from
Confluent is useful:
https://docs.confluent.io/platform/current/connect/filestream_connector.html
Note that if you don't provide a file property, (this isn't documented
either(!)) it will use standard input for the file source, and standard
output for the file sink. You can see example configs for this connector
reading from a file or console under that same config/ directory, and ditto
for writing.
These connectors might also be useful for playing with KC, and are all free
and downloadable:
https://www.confluent.io/hub/confluentinc/kafka-connect-datagen <-
generates a stream of test data
https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter <-
disregard, I saw you mentioned not having Twitter
https://www.confluent.io/hub/C0urante/kafka-connect-reddit <- I haven't
used this, but could be interesting?
I hope this helps you get started, and please let me know if I can help
with anything else :)
Cheers,
Liam Clarke
On Wed, 30 Mar 2022 at 11:54, andrew davidson <an...@santacruzanalytics.com>
wrote:
> I found the quick start https://kafka.apache.org/quickstart example very
> helpful. It made it really easy to understand how download, start up,
> create topic, push some data through the Kafka. I did not find
> https://kafka.apache.org/quickstart#quickstart_kafkaconnect useful.
>
> I am looking for something very simple to learning how to configure and
> use connectors using Apache Kafka distribution, not Confluent. I can run on
> my mac or Linux server. Being a newbie I want to keep things super simple.
> I do not want to have to debug firewalls, ACL, …
>
> I do not have a data base, access to twitter, …
>
> I thought maybe something some sort source/sink using the local file
> system?
>
> Any suggestions?
>
> Kind regards
>
> Andy
>
> p.s. I have read a lot of documentation most of it is very high level. Can
> anyone recommend a “hand on” tutorial?
>
>
>
Re: Newbie looking for a connector I can configure on my mac
Posted by Liam Clarke-Hutchinson <lc...@redhat.com>.
Hi Andrew Otto (Andrew 2? :D ),
Well noted. I should have worded it as "You don't need a Confluent
subscription to use them" :)
Cheers,
Liam
On Wed, 30 Mar 2022 at 13:49, Andrew Otto <ot...@wikimedia.org> wrote:
> > And while a lot of the connector documentation is on the Confluent
> website, you can still use them with FOSS Kafka so long as you're in line
> with the Confluent Community Licence
>
> *Drive by unhelpful comment:*
> While this is true legally, the fact that (most?) actual connector
> implementations are CCL and not FOSS, means that organizations that use
> purely FOSS software (like the Wikimedia Foundation) makes Kafka Connect
> effectively unusable.
>
> Okay carry on! :)
>
> - Andrew Otto
>
>
>
>
>
> On Tue, Mar 29, 2022 at 8:27 PM Liam Clarke-Hutchinson <
> lclarkeh@redhat.com>
> wrote:
>
> > Hi Andrew,
> >
> > So if you've downloaded Apache Kafka, you can run a standalone connect
> > instance using the bin/connect-standalone.sh script mentioned. And while
> a
> > lot of the connector documentation is on the Confluent website, you can
> > still use them with FOSS Kafka so long as you're in line with the
> Confluent
> > Community Licence (basically, IIRC, you can use them for free, but not to
> > run a SAAS or similar that competes with Confluent, but IANAL).
> >
> > I agree that there's not much useful documentation for your use case. I
> > will look into writing a tutorial for your use case, would you be happy
> to
> > give me feedback on it as I go?
> >
> > The most important configuration initially is the plugin.path, where your
> > standalone KC process will look for those JARs. You can see an example
> > properties file for standalone Connect under the config/ dir in the Kafka
> > you downloaded. Note that it has the plugin path commented out initially.
> >
> > So, Kafka ships with a connector that exposes a file source and file
> sink,
> > which is good for testing out KC and getting used to it. You can either
> > build it from source, or download it from here:
> > https://mvnrepository.com/artifact/org.apache.kafka/connect-file -
> choose
> > the version that matches the version of Kafka you've downloaded, and then
> > you can download the JAR under files up the top. This documentation from
> > Confluent is useful:
> >
> >
> https://docs.confluent.io/platform/current/connect/filestream_connector.html
> >
> > Note that if you don't provide a file property, (this isn't documented
> > either(!)) it will use standard input for the file source, and standard
> > output for the file sink. You can see example configs for this connector
> > reading from a file or console under that same config/ directory, and
> ditto
> > for writing.
> >
> > These connectors might also be useful for playing with KC, and are all
> free
> > and downloadable:
> > https://www.confluent.io/hub/confluentinc/kafka-connect-datagen <-
> > generates a stream of test data
> > https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter <-
> > disregard, I saw you mentioned not having Twitter
> > https://www.confluent.io/hub/C0urante/kafka-connect-reddit <- I haven't
> > used this, but could be interesting?
> >
> >
> > I hope this helps you get started, and please let me know if I can help
> > with anything else :)
> >
> > Cheers,
> >
> > Liam Clarke
> >
> >
> >
> > On Wed, 30 Mar 2022 at 11:54, andrew davidson <
> andy@santacruzanalytics.com
> > >
> > wrote:
> >
> > > I found the quick start https://kafka.apache.org/quickstart example
> very
> > > helpful. It made it really easy to understand how download, start up,
> > > create topic, push some data through the Kafka. I did not find
> > > https://kafka.apache.org/quickstart#quickstart_kafkaconnect useful.
> > >
> > > I am looking for something very simple to learning how to configure
> and
> > > use connectors using Apache Kafka distribution, not Confluent. I can
> run
> > on
> > > my mac or Linux server. Being a newbie I want to keep things super
> > simple.
> > > I do not want to have to debug firewalls, ACL, …
> > >
> > > I do not have a data base, access to twitter, …
> > >
> > > I thought maybe something some sort source/sink using the local file
> > > system?
> > >
> > > Any suggestions?
> > >
> > > Kind regards
> > >
> > > Andy
> > >
> > > p.s. I have read a lot of documentation most of it is very high level.
> > Can
> > > anyone recommend a “hand on” tutorial?
> > >
> > >
> > >
> >
>
Re: Newbie looking for a connector I can configure on my mac
Posted by andrew davidson <an...@santacruzanalytics.com>.
Thanks for the heads up Andrew
Andy
On 3/29/22, 5:43 PM, "Andrew Otto" <ot...@wikimedia.org> wrote:
> And while a lot of the connector documentation is on the Confluent
website, you can still use them with FOSS Kafka so long as you're in line
with the Confluent Community Licence
*Drive by unhelpful comment:*
While this is true legally, the fact that (most?) actual connector
implementations are CCL and not FOSS, means that organizations that use
purely FOSS software (like the Wikimedia Foundation) makes Kafka Connect
effectively unusable.
Okay carry on! :)
- Andrew Otto
On Tue, Mar 29, 2022 at 8:27 PM Liam Clarke-Hutchinson <lc...@redhat.com>
wrote:
> Hi Andrew,
>
> So if you've downloaded Apache Kafka, you can run a standalone connect
> instance using the bin/connect-standalone.sh script mentioned. And while a
> lot of the connector documentation is on the Confluent website, you can
> still use them with FOSS Kafka so long as you're in line with the Confluent
> Community Licence (basically, IIRC, you can use them for free, but not to
> run a SAAS or similar that competes with Confluent, but IANAL).
>
> I agree that there's not much useful documentation for your use case. I
> will look into writing a tutorial for your use case, would you be happy to
> give me feedback on it as I go?
>
> The most important configuration initially is the plugin.path, where your
> standalone KC process will look for those JARs. You can see an example
> properties file for standalone Connect under the config/ dir in the Kafka
> you downloaded. Note that it has the plugin path commented out initially.
>
> So, Kafka ships with a connector that exposes a file source and file sink,
> which is good for testing out KC and getting used to it. You can either
> build it from source, or download it from here:
> https://mvnrepository.com/artifact/org.apache.kafka/connect-file - choose
> the version that matches the version of Kafka you've downloaded, and then
> you can download the JAR under files up the top. This documentation from
> Confluent is useful:
>
> https://docs.confluent.io/platform/current/connect/filestream_connector.html
>
> Note that if you don't provide a file property, (this isn't documented
> either(!)) it will use standard input for the file source, and standard
> output for the file sink. You can see example configs for this connector
> reading from a file or console under that same config/ directory, and ditto
> for writing.
>
> These connectors might also be useful for playing with KC, and are all free
> and downloadable:
> https://www.confluent.io/hub/confluentinc/kafka-connect-datagen <-
> generates a stream of test data
> https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter <-
> disregard, I saw you mentioned not having Twitter
> https://www.confluent.io/hub/C0urante/kafka-connect-reddit <- I haven't
> used this, but could be interesting?
>
>
> I hope this helps you get started, and please let me know if I can help
> with anything else :)
>
> Cheers,
>
> Liam Clarke
>
>
>
> On Wed, 30 Mar 2022 at 11:54, andrew davidson <andy@santacruzanalytics.com
> >
> wrote:
>
> > I found the quick start https://kafka.apache.org/quickstart example very
> > helpful. It made it really easy to understand how download, start up,
> > create topic, push some data through the Kafka. I did not find
> > https://kafka.apache.org/quickstart#quickstart_kafkaconnect useful.
> >
> > I am looking for something very simple to learning how to configure and
> > use connectors using Apache Kafka distribution, not Confluent. I can run
> on
> > my mac or Linux server. Being a newbie I want to keep things super
> simple.
> > I do not want to have to debug firewalls, ACL, …
> >
> > I do not have a data base, access to twitter, …
> >
> > I thought maybe something some sort source/sink using the local file
> > system?
> >
> > Any suggestions?
> >
> > Kind regards
> >
> > Andy
> >
> > p.s. I have read a lot of documentation most of it is very high level.
> Can
> > anyone recommend a “hand on” tutorial?
> >
> >
> >
>
Re: Newbie looking for a connector I can configure on my mac
Posted by Andrew Otto <ot...@wikimedia.org>.
> And while a lot of the connector documentation is on the Confluent
website, you can still use them with FOSS Kafka so long as you're in line
with the Confluent Community Licence
*Drive by unhelpful comment:*
While this is true legally, the fact that (most?) actual connector
implementations are CCL and not FOSS, means that organizations that use
purely FOSS software (like the Wikimedia Foundation) makes Kafka Connect
effectively unusable.
Okay carry on! :)
- Andrew Otto
On Tue, Mar 29, 2022 at 8:27 PM Liam Clarke-Hutchinson <lc...@redhat.com>
wrote:
> Hi Andrew,
>
> So if you've downloaded Apache Kafka, you can run a standalone connect
> instance using the bin/connect-standalone.sh script mentioned. And while a
> lot of the connector documentation is on the Confluent website, you can
> still use them with FOSS Kafka so long as you're in line with the Confluent
> Community Licence (basically, IIRC, you can use them for free, but not to
> run a SAAS or similar that competes with Confluent, but IANAL).
>
> I agree that there's not much useful documentation for your use case. I
> will look into writing a tutorial for your use case, would you be happy to
> give me feedback on it as I go?
>
> The most important configuration initially is the plugin.path, where your
> standalone KC process will look for those JARs. You can see an example
> properties file for standalone Connect under the config/ dir in the Kafka
> you downloaded. Note that it has the plugin path commented out initially.
>
> So, Kafka ships with a connector that exposes a file source and file sink,
> which is good for testing out KC and getting used to it. You can either
> build it from source, or download it from here:
> https://mvnrepository.com/artifact/org.apache.kafka/connect-file - choose
> the version that matches the version of Kafka you've downloaded, and then
> you can download the JAR under files up the top. This documentation from
> Confluent is useful:
>
> https://docs.confluent.io/platform/current/connect/filestream_connector.html
>
> Note that if you don't provide a file property, (this isn't documented
> either(!)) it will use standard input for the file source, and standard
> output for the file sink. You can see example configs for this connector
> reading from a file or console under that same config/ directory, and ditto
> for writing.
>
> These connectors might also be useful for playing with KC, and are all free
> and downloadable:
> https://www.confluent.io/hub/confluentinc/kafka-connect-datagen <-
> generates a stream of test data
> https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter <-
> disregard, I saw you mentioned not having Twitter
> https://www.confluent.io/hub/C0urante/kafka-connect-reddit <- I haven't
> used this, but could be interesting?
>
>
> I hope this helps you get started, and please let me know if I can help
> with anything else :)
>
> Cheers,
>
> Liam Clarke
>
>
>
> On Wed, 30 Mar 2022 at 11:54, andrew davidson <andy@santacruzanalytics.com
> >
> wrote:
>
> > I found the quick start https://kafka.apache.org/quickstart example very
> > helpful. It made it really easy to understand how download, start up,
> > create topic, push some data through the Kafka. I did not find
> > https://kafka.apache.org/quickstart#quickstart_kafkaconnect useful.
> >
> > I am looking for something very simple to learning how to configure and
> > use connectors using Apache Kafka distribution, not Confluent. I can run
> on
> > my mac or Linux server. Being a newbie I want to keep things super
> simple.
> > I do not want to have to debug firewalls, ACL, …
> >
> > I do not have a data base, access to twitter, …
> >
> > I thought maybe something some sort source/sink using the local file
> > system?
> >
> > Any suggestions?
> >
> > Kind regards
> >
> > Andy
> >
> > p.s. I have read a lot of documentation most of it is very high level.
> Can
> > anyone recommend a “hand on” tutorial?
> >
> >
> >
>
Re: Newbie looking for a connector I can configure on my mac
Posted by Liam Clarke-Hutchinson <lc...@redhat.com>.
Hi Andrew,
So if you've downloaded Apache Kafka, you can run a standalone connect
instance using the bin/connect-standalone.sh script mentioned. And while a
lot of the connector documentation is on the Confluent website, you can
still use them with FOSS Kafka so long as you're in line with the Confluent
Community Licence (basically, IIRC, you can use them for free, but not to
run a SAAS or similar that competes with Confluent, but IANAL).
I agree that there's not much useful documentation for your use case. I
will look into writing a tutorial for your use case, would you be happy to
give me feedback on it as I go?
The most important configuration initially is the plugin.path, where your
standalone KC process will look for those JARs. You can see an example
properties file for standalone Connect under the config/ dir in the Kafka
you downloaded. Note that it has the plugin path commented out initially.
So, Kafka ships with a connector that exposes a file source and file sink,
which is good for testing out KC and getting used to it. You can either
build it from source, or download it from here:
https://mvnrepository.com/artifact/org.apache.kafka/connect-file - choose
the version that matches the version of Kafka you've downloaded, and then
you can download the JAR under files up the top. This documentation from
Confluent is useful:
https://docs.confluent.io/platform/current/connect/filestream_connector.html
Note that if you don't provide a file property, (this isn't documented
either(!)) it will use standard input for the file source, and standard
output for the file sink. You can see example configs for this connector
reading from a file or console under that same config/ directory, and ditto
for writing.
These connectors might also be useful for playing with KC, and are all free
and downloadable:
https://www.confluent.io/hub/confluentinc/kafka-connect-datagen <-
generates a stream of test data
https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter <-
disregard, I saw you mentioned not having Twitter
https://www.confluent.io/hub/C0urante/kafka-connect-reddit <- I haven't
used this, but could be interesting?
I hope this helps you get started, and please let me know if I can help
with anything else :)
Cheers,
Liam Clarke
On Wed, 30 Mar 2022 at 11:54, andrew davidson <an...@santacruzanalytics.com>
wrote:
> I found the quick start https://kafka.apache.org/quickstart example very
> helpful. It made it really easy to understand how download, start up,
> create topic, push some data through the Kafka. I did not find
> https://kafka.apache.org/quickstart#quickstart_kafkaconnect useful.
>
> I am looking for something very simple to learning how to configure and
> use connectors using Apache Kafka distribution, not Confluent. I can run on
> my mac or Linux server. Being a newbie I want to keep things super simple.
> I do not want to have to debug firewalls, ACL, …
>
> I do not have a data base, access to twitter, …
>
> I thought maybe something some sort source/sink using the local file
> system?
>
> Any suggestions?
>
> Kind regards
>
> Andy
>
> p.s. I have read a lot of documentation most of it is very high level. Can
> anyone recommend a “hand on” tutorial?
>
>
>