You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@predictionio.apache.org by Thomas POCREAU <th...@iadvize.com> on 2017/07/04 15:34:45 UTC

Kafka support as Event Store

Hi,

Thanks a lot for this awesome project.

I have a question regarding Kafka and it's possible integration as an Event
Store.
Do you have any plan on this matter ?
Are you aware of someone working on a similar sujet ?

Regards,
Thomas.

Re: Exception - pio train - Caued by permission denied

Posted by Pat Ferrel <pa...@occamsmachete.com>.
Sebastian from the other posting it looks like you need to change permission for `/` in hdfs. Do this with:

    sudo /usr/local/hadoop/bin/hdfs dfs -chmod 777 -R /

this will grant global permissions to all users in Linux to all part of HDFS.

The permission system is very much like ‘Nix OSes so whatever user is running a process will have the same permissions in hdfs as in the local fs so if the permission to write is not granted to “ubuntu” or “aml” or … then you must switch to root and change permissions. 

Read a similar problem here: https://community.cloudera.com/t5/CDH-Manual-Installation/Permission-denied-user-mapred-access-WRITE-inode-quot-quot-hdfs/td-p/16318 <https://community.cloudera.com/t5/CDH-Manual-Installation/Permission-denied-user-mapred-access-WRITE-inode-quot-quot-hdfs/td-p/16318>

On Jul 4, 2017, at 2:31 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:

Sebastian:

Try logging in to the “aml” user. Hadoop HDFS has it’s own permission system and conventions similar to ‘Nix OSes. There is `/user/aml/`directory in HDFS which acts as the home directory for any user logged in as “aml”. If you `dfs -put file` it will go to `/user/<’nix-username>/file` where <’nix-username> is the uname for the user logged in. Further you can sudo to change permissions. Anyway logging in to the “aml” user is all that should be required since the processes created will be under that user’s account. If you need to set permissions in the local file system create them for the “aml” account.

Marius:

Yes this is the typical way to install HDFS in the wide world (non-pio)  He is using an AWS AMI created by ActionML. 


On Jul 4, 2017, at 1:32 PM, Marius Rabenarivo <mariusrabenarivo@gmail.com <ma...@gmail.com>> wrote:

It seems like you installed Hadoop using hadoop user account.

Try to re-install it using the user account you intend to use
when running PredictionIO.

2017-07-04 21:54 GMT+04:00 Pat Ferrel <pat@occamsmachete.com <ma...@occamsmachete.com>>:
I you are using the AWS AMI from ActionML you should grant access to user “aml” since PIO is setup to run with this user.


On Jul 4, 2017, at 9:27 AM, Dan Guja <danguja@gmail.com <ma...@gmail.com>> wrote:

Hi

Did you try to give write access to /tmp/ for user ubuntu (as the error says) ?

On Tue, Jul 4, 2017 at 10:37 AM, Sebastian Fix <sebastian.fix@smartgate.ch <ma...@smartgate.ch>> wrote:
Hey together

Anyone experiencing similar problems?

Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=ubuntu, access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU1i-0-als/rank/_temporary/0":hadoop:supergroup:drwxr-xr-x

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

..

..

..

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=ubuntu, access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU1i-0-als/rank/_temporary/0":hadoop:supergroup:drwxr-xr-x

..

..

A Link to the console error (error2.mp4): https://www.dropbox.com/s/0t9laycpkbqpkfr/error2.mp4?dl=0 <https://www.dropbox.com/s/0t9laycpkbqpkfr/error2.mp4?dl=0>
Cheers, Felix







Re: Exception - pio train - Caued by permission denied

Posted by Pat Ferrel <pa...@occamsmachete.com>.
Sebastian:

Try logging in to the “aml” user. Hadoop HDFS has it’s own permission system and conventions similar to ‘Nix OSes. There is `/user/aml/`directory in HDFS which acts as the home directory for any user logged in as “aml”. If you `dfs -put file` it will go to `/user/<’nix-username>/file` where <’nix-username> is the uname for the user logged in. Further you can sudo to change permissions. Anyway logging in to the “aml” user is all that should be required since the processes created will be under that user’s account. If you need to set permissions in the local file system create them for the “aml” account.

Marius:

Yes this is the typical way to install HDFS in the wide world (non-pio)  He is using an AWS AMI created by ActionML. 


On Jul 4, 2017, at 1:32 PM, Marius Rabenarivo <ma...@gmail.com> wrote:

It seems like you installed Hadoop using hadoop user account.

Try to re-install it using the user account you intend to use
when running PredictionIO.

2017-07-04 21:54 GMT+04:00 Pat Ferrel <pat@occamsmachete.com <ma...@occamsmachete.com>>:
I you are using the AWS AMI from ActionML you should grant access to user “aml” since PIO is setup to run with this user.


On Jul 4, 2017, at 9:27 AM, Dan Guja <danguja@gmail.com <ma...@gmail.com>> wrote:

Hi

Did you try to give write access to /tmp/ for user ubuntu (as the error says) ?

On Tue, Jul 4, 2017 at 10:37 AM, Sebastian Fix <sebastian.fix@smartgate.ch <ma...@smartgate.ch>> wrote:
Hey together

Anyone experiencing similar problems?

Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=ubuntu, access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU1i-0-als/rank/_temporary/0":hadoop:supergroup:drwxr-xr-x

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

..

..

..

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=ubuntu, access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU1i-0-als/rank/_temporary/0":hadoop:supergroup:drwxr-xr-x

..

..

A Link to the console error (error2.mp4): https://www.dropbox.com/s/0t9laycpkbqpkfr/error2.mp4?dl=0 <https://www.dropbox.com/s/0t9laycpkbqpkfr/error2.mp4?dl=0>
Cheers, Felix






Re: Exception - pio train - Caued by permission denied

Posted by Marius Rabenarivo <ma...@gmail.com>.
It seems like you installed Hadoop using hadoop user account.

Try to re-install it using the user account you intend to use
when running PredictionIO.

2017-07-04 21:54 GMT+04:00 Pat Ferrel <pa...@occamsmachete.com>:

> I you are using the AWS AMI from ActionML you should grant access to user
> “aml” since PIO is setup to run with this user.
>
>
> On Jul 4, 2017, at 9:27 AM, Dan Guja <da...@gmail.com> wrote:
>
> Hi
>
> Did you try to give write access to /tmp/ for user ubuntu (as the error
> says) ?
>
> On Tue, Jul 4, 2017 at 10:37 AM, Sebastian Fix <sebastian.fix@smartgate.ch
> > wrote:
>
>> Hey together
>> Anyone experiencing similar problems?
>>
>> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
>> Permission denied: user=ubuntu, access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU
>> 1i-0-als/rank/_temporary/0":hadoop:supergroup:drwxr-xr-x
>>
>> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.c
>> heck(FSPermissionChecker.java:319)
>>
>> ..
>>
>> ..
>>
>> ..
>>
>> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
>> Permission denied: user=ubuntu, access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU
>> 1i-0-als/rank/_temporary/0":hadoop:supergroup:drwxr-xr-x
>>
>> ..
>>
>> ..
>>
>> A Link to the console error (error2.mp4): https://www.drop
>> box.com/s/0t9laycpkbqpkfr/error2.mp4?dl=0
>>
>> Cheers, Felix
>>
>
>
>

Re: Exception - pio train - Caued by permission denied

Posted by Pat Ferrel <pa...@occamsmachete.com>.
I you are using the AWS AMI from ActionML you should grant access to user “aml” since PIO is setup to run with this user.


On Jul 4, 2017, at 9:27 AM, Dan Guja <da...@gmail.com> wrote:

Hi

Did you try to give write access to /tmp/ for user ubuntu (as the error says) ?

On Tue, Jul 4, 2017 at 10:37 AM, Sebastian Fix <sebastian.fix@smartgate.ch <ma...@smartgate.ch>> wrote:
Hey together

Anyone experiencing similar problems?

Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=ubuntu, access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU1i-0-als/rank/_temporary/0":hadoop:supergroup:drwxr-xr-x

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

..

..

..

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=ubuntu, access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU1i-0-als/rank/_temporary/0":hadoop:supergroup:drwxr-xr-x

..

..

A Link to the console error (error2.mp4): https://www.dropbox.com/s/0t9laycpkbqpkfr/error2.mp4?dl=0 <https://www.dropbox.com/s/0t9laycpkbqpkfr/error2.mp4?dl=0>
Cheers, Felix




Re: Exception - pio train - Caued by permission denied

Posted by Dan Guja <da...@gmail.com>.
Hi

Did you try to give write access to /tmp/ for user ubuntu (as the error
says) ?

On Tue, Jul 4, 2017 at 10:37 AM, Sebastian Fix <se...@smartgate.ch>
wrote:

> Hey together
> Anyone experiencing similar problems?
>
> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
> Permission denied: user=ubuntu, access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU
> 1i-0-als/rank/_temporary/0":hadoop:supergroup:drwxr-xr-x
>
> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.c
> heck(FSPermissionChecker.java:319)
>
> ..
>
> ..
>
> ..
>
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.
> security.AccessControlException): Permission denied: user=ubuntu,
> access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU1i-0-als/rank/_temporary/0":
> hadoop:supergroup:drwxr-xr-x
>
> ..
>
> ..
>
> A Link to the console error (error2.mp4): https://www.
> dropbox.com/s/0t9laycpkbqpkfr/error2.mp4?dl=0
>
> Cheers, Felix
>

Exception - pio train - Caued by permission denied

Posted by Sebastian Fix <se...@smartgate.ch>.
Hey together

Anyone experiencing similar problems?

Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=ubuntu, access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU1i-0-als/rank/_temporary/0":hadoop:supergroup:drwxr-xr-x


at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

..

..

..

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=ubuntu, access=WRITE, inode="/tmp/AV0N2sqO1Bi_4YGMpU1i-0-als/rank/_temporary/0":hadoop:supergroup:drwxr-xr-x

..

..

A Link to the console error (error2.mp4): https://www.dropbox.com/s/0t9laycpkbqpkfr/error2.mp4?dl=0

Cheers, Felix

Re: Kafka support as Event Store

Posted by Kenneth Chan <ke...@apache.org>.
thomas, if you always your own data store backend for events or your
application data, you don't have to use EventServer. Just implement a PIO
engine which directly reads from your data source.


On Thu, Jul 6, 2017 at 7:32 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:

> The EventStore is queryable in flexible ways and so needs to have random
> access and indexing of events. In short it needs to be backed by a DB.
>
> There are ML algorithms that do not need stored events like online
> learners in the Kappa style. But existing Templates and the PIO
> architecture do not support Kappa yet. So unless you are developing your
> own Template you will find that Kafka will not fill all the requirements.
>
> It is, however an excellent way to manage streams so I tend to see it as a
> source for events and have seen it used as such.
>
> To be more precise about why your method may not work is that not all
> events can be aged out. The item property events in PIO $set, $unset, and
> $delete are embedded in the stream but are only looked at in aggregate
> since they record a set of changes to an object. If you drop one of these
> events the current state of the object can not be calculated with certainty.
>
> To truly use a streaming model we have to introduce the idea of watermarks
> that snapshot state in the stream so that any event can be dropped. This is
> how kappa learners work but for existing Lambda learners is often not
> possible. Many of the Lambda learners can be converted but not easily and
> not all.
>
>
> On Jul 6, 2017, at 1:59 AM, Thomas POCREAU <th...@iadvize.com>
> wrote:
>
> After some talks in intern, I misunderstood our needs.
> Indeed, we will use Kafka HDFS connector
> <http://docs.confluent.io/current/connect/connect-hdfs/docs/index.html>
> to dump expired data.
> So we will basically have Kafka for the fresh events and HDFS for the past
> events.
>
>
> 2017-07-06 7:06 GMT+02:00 Thomas POCREAU <th...@iadvize.com>:
>
>> Hi,
>>
>> Thanks for your responses.
>>
>> Our goal is to use kafka as our main event store for event sourcing. I'm
>> pretty sure that kafka can be used with an infinite retention time.
>>
>> We could use KStream and the Java sdk but I would like to give a try to
>> an implementation of PStore on top of spark-streaming-kafka (
>> https://spark.apache.org/docs/latest/streaming-kafka-0-10-
>> integration.html).
>>
>> My main concern is related to the http interface to push events. We will
>> probably use KStream or websockets to load events in our Kafka topic used
>> by an app channel.
>>
>> Are you planning on supporting websockets as an alternative to batch
>> import?
>>
>> Regards,
>> Thomas Pocreau
>>
>> Le 5 juil. 2017 21:36, "Pat Ferrel" <pa...@occamsmachete.com> a écrit :
>>
>>> No, we try not to fork :-) But it would be nice as you say. It can be
>>> done with a small intermediary app that just reads from a Kafka topic and
>>> send events to a localhost EventServer, which would allow events to be
>>> custom extracted from say log files (typical contents of Kafka). We’ve done
>>> this in non-PIO projects.
>>>
>>> The intermediary app should use Spark streaming. I may have a snippet of
>>> code around if you need it but it just saves to micro-batch files. You’d
>>> have to use the PIO Java-SDK to send them to the EventServer. A relatively
>>> simple thing.
>>>
>>> Donald, what did you have in mind for deeper integration? I guess we
>>> could cut out the intermediate app and integrate into a new Kafka aware
>>> EventServer endpoint where the raw topic input is stored in the EventStore.
>>> This would force any log filtering onto the Kafka source.
>>>
>>>
>>> On Jul 5, 2017, at 10:20 AM, Donald Szeto <do...@apache.org> wrote:
>>>
>>> Hi Thomas,
>>>
>>> Supporting Kafka is definitely interesting and desirable. Are you
>>> looking to sinking your Kafka messages to event store for batch processing,
>>> or stream processing directly from Kafka? The latter would require more
>>> work because Apache PIO does not yet support streaming properly.
>>>
>>> Folks from ActionML might have a flavor of PIO that works with Kafka.
>>>
>>> Regards,
>>> Donald
>>>
>>> On Tue, Jul 4, 2017 at 8:34 AM, Thomas POCREAU <
>>> thomas.pocreau@iadvize.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Thanks a lot for this awesome project.
>>>>
>>>> I have a question regarding Kafka and it's possible integration as an
>>>> Event Store.
>>>> Do you have any plan on this matter ?
>>>> Are you aware of someone working on a similar sujet ?
>>>>
>>>> Regards,
>>>> Thomas.
>>>>
>>>>
>>>
>>>
>
>

Re: Kafka support as Event Store

Posted by Pat Ferrel <pa...@occamsmachete.com>.
The EventStore is queryable in flexible ways and so needs to have random access and indexing of events. In short it needs to be backed by a DB.

There are ML algorithms that do not need stored events like online learners in the Kappa style. But existing Templates and the PIO architecture do not support Kappa yet. So unless you are developing your own Template you will find that Kafka will not fill all the requirements.

It is, however an excellent way to manage streams so I tend to see it as a source for events and have seen it used as such.

To be more precise about why your method may not work is that not all events can be aged out. The item property events in PIO $set, $unset, and $delete are embedded in the stream but are only looked at in aggregate since they record a set of changes to an object. If you drop one of these events the current state of the object can not be calculated with certainty.

To truly use a streaming model we have to introduce the idea of watermarks that snapshot state in the stream so that any event can be dropped. This is how kappa learners work but for existing Lambda learners is often not possible. Many of the Lambda learners can be converted but not easily and not all.

 
On Jul 6, 2017, at 1:59 AM, Thomas POCREAU <th...@iadvize.com> wrote:

After some talks in intern, I misunderstood our needs.
Indeed, we will use Kafka HDFS connector <http://docs.confluent.io/current/connect/connect-hdfs/docs/index.html> to dump expired data.
So we will basically have Kafka for the fresh events and HDFS for the past events.


2017-07-06 7:06 GMT+02:00 Thomas POCREAU <thomas.pocreau@iadvize.com <ma...@iadvize.com>>:
Hi,

Thanks for your responses.

Our goal is to use kafka as our main event store for event sourcing. I'm pretty sure that kafka can be used with an infinite retention time.

We could use KStream and the Java sdk but I would like to give a try to an implementation of PStore on top of spark-streaming-kafka (https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html <https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html>). 

My main concern is related to the http interface to push events. We will probably use KStream or websockets to load events in our Kafka topic used by an app channel.

Are you planning on supporting websockets as an alternative to batch import? 

Regards, 
Thomas Pocreau 

Le 5 juil. 2017 21:36, "Pat Ferrel" <pat@occamsmachete.com <ma...@occamsmachete.com>> a écrit :
No, we try not to fork :-) But it would be nice as you say. It can be done with a small intermediary app that just reads from a Kafka topic and send events to a localhost EventServer, which would allow events to be custom extracted from say log files (typical contents of Kafka). We’ve done this in non-PIO projects.

The intermediary app should use Spark streaming. I may have a snippet of code around if you need it but it just saves to micro-batch files. You’d have to use the PIO Java-SDK to send them to the EventServer. A relatively simple thing.

Donald, what did you have in mind for deeper integration? I guess we could cut out the intermediate app and integrate into a new Kafka aware EventServer endpoint where the raw topic input is stored in the EventStore. This would force any log filtering onto the Kafka source.


On Jul 5, 2017, at 10:20 AM, Donald Szeto <donald@apache.org <ma...@apache.org>> wrote:

Hi Thomas,

Supporting Kafka is definitely interesting and desirable. Are you looking to sinking your Kafka messages to event store for batch processing, or stream processing directly from Kafka? The latter would require more work because Apache PIO does not yet support streaming properly.

Folks from ActionML might have a flavor of PIO that works with Kafka.

Regards,
Donald

On Tue, Jul 4, 2017 at 8:34 AM, Thomas POCREAU <thomas.pocreau@iadvize.com <ma...@iadvize.com>> wrote:
Hi,

Thanks a lot for this awesome project.

I have a question regarding Kafka and it's possible integration as an Event Store.
Do you have any plan on this matter ? 
Are you aware of someone working on a similar sujet ?

Regards,
Thomas.






Re: Kafka support as Event Store

Posted by Thomas POCREAU <th...@iadvize.com>.
After some talks in intern, I misunderstood our needs.
Indeed, we will use Kafka HDFS connector
<http://docs.confluent.io/current/connect/connect-hdfs/docs/index.html> to
dump expired data.
So we will basically have Kafka for the fresh events and HDFS for the past
events.


2017-07-06 7:06 GMT+02:00 Thomas POCREAU <th...@iadvize.com>:

> Hi,
>
> Thanks for your responses.
>
> Our goal is to use kafka as our main event store for event sourcing. I'm
> pretty sure that kafka can be used with an infinite retention time.
>
> We could use KStream and the Java sdk but I would like to give a try to an
> implementation of PStore on top of spark-streaming-kafka (
> https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> ).
>
> My main concern is related to the http interface to push events. We will
> probably use KStream or websockets to load events in our Kafka topic used
> by an app channel.
>
> Are you planning on supporting websockets as an alternative to batch
> import?
>
> Regards,
> Thomas Pocreau
>
> Le 5 juil. 2017 21:36, "Pat Ferrel" <pa...@occamsmachete.com> a écrit :
>
>> No, we try not to fork :-) But it would be nice as you say. It can be
>> done with a small intermediary app that just reads from a Kafka topic and
>> send events to a localhost EventServer, which would allow events to be
>> custom extracted from say log files (typical contents of Kafka). We’ve done
>> this in non-PIO projects.
>>
>> The intermediary app should use Spark streaming. I may have a snippet of
>> code around if you need it but it just saves to micro-batch files. You’d
>> have to use the PIO Java-SDK to send them to the EventServer. A relatively
>> simple thing.
>>
>> Donald, what did you have in mind for deeper integration? I guess we
>> could cut out the intermediate app and integrate into a new Kafka aware
>> EventServer endpoint where the raw topic input is stored in the EventStore.
>> This would force any log filtering onto the Kafka source.
>>
>>
>> On Jul 5, 2017, at 10:20 AM, Donald Szeto <do...@apache.org> wrote:
>>
>> Hi Thomas,
>>
>> Supporting Kafka is definitely interesting and desirable. Are you looking
>> to sinking your Kafka messages to event store for batch processing, or
>> stream processing directly from Kafka? The latter would require more work
>> because Apache PIO does not yet support streaming properly.
>>
>> Folks from ActionML might have a flavor of PIO that works with Kafka.
>>
>> Regards,
>> Donald
>>
>> On Tue, Jul 4, 2017 at 8:34 AM, Thomas POCREAU <
>> thomas.pocreau@iadvize.com> wrote:
>>
>>> Hi,
>>>
>>> Thanks a lot for this awesome project.
>>>
>>> I have a question regarding Kafka and it's possible integration as an
>>> Event Store.
>>> Do you have any plan on this matter ?
>>> Are you aware of someone working on a similar sujet ?
>>>
>>> Regards,
>>> Thomas.
>>>
>>>
>>
>>

Re: Kafka support as Event Store

Posted by Thomas POCREAU <th...@iadvize.com>.
Hi,

Thanks for your responses.

Our goal is to use kafka as our main event store for event sourcing. I'm
pretty sure that kafka can be used with an infinite retention time.

We could use KStream and the Java sdk but I would like to give a try to an
implementation of PStore on top of spark-streaming-kafka (
https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
).

My main concern is related to the http interface to push events. We will
probably use KStream or websockets to load events in our Kafka topic used
by an app channel.

Are you planning on supporting websockets as an alternative to batch
import?

Regards,
Thomas Pocreau

Le 5 juil. 2017 21:36, "Pat Ferrel" <pa...@occamsmachete.com> a écrit :

> No, we try not to fork :-) But it would be nice as you say. It can be done
> with a small intermediary app that just reads from a Kafka topic and send
> events to a localhost EventServer, which would allow events to be custom
> extracted from say log files (typical contents of Kafka). We’ve done this
> in non-PIO projects.
>
> The intermediary app should use Spark streaming. I may have a snippet of
> code around if you need it but it just saves to micro-batch files. You’d
> have to use the PIO Java-SDK to send them to the EventServer. A relatively
> simple thing.
>
> Donald, what did you have in mind for deeper integration? I guess we could
> cut out the intermediate app and integrate into a new Kafka aware
> EventServer endpoint where the raw topic input is stored in the EventStore.
> This would force any log filtering onto the Kafka source.
>
>
> On Jul 5, 2017, at 10:20 AM, Donald Szeto <do...@apache.org> wrote:
>
> Hi Thomas,
>
> Supporting Kafka is definitely interesting and desirable. Are you looking
> to sinking your Kafka messages to event store for batch processing, or
> stream processing directly from Kafka? The latter would require more work
> because Apache PIO does not yet support streaming properly.
>
> Folks from ActionML might have a flavor of PIO that works with Kafka.
>
> Regards,
> Donald
>
> On Tue, Jul 4, 2017 at 8:34 AM, Thomas POCREAU <thomas.pocreau@iadvize.com
> > wrote:
>
>> Hi,
>>
>> Thanks a lot for this awesome project.
>>
>> I have a question regarding Kafka and it's possible integration as an
>> Event Store.
>> Do you have any plan on this matter ?
>> Are you aware of someone working on a similar sujet ?
>>
>> Regards,
>> Thomas.
>>
>>
>
>

Re: Kafka support as Event Store

Posted by Pat Ferrel <pa...@occamsmachete.com>.
No, we try not to fork :-) But it would be nice as you say. It can be done with a small intermediary app that just reads from a Kafka topic and send events to a localhost EventServer, which would allow events to be custom extracted from say log files (typical contents of Kafka). We’ve done this in non-PIO projects.

The intermediary app should use Spark streaming. I may have a snippet of code around if you need it but it just saves to micro-batch files. You’d have to use the PIO Java-SDK to send them to the EventServer. A relatively simple thing.

Donald, what did you have in mind for deeper integration? I guess we could cut out the intermediate app and integrate into a new Kafka aware EventServer endpoint where the raw topic input is stored in the EventStore. This would force any log filtering onto the Kafka source.


On Jul 5, 2017, at 10:20 AM, Donald Szeto <do...@apache.org> wrote:

Hi Thomas,

Supporting Kafka is definitely interesting and desirable. Are you looking to sinking your Kafka messages to event store for batch processing, or stream processing directly from Kafka? The latter would require more work because Apache PIO does not yet support streaming properly.

Folks from ActionML might have a flavor of PIO that works with Kafka.

Regards,
Donald

On Tue, Jul 4, 2017 at 8:34 AM, Thomas POCREAU <thomas.pocreau@iadvize.com <ma...@iadvize.com>> wrote:
Hi,

Thanks a lot for this awesome project.

I have a question regarding Kafka and it's possible integration as an Event Store.
Do you have any plan on this matter ? 
Are you aware of someone working on a similar sujet ?

Regards,
Thomas.




Re: Kafka support as Event Store

Posted by Mars Hall <ma...@heroku.com>.
Hello Thomas,

Kafka is not storage, it is fundamentally streaming logs with a limited retention window.

To work with PIO today, a Kafka consumer would need to be implemented to take messages and transform/insert them into persistent PIO event storage, like Postgres or HBase.

As Donald asked, are you trying to build an engine with Spark Streaming? I'm curious about how to do this too: both the PIO `predict` runtime for Spark Streaming and the [Kafka] events stream consumer to push events into PIO.

*Mars

( <> .. <> )

> On Jul 5, 2017, at 10:20, Donald Szeto <do...@apache.org> wrote:
> 
> Hi Thomas,
> 
> Supporting Kafka is definitely interesting and desirable. Are you looking to sinking your Kafka messages to event store for batch processing, or stream processing directly from Kafka? The latter would require more work because Apache PIO does not yet support streaming properly.
> 
> Folks from ActionML might have a flavor of PIO that works with Kafka.
> 
> Regards,
> Donald
> 
> On Tue, Jul 4, 2017 at 8:34 AM, Thomas POCREAU <th...@iadvize.com> wrote:
> Hi,
> 
> Thanks a lot for this awesome project.
> 
> I have a question regarding Kafka and it's possible integration as an Event Store.
> Do you have any plan on this matter ? 
> Are you aware of someone working on a similar sujet ?
> 
> Regards,
> Thomas.
> 
> 


Re: Kafka support as Event Store

Posted by Donald Szeto <do...@apache.org>.
Hi Thomas,

Supporting Kafka is definitely interesting and desirable. Are you looking
to sinking your Kafka messages to event store for batch processing, or
stream processing directly from Kafka? The latter would require more work
because Apache PIO does not yet support streaming properly.

Folks from ActionML might have a flavor of PIO that works with Kafka.

Regards,
Donald

On Tue, Jul 4, 2017 at 8:34 AM, Thomas POCREAU <th...@iadvize.com>
wrote:

> Hi,
>
> Thanks a lot for this awesome project.
>
> I have a question regarding Kafka and it's possible integration as an
> Event Store.
> Do you have any plan on this matter ?
> Are you aware of someone working on a similar sujet ?
>
> Regards,
> Thomas.
>
>