You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Tinte garcia, Miguel Angel" <mi...@atos.net> on 2015/06/03 18:18:45 UTC
HBaseSink for Flume
Hi devs,
Sorry for asking in developer list but I’d like to ask you about a very technical issue and probably you know better than anyone about flume code. I want to include HBaseSink in EmbeddedAgent configuration but currently seems not to be included, despite I’ve changed the code and compiled it again:
org.apache.flume.FlumeException: Component type of HBaseSink is not in allowed types of [AVRO, HBASE, HDFS]
Could you please let me know if there is somehow to add this functionality to EmbeddedAgent class?
Thanks in advance
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
RE: HBaseSink for Flume
Posted by "Tinte garcia, Miguel Angel" <mi...@atos.net>.
Hi Roshan,
I've been discussing about this with Hari Shreedharan in thread. Basically, I have already changed that point to allow HBASE sink and I've recompiled the flume-ng-embedded jar. Also, I've included flume-ng-auth library and executed the Embedded agent. At this moment, I've detected some error during starting Embedded Agent. It looks it can't access to HBase table list from Embedded agent code. Any idea to solve this?
Thanks for your help
-----Original Message-----
From: Roshan Naik [mailto:roshan@hortonworks.com]
Sent: Wednesday, June 10, 2015 10:12 PM
To: dev@flume.apache.org
Subject: Re: HBaseSink for Flume
The Embedded agent actually has some code in it to limit itself only to a list of allowed sinks/sources/channels in EmbeddedAgentConfiguration.java
Below is the list
private static final String[] ALLOWED_SOURCES = {
SOURCE_TYPE_EMBEDDED_ALIAS,
SOURCE_TYPE_EMBEDDED,
};
private static final String[] ALLOWED_CHANNELS = {
CHANNEL_TYPE_MEMORY,
CHANNEL_TYPE_FILE
};
private static final String[] ALLOWED_SINKS = {
SINK_TYPE_AVRO
};
private static final String[] ALLOWED_SINK_PROCESSORS = {
SINK_PROCESSOR_TYPE_DEFAULT,
SINK_PROCESSOR_TYPE_FAILOVER,
SINK_PROCESSOR_TYPE_LOAD_BALANCE
};
-roshan
On 6/10/15 12:37 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>Adding HBase and HDFS to embedded agent is non-trivial, especially with
>changing versions and dependencies. It is unlikely your hack will
>actually work without having everything hbase-client needs in its
>classpath (take a look at how flume-ng script adds everything to its
>own classpath). I would suggest running hbase classpath and adding
>eveyrhing from that to your app¹s classpath and trying.
>
>Also, it looks like at some point the hbase sink is not started, not
>sure why. Frankly, no one has really tested this, neither is it really
>supported.
>
>Thanks,
>Hari Shreedharan
>
>
>
>
>> On Jun 10, 2015, at 3:13 AM, Tinte garcia, Miguel Angel
>><mi...@atos.net> wrote:
>>
>> Hi,
>> Yes, it looks it works including flume-ng-auth in the classpath.
>>However, after configuring agent with properties it fails when tries
>>to
>>start:
>> ...
>> properties.put("hbaseSink.table", "Models");
>> ...
>> agent.configure(properties);
>> agent.start();
>> ...
>> org.apache.flume.FlumeException: Could not load table, Models
>>from Hbase
>> SEVERE: ...Error while sending data to
>>HBasejava.lang.IllegalStateException: Cannot put events unless started
>> java.lang.IllegalStateException: Cannot put events unless started
>> ...
>>
>> It looks it can't access to Hbase table defined in properties. Should
>>I configure some property in the agent to access to HBase?
>>
>> Thanks again
>>
>>
>> -----Original Message-----
>> From: Hari Shreedharan [mailto:hshreedharan@apache.org]
>> Sent: Saturday, June 06, 2015 12:08 AM
>> To: dev@flume.apache.org
>> Subject: Re: HBaseSink for Flume
>>
>> No, you need to add flume-ng-auth module to your classpath.
>>
>> On Fri, Jun 5, 2015 at 2:38 AM, Tinte garcia, Miguel Angel <
>>miguel.tinte@atos.net> wrote:
>>
>>> Hi Hari,
>>> Thanks for the answer. I've changed it to "Hbase" and this error has
>>> disappeared but it seems that there is a problem with sink factory
>>> when looking for HBase sink:
>>>
>>> Jun 05, 2015 11:25:47 AM
>>> com.sun.jersey.spi.container.ContainerResponse
>>> mapMappableContainerException
>>> SEVERE: The exception contained within MappableContainerException
>>>could not be mapped to a response, re-throwing to the HTTP container
>>> java.lang.NoClassDefFoundError:
>>>org/apache/flume/auth/PrivilegedExecutor
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:191)
>>> at
>>>
>>>org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.
>>>jav
>>>a:67)
>>> at
>>>
>>>org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:
>>>41)
>>> at
>>>
>>>org.apache.flume.node.AbstractConfigurationProvider.loadSinks(Abstrac
>>>tCo
>>>nfigurationProvider.java:415)
>>> at
>>>
>>>org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(
>>>A
>>> bstractConfigurationProvider.java:103)
>>>
>>> Should I modify some class on org.apache.flume.sink package as well?
>>>
>>> Thanks
>>>
>>>
>>> -----Original Message-----
>>> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
>>> Sent: Thursday, June 04, 2015 12:45 AM
>>> To: dev@flume.apache.org
>>> Subject: Re: HBaseSink for Flume
>>>
>>> Did you simply try ³hbase² rather than HBaseSink?
>>>
>>> Thanks,
>>> Hari Shreedharan
>>>
>>>
>>>
>>>
>>>> On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <
>>> miguel.tinte@atos.net> wrote:
>>>>
>>>> Hi devs,
>>>> Sorry for asking in developer list but I¹d like to ask you about a
>>>> very
>>> technical issue and probably you know better than anyone about flume
>>>code.
>>> I want to include HBaseSink in EmbeddedAgent configuration but
>>>currently seems not to be included, despite I¹ve changed the code and
>>>compiled it
>>> again:
>>>> org.apache.flume.FlumeException: Component type of HBaseSink is not
>>>> in
>>> allowed types of [AVRO, HBASE, HDFS]
>>>>
>>>> Could you please let me know if there is somehow to add this
>>> functionality to EmbeddedAgent class?
>>>>
>>>> Thanks in advance
>>>>
>>>> This e-mail and the documents attached are confidential and
>>>> intended
>>> solely for the addressee; it may also be privileged. If you receive
>>>this e-mail in error, please notify the sender immediately and
>>>destroy it.
>>>> As its integrity cannot be secured on the Internet, the Atos group
>>> liability cannot be triggered for the message content. Although the
>>> sender endeavors to maintain a computer virus-free network, the
>>> sender does not warrant that this transmission is virus-free and
>>> will not be liable for any damages resulting from any virus transmitted.
>>>>
>>>> Este mensaje y los ficheros adjuntos pueden contener información
>>> confidencial destinada solamente a la(s) persona(s) mencionadas
>>> anteriormente y pueden estar protegidos por secreto profesional.
>>>> Si usted recibe este correo electrónico por error, gracias por
>>>> informar
>>> inmediatamente al remitente y destruir el mensaje.
>>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>>> Atos
>>> no se hace responsable por su contenido. Su contenido no constituye
>>>ningún compromiso para el grupo Atos, salvo ratificación escrita por
>>>ambas partes.
>>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>> emisor no puede garantizar nada al respecto y no será responsable de
>>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>>
>>> This e-mail and the documents attached are confidential and intended
>>>solely for the addressee; it may also be privileged. If you receive
>>>this e-mail in error, please notify the sender immediately and
>>>destroy it.
>>> As its integrity cannot be secured on the Internet, the Atos group
>>>liability cannot be triggered for the message content. Although the
>>>sender endeavors to maintain a computer virus-free network, the
>>>sender does not warrant that this transmission is virus-free and
>>>will not be liable for any damages resulting from any virus transmitted.
>>>
>>> Este mensaje y los ficheros adjuntos pueden contener información
>>>confidencial destinada solamente a la(s) persona(s) mencionadas
>>>anteriormente y pueden estar protegidos por secreto profesional.
>>> Si usted recibe este correo electrónico por error, gracias por
>>>informar inmediatamente al remitente y destruir el mensaje.
>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>>Atos no se hace responsable por su contenido. Su contenido no
>>>constituye ningún compromiso para el grupo Atos, salvo ratificación
>>>escrita por ambas partes.
>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>>emisor no puede garantizar nada al respecto y no será responsable de
>>>cualesquiera daños que puedan resultar de una transmisión de virus.
>>>
>> This e-mail and the documents attached are confidential and intended
>>solely for the addressee; it may also be privileged. If you receive
>>this e-mail in error, please notify the sender immediately and destroy it.
>> As its integrity cannot be secured on the Internet, the Atos group
>>liability cannot be triggered for the message content. Although the
>>sender endeavors to maintain a computer virus-free network, the sender
>>does not warrant that this transmission is virus-free and will not be
>>liable for any damages resulting from any virus transmitted.
>>
>> Este mensaje y los ficheros adjuntos pueden contener información
>>confidencial destinada solamente a la(s) persona(s) mencionadas
>>anteriormente y pueden estar protegidos por secreto profesional.
>> Si usted recibe este correo electrónico por error, gracias por
>>informar inmediatamente al remitente y destruir el mensaje.
>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>Atos no se hace responsable por su contenido. Su contenido no
>>constituye ningún compromiso para el grupo Atos, salvo ratificación
>>escrita por ambas partes.
>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>emisor no puede garantizar nada al respecto y no será responsable de
>>cualesquiera daños que puedan resultar de una transmisión de virus.
>
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
RE: HBaseSink for Flume
Posted by Hari Shreedharan <hs...@cloudera.com>.
No, you need to include HBase’s classpath into your own app’s so that HBase Client API has all the classes it requires to start up. Also required in your classpath is hbase-site.xml, which contains the config used to connect to HBase.
Thanks, Hari
On Thu, Jun 11, 2015 at 9:00 AM, Tinte garcia, Miguel Angel
<mi...@atos.net> wrote:
> Could you please specify a bit more what you mean? Is it including HBASE_PATH as an environment variable into java war?
> -----Original Message-----
> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
> Sent: Thursday, June 11, 2015 5:47 PM
> To: dev@flume.apache.org
> Cc: dev@flume.apache.org
> Subject: RE: HBaseSink for Flume
> Did you add the hbase class path to your own and also the hbase config to your class path?
> Thanks, Hari
> On Thu, Jun 11, 2015 at 1:36 AM, Tinte garcia, Miguel Angel <mi...@atos.net> wrote:
>> Thanks for your comments Hari,
>> I think the problem is at the time of accessing Hbase tables:
>> org.apache.flume.FlumeException: Could not load table, Models from
>> Hbase As it can't check tablename and ColumnFamilies configuration, agent can't start. I've trying different configurations unsuccessfully:
>> hBaseConfig.set("hbase.master", hostname)
>> hBaseConfig.set("hbase.zookeeper.quorum", hostname)
>>
>> hBaseConfig.set("hbase.zookeeper.property.clientPort", "2181"); Is there something wrong with this configuration?
>> Thanks
>> -----Original Message-----
>> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
>> Sent: Wednesday, June 10, 2015 9:38 PM
>> To: dev@flume.apache.org
>> Subject: Re: HBaseSink for Flume
>> Adding HBase and HDFS to embedded agent is non-trivial, especially with changing versions and dependencies. It is unlikely your hack will actually work without having everything hbase-client needs in its classpath (take a look at how flume-ng script adds everything to its own classpath). I would suggest running hbase classpath and adding eveyrhing from that to your app’s classpath and trying.
>> Also, it looks like at some point the hbase sink is not started, not sure why. Frankly, no one has really tested this, neither is it really supported.
>> Thanks,
>> Hari Shreedharan
>>> On Jun 10, 2015, at 3:13 AM, Tinte garcia, Miguel Angel <mi...@atos.net> wrote:
>>>
>>> Hi,
>>> Yes, it looks it works including flume-ng-auth in the classpath. However, after configuring agent with properties it fails when tries to start:
>>> ...
>>> properties.put("hbaseSink.table", "Models");
>>> ...
>>> agent.configure(properties);
>>> agent.start();
>>> ...
>>> org.apache.flume.FlumeException: Could not load table, Models from Hbase
>>> SEVERE: ...Error while sending data to HBasejava.lang.IllegalStateException: Cannot put events unless started
>>> java.lang.IllegalStateException: Cannot put events unless started
>>> ...
>>>
>>> It looks it can't access to Hbase table defined in properties. Should I configure some property in the agent to access to HBase?
>>>
>>> Thanks again
>>>
>>>
>>> -----Original Message-----
>>> From: Hari Shreedharan [mailto:hshreedharan@apache.org]
>>> Sent: Saturday, June 06, 2015 12:08 AM
>>> To: dev@flume.apache.org
>>> Subject: Re: HBaseSink for Flume
>>>
>>> No, you need to add flume-ng-auth module to your classpath.
>>>
>>> On Fri, Jun 5, 2015 at 2:38 AM, Tinte garcia, Miguel Angel < miguel.tinte@atos.net> wrote:
>>>
>>>> Hi Hari,
>>>> Thanks for the answer. I've changed it to "Hbase" and this error has
>>>> disappeared but it seems that there is a problem with sink factory
>>>> when looking for HBase sink:
>>>>
>>>> Jun 05, 2015 11:25:47 AM
>>>> com.sun.jersey.spi.container.ContainerResponse
>>>> mapMappableContainerException
>>>> SEVERE: The exception contained within MappableContainerException
>>>> could not be mapped to a response, re-throwing to the HTTP container
>>>> java.lang.NoClassDefFoundError: org/apache/flume/auth/PrivilegedExecutor
>>>> at java.lang.Class.forName0(Native Method)
>>>> at java.lang.Class.forName(Class.java:191)
>>>> at
>>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
>>>> at
>>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
>>>> at
>>>> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
>>>> at
>>>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration
>>>> (
>>>> A
>>>> bstractConfigurationProvider.java:103)
>>>>
>>>> Should I modify some class on org.apache.flume.sink package as well?
>>>>
>>>> Thanks
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
>>>> Sent: Thursday, June 04, 2015 12:45 AM
>>>> To: dev@flume.apache.org
>>>> Subject: Re: HBaseSink for Flume
>>>>
>>>> Did you simply try “hbase” rather than HBaseSink?
>>>>
>>>> Thanks,
>>>> Hari Shreedharan
>>>>
>>>>
>>>>
>>>>
>>>>> On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <
>>>> miguel.tinte@atos.net> wrote:
>>>>>
>>>>> Hi devs,
>>>>> Sorry for asking in developer list but I’d like to ask you about a
>>>>> very
>>>> technical issue and probably you know better than anyone about flume code.
>>>> I want to include HBaseSink in EmbeddedAgent configuration but
>>>> currently seems not to be included, despite I’ve changed the code
>>>> and compiled it
>>>> again:
>>>>> org.apache.flume.FlumeException: Component type of HBaseSink is not
>>>>> in
>>>> allowed types of [AVRO, HBASE, HDFS]
>>>>>
>>>>> Could you please let me know if there is somehow to add this
>>>> functionality to EmbeddedAgent class?
>>>>>
>>>>> Thanks in advance
>>>>>
>>>>> This e-mail and the documents attached are confidential and
>>>>> intended
>>>> solely for the addressee; it may also be privileged. If you receive
>>>> this e-mail in error, please notify the sender immediately and destroy it.
>>>>> As its integrity cannot be secured on the Internet, the Atos group
>>>> liability cannot be triggered for the message content. Although the
>>>> sender endeavors to maintain a computer virus-free network, the
>>>> sender does not warrant that this transmission is virus-free and
>>>> will not be liable for any damages resulting from any virus transmitted.
>>>>>
>>>>> Este mensaje y los ficheros adjuntos pueden contener información
>>>> confidencial destinada solamente a la(s) persona(s) mencionadas
>>>> anteriormente y pueden estar protegidos por secreto profesional.
>>>>> Si usted recibe este correo electrónico por error, gracias por
>>>>> informar
>>>> inmediatamente al remitente y destruir el mensaje.
>>>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>>>> Atos
>>>> no se hace responsable por su contenido. Su contenido no constituye
>>>> ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>>>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>>> emisor no puede garantizar nada al respecto y no será responsable de
>>>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>>>
>>>> This e-mail and the documents attached are confidential and intended
>>>> solely for the addressee; it may also be privileged. If you receive
>>>> this e-mail in error, please notify the sender immediately and destroy it.
>>>> As its integrity cannot be secured on the Internet, the Atos group
>>>> liability cannot be triggered for the message content. Although the
>>>> sender endeavors to maintain a computer virus-free network, the
>>>> sender does not warrant that this transmission is virus-free and
>>>> will not be liable for any damages resulting from any virus transmitted.
>>>>
>>>> Este mensaje y los ficheros adjuntos pueden contener información
>>>> confidencial destinada solamente a la(s) persona(s) mencionadas
>>>> anteriormente y pueden estar protegidos por secreto profesional.
>>>> Si usted recibe este correo electrónico por error, gracias por
>>>> informar inmediatamente al remitente y destruir el mensaje.
>>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>>> Atos no se hace responsable por su contenido. Su contenido no
>>>> constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>>> emisor no puede garantizar nada al respecto y no será responsable de
>>>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>>>
>>> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
>>> As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
>>>
>>> Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
>>> Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
>>> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
>> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
>> As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
>> Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
>> Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
>> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
> As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
> Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
> Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
RE: HBaseSink for Flume
Posted by "Tinte garcia, Miguel Angel" <mi...@atos.net>.
Could you please specify a bit more what you mean? Is it including HBASE_PATH as an environment variable into java war?
-----Original Message-----
From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Thursday, June 11, 2015 5:47 PM
To: dev@flume.apache.org
Cc: dev@flume.apache.org
Subject: RE: HBaseSink for Flume
Did you add the hbase class path to your own and also the hbase config to your class path?
Thanks, Hari
On Thu, Jun 11, 2015 at 1:36 AM, Tinte garcia, Miguel Angel <mi...@atos.net> wrote:
> Thanks for your comments Hari,
> I think the problem is at the time of accessing Hbase tables:
> org.apache.flume.FlumeException: Could not load table, Models from
> Hbase As it can't check tablename and ColumnFamilies configuration, agent can't start. I've trying different configurations unsuccessfully:
> hBaseConfig.set("hbase.master", hostname)
> hBaseConfig.set("hbase.zookeeper.quorum", hostname)
>
> hBaseConfig.set("hbase.zookeeper.property.clientPort", "2181"); Is there something wrong with this configuration?
> Thanks
> -----Original Message-----
> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
> Sent: Wednesday, June 10, 2015 9:38 PM
> To: dev@flume.apache.org
> Subject: Re: HBaseSink for Flume
> Adding HBase and HDFS to embedded agent is non-trivial, especially with changing versions and dependencies. It is unlikely your hack will actually work without having everything hbase-client needs in its classpath (take a look at how flume-ng script adds everything to its own classpath). I would suggest running hbase classpath and adding eveyrhing from that to your app’s classpath and trying.
> Also, it looks like at some point the hbase sink is not started, not sure why. Frankly, no one has really tested this, neither is it really supported.
> Thanks,
> Hari Shreedharan
>> On Jun 10, 2015, at 3:13 AM, Tinte garcia, Miguel Angel <mi...@atos.net> wrote:
>>
>> Hi,
>> Yes, it looks it works including flume-ng-auth in the classpath. However, after configuring agent with properties it fails when tries to start:
>> ...
>> properties.put("hbaseSink.table", "Models");
>> ...
>> agent.configure(properties);
>> agent.start();
>> ...
>> org.apache.flume.FlumeException: Could not load table, Models from Hbase
>> SEVERE: ...Error while sending data to HBasejava.lang.IllegalStateException: Cannot put events unless started
>> java.lang.IllegalStateException: Cannot put events unless started
>> ...
>>
>> It looks it can't access to Hbase table defined in properties. Should I configure some property in the agent to access to HBase?
>>
>> Thanks again
>>
>>
>> -----Original Message-----
>> From: Hari Shreedharan [mailto:hshreedharan@apache.org]
>> Sent: Saturday, June 06, 2015 12:08 AM
>> To: dev@flume.apache.org
>> Subject: Re: HBaseSink for Flume
>>
>> No, you need to add flume-ng-auth module to your classpath.
>>
>> On Fri, Jun 5, 2015 at 2:38 AM, Tinte garcia, Miguel Angel < miguel.tinte@atos.net> wrote:
>>
>>> Hi Hari,
>>> Thanks for the answer. I've changed it to "Hbase" and this error has
>>> disappeared but it seems that there is a problem with sink factory
>>> when looking for HBase sink:
>>>
>>> Jun 05, 2015 11:25:47 AM
>>> com.sun.jersey.spi.container.ContainerResponse
>>> mapMappableContainerException
>>> SEVERE: The exception contained within MappableContainerException
>>> could not be mapped to a response, re-throwing to the HTTP container
>>> java.lang.NoClassDefFoundError: org/apache/flume/auth/PrivilegedExecutor
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:191)
>>> at
>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
>>> at
>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
>>> at
>>> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
>>> at
>>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration
>>> (
>>> A
>>> bstractConfigurationProvider.java:103)
>>>
>>> Should I modify some class on org.apache.flume.sink package as well?
>>>
>>> Thanks
>>>
>>>
>>> -----Original Message-----
>>> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
>>> Sent: Thursday, June 04, 2015 12:45 AM
>>> To: dev@flume.apache.org
>>> Subject: Re: HBaseSink for Flume
>>>
>>> Did you simply try “hbase” rather than HBaseSink?
>>>
>>> Thanks,
>>> Hari Shreedharan
>>>
>>>
>>>
>>>
>>>> On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <
>>> miguel.tinte@atos.net> wrote:
>>>>
>>>> Hi devs,
>>>> Sorry for asking in developer list but I’d like to ask you about a
>>>> very
>>> technical issue and probably you know better than anyone about flume code.
>>> I want to include HBaseSink in EmbeddedAgent configuration but
>>> currently seems not to be included, despite I’ve changed the code
>>> and compiled it
>>> again:
>>>> org.apache.flume.FlumeException: Component type of HBaseSink is not
>>>> in
>>> allowed types of [AVRO, HBASE, HDFS]
>>>>
>>>> Could you please let me know if there is somehow to add this
>>> functionality to EmbeddedAgent class?
>>>>
>>>> Thanks in advance
>>>>
>>>> This e-mail and the documents attached are confidential and
>>>> intended
>>> solely for the addressee; it may also be privileged. If you receive
>>> this e-mail in error, please notify the sender immediately and destroy it.
>>>> As its integrity cannot be secured on the Internet, the Atos group
>>> liability cannot be triggered for the message content. Although the
>>> sender endeavors to maintain a computer virus-free network, the
>>> sender does not warrant that this transmission is virus-free and
>>> will not be liable for any damages resulting from any virus transmitted.
>>>>
>>>> Este mensaje y los ficheros adjuntos pueden contener información
>>> confidencial destinada solamente a la(s) persona(s) mencionadas
>>> anteriormente y pueden estar protegidos por secreto profesional.
>>>> Si usted recibe este correo electrónico por error, gracias por
>>>> informar
>>> inmediatamente al remitente y destruir el mensaje.
>>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>>> Atos
>>> no se hace responsable por su contenido. Su contenido no constituye
>>> ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>> emisor no puede garantizar nada al respecto y no será responsable de
>>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>>
>>> This e-mail and the documents attached are confidential and intended
>>> solely for the addressee; it may also be privileged. If you receive
>>> this e-mail in error, please notify the sender immediately and destroy it.
>>> As its integrity cannot be secured on the Internet, the Atos group
>>> liability cannot be triggered for the message content. Although the
>>> sender endeavors to maintain a computer virus-free network, the
>>> sender does not warrant that this transmission is virus-free and
>>> will not be liable for any damages resulting from any virus transmitted.
>>>
>>> Este mensaje y los ficheros adjuntos pueden contener información
>>> confidencial destinada solamente a la(s) persona(s) mencionadas
>>> anteriormente y pueden estar protegidos por secreto profesional.
>>> Si usted recibe este correo electrónico por error, gracias por
>>> informar inmediatamente al remitente y destruir el mensaje.
>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>> Atos no se hace responsable por su contenido. Su contenido no
>>> constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>> emisor no puede garantizar nada al respecto y no será responsable de
>>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>>
>> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
>> As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
>>
>> Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
>> Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
>> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
> As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
> Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
> Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
RE: HBaseSink for Flume
Posted by Hari Shreedharan <hs...@cloudera.com>.
Did you add the hbase class path to your own and also the hbase config to your class path?
Thanks, Hari
On Thu, Jun 11, 2015 at 1:36 AM, Tinte garcia, Miguel Angel
<mi...@atos.net> wrote:
> Thanks for your comments Hari,
> I think the problem is at the time of accessing Hbase tables:
> org.apache.flume.FlumeException: Could not load table, Models from Hbase
> As it can't check tablename and ColumnFamilies configuration, agent can't start. I've trying different configurations unsuccessfully:
> hBaseConfig.set("hbase.master", hostname)
> hBaseConfig.set("hbase.zookeeper.quorum", hostname)
> hBaseConfig.set("hbase.zookeeper.property.clientPort", "2181");
> Is there something wrong with this configuration?
> Thanks
> -----Original Message-----
> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
> Sent: Wednesday, June 10, 2015 9:38 PM
> To: dev@flume.apache.org
> Subject: Re: HBaseSink for Flume
> Adding HBase and HDFS to embedded agent is non-trivial, especially with changing versions and dependencies. It is unlikely your hack will actually work without having everything hbase-client needs in its classpath (take a look at how flume-ng script adds everything to its own classpath). I would suggest running hbase classpath and adding eveyrhing from that to your app’s classpath and trying.
> Also, it looks like at some point the hbase sink is not started, not sure why. Frankly, no one has really tested this, neither is it really supported.
> Thanks,
> Hari Shreedharan
>> On Jun 10, 2015, at 3:13 AM, Tinte garcia, Miguel Angel <mi...@atos.net> wrote:
>>
>> Hi,
>> Yes, it looks it works including flume-ng-auth in the classpath. However, after configuring agent with properties it fails when tries to start:
>> ...
>> properties.put("hbaseSink.table", "Models");
>> ...
>> agent.configure(properties);
>> agent.start();
>> ...
>> org.apache.flume.FlumeException: Could not load table, Models from Hbase
>> SEVERE: ...Error while sending data to HBasejava.lang.IllegalStateException: Cannot put events unless started
>> java.lang.IllegalStateException: Cannot put events unless started
>> ...
>>
>> It looks it can't access to Hbase table defined in properties. Should I configure some property in the agent to access to HBase?
>>
>> Thanks again
>>
>>
>> -----Original Message-----
>> From: Hari Shreedharan [mailto:hshreedharan@apache.org]
>> Sent: Saturday, June 06, 2015 12:08 AM
>> To: dev@flume.apache.org
>> Subject: Re: HBaseSink for Flume
>>
>> No, you need to add flume-ng-auth module to your classpath.
>>
>> On Fri, Jun 5, 2015 at 2:38 AM, Tinte garcia, Miguel Angel < miguel.tinte@atos.net> wrote:
>>
>>> Hi Hari,
>>> Thanks for the answer. I've changed it to "Hbase" and this error has
>>> disappeared but it seems that there is a problem with sink factory
>>> when looking for HBase sink:
>>>
>>> Jun 05, 2015 11:25:47 AM
>>> com.sun.jersey.spi.container.ContainerResponse
>>> mapMappableContainerException
>>> SEVERE: The exception contained within MappableContainerException
>>> could not be mapped to a response, re-throwing to the HTTP container
>>> java.lang.NoClassDefFoundError: org/apache/flume/auth/PrivilegedExecutor
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:191)
>>> at
>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
>>> at
>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
>>> at
>>> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
>>> at
>>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(
>>> A
>>> bstractConfigurationProvider.java:103)
>>>
>>> Should I modify some class on org.apache.flume.sink package as well?
>>>
>>> Thanks
>>>
>>>
>>> -----Original Message-----
>>> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
>>> Sent: Thursday, June 04, 2015 12:45 AM
>>> To: dev@flume.apache.org
>>> Subject: Re: HBaseSink for Flume
>>>
>>> Did you simply try “hbase” rather than HBaseSink?
>>>
>>> Thanks,
>>> Hari Shreedharan
>>>
>>>
>>>
>>>
>>>> On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <
>>> miguel.tinte@atos.net> wrote:
>>>>
>>>> Hi devs,
>>>> Sorry for asking in developer list but I’d like to ask you about a
>>>> very
>>> technical issue and probably you know better than anyone about flume code.
>>> I want to include HBaseSink in EmbeddedAgent configuration but
>>> currently seems not to be included, despite I’ve changed the code and
>>> compiled it
>>> again:
>>>> org.apache.flume.FlumeException: Component type of HBaseSink is not
>>>> in
>>> allowed types of [AVRO, HBASE, HDFS]
>>>>
>>>> Could you please let me know if there is somehow to add this
>>> functionality to EmbeddedAgent class?
>>>>
>>>> Thanks in advance
>>>>
>>>> This e-mail and the documents attached are confidential and intended
>>> solely for the addressee; it may also be privileged. If you receive
>>> this e-mail in error, please notify the sender immediately and destroy it.
>>>> As its integrity cannot be secured on the Internet, the Atos group
>>> liability cannot be triggered for the message content. Although the
>>> sender endeavors to maintain a computer virus-free network, the
>>> sender does not warrant that this transmission is virus-free and will
>>> not be liable for any damages resulting from any virus transmitted.
>>>>
>>>> Este mensaje y los ficheros adjuntos pueden contener información
>>> confidencial destinada solamente a la(s) persona(s) mencionadas
>>> anteriormente y pueden estar protegidos por secreto profesional.
>>>> Si usted recibe este correo electrónico por error, gracias por
>>>> informar
>>> inmediatamente al remitente y destruir el mensaje.
>>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>>> Atos
>>> no se hace responsable por su contenido. Su contenido no constituye
>>> ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>> emisor no puede garantizar nada al respecto y no será responsable de
>>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>>
>>> This e-mail and the documents attached are confidential and intended
>>> solely for the addressee; it may also be privileged. If you receive
>>> this e-mail in error, please notify the sender immediately and destroy it.
>>> As its integrity cannot be secured on the Internet, the Atos group
>>> liability cannot be triggered for the message content. Although the
>>> sender endeavors to maintain a computer virus-free network, the
>>> sender does not warrant that this transmission is virus-free and will
>>> not be liable for any damages resulting from any virus transmitted.
>>>
>>> Este mensaje y los ficheros adjuntos pueden contener información
>>> confidencial destinada solamente a la(s) persona(s) mencionadas
>>> anteriormente y pueden estar protegidos por secreto profesional.
>>> Si usted recibe este correo electrónico por error, gracias por
>>> informar inmediatamente al remitente y destruir el mensaje.
>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>> Atos no se hace responsable por su contenido. Su contenido no
>>> constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>> emisor no puede garantizar nada al respecto y no será responsable de
>>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>>
>> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
>> As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
>>
>> Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
>> Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
>> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
> As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
> Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
> Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
RE: HBaseSink for Flume
Posted by "Tinte garcia, Miguel Angel" <mi...@atos.net>.
Thanks for your comments Hari,
I think the problem is at the time of accessing Hbase tables:
org.apache.flume.FlumeException: Could not load table, Models from Hbase
As it can't check tablename and ColumnFamilies configuration, agent can't start. I've trying different configurations unsuccessfully:
hBaseConfig.set("hbase.master", hostname)
hBaseConfig.set("hbase.zookeeper.quorum", hostname)
hBaseConfig.set("hbase.zookeeper.property.clientPort", "2181");
Is there something wrong with this configuration?
Thanks
-----Original Message-----
From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Wednesday, June 10, 2015 9:38 PM
To: dev@flume.apache.org
Subject: Re: HBaseSink for Flume
Adding HBase and HDFS to embedded agent is non-trivial, especially with changing versions and dependencies. It is unlikely your hack will actually work without having everything hbase-client needs in its classpath (take a look at how flume-ng script adds everything to its own classpath). I would suggest running hbase classpath and adding eveyrhing from that to your app’s classpath and trying.
Also, it looks like at some point the hbase sink is not started, not sure why. Frankly, no one has really tested this, neither is it really supported.
Thanks,
Hari Shreedharan
> On Jun 10, 2015, at 3:13 AM, Tinte garcia, Miguel Angel <mi...@atos.net> wrote:
>
> Hi,
> Yes, it looks it works including flume-ng-auth in the classpath. However, after configuring agent with properties it fails when tries to start:
> ...
> properties.put("hbaseSink.table", "Models");
> ...
> agent.configure(properties);
> agent.start();
> ...
> org.apache.flume.FlumeException: Could not load table, Models from Hbase
> SEVERE: ...Error while sending data to HBasejava.lang.IllegalStateException: Cannot put events unless started
> java.lang.IllegalStateException: Cannot put events unless started
> ...
>
> It looks it can't access to Hbase table defined in properties. Should I configure some property in the agent to access to HBase?
>
> Thanks again
>
>
> -----Original Message-----
> From: Hari Shreedharan [mailto:hshreedharan@apache.org]
> Sent: Saturday, June 06, 2015 12:08 AM
> To: dev@flume.apache.org
> Subject: Re: HBaseSink for Flume
>
> No, you need to add flume-ng-auth module to your classpath.
>
> On Fri, Jun 5, 2015 at 2:38 AM, Tinte garcia, Miguel Angel < miguel.tinte@atos.net> wrote:
>
>> Hi Hari,
>> Thanks for the answer. I've changed it to "Hbase" and this error has
>> disappeared but it seems that there is a problem with sink factory
>> when looking for HBase sink:
>>
>> Jun 05, 2015 11:25:47 AM
>> com.sun.jersey.spi.container.ContainerResponse
>> mapMappableContainerException
>> SEVERE: The exception contained within MappableContainerException
>> could not be mapped to a response, re-throwing to the HTTP container
>> java.lang.NoClassDefFoundError: org/apache/flume/auth/PrivilegedExecutor
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:191)
>> at
>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
>> at
>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
>> at
>> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
>> at
>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(
>> A
>> bstractConfigurationProvider.java:103)
>>
>> Should I modify some class on org.apache.flume.sink package as well?
>>
>> Thanks
>>
>>
>> -----Original Message-----
>> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
>> Sent: Thursday, June 04, 2015 12:45 AM
>> To: dev@flume.apache.org
>> Subject: Re: HBaseSink for Flume
>>
>> Did you simply try “hbase” rather than HBaseSink?
>>
>> Thanks,
>> Hari Shreedharan
>>
>>
>>
>>
>>> On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <
>> miguel.tinte@atos.net> wrote:
>>>
>>> Hi devs,
>>> Sorry for asking in developer list but I’d like to ask you about a
>>> very
>> technical issue and probably you know better than anyone about flume code.
>> I want to include HBaseSink in EmbeddedAgent configuration but
>> currently seems not to be included, despite I’ve changed the code and
>> compiled it
>> again:
>>> org.apache.flume.FlumeException: Component type of HBaseSink is not
>>> in
>> allowed types of [AVRO, HBASE, HDFS]
>>>
>>> Could you please let me know if there is somehow to add this
>> functionality to EmbeddedAgent class?
>>>
>>> Thanks in advance
>>>
>>> This e-mail and the documents attached are confidential and intended
>> solely for the addressee; it may also be privileged. If you receive
>> this e-mail in error, please notify the sender immediately and destroy it.
>>> As its integrity cannot be secured on the Internet, the Atos group
>> liability cannot be triggered for the message content. Although the
>> sender endeavors to maintain a computer virus-free network, the
>> sender does not warrant that this transmission is virus-free and will
>> not be liable for any damages resulting from any virus transmitted.
>>>
>>> Este mensaje y los ficheros adjuntos pueden contener información
>> confidencial destinada solamente a la(s) persona(s) mencionadas
>> anteriormente y pueden estar protegidos por secreto profesional.
>>> Si usted recibe este correo electrónico por error, gracias por
>>> informar
>> inmediatamente al remitente y destruir el mensaje.
>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>> Atos
>> no se hace responsable por su contenido. Su contenido no constituye
>> ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>> emisor no puede garantizar nada al respecto y no será responsable de
>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>
>> This e-mail and the documents attached are confidential and intended
>> solely for the addressee; it may also be privileged. If you receive
>> this e-mail in error, please notify the sender immediately and destroy it.
>> As its integrity cannot be secured on the Internet, the Atos group
>> liability cannot be triggered for the message content. Although the
>> sender endeavors to maintain a computer virus-free network, the
>> sender does not warrant that this transmission is virus-free and will
>> not be liable for any damages resulting from any virus transmitted.
>>
>> Este mensaje y los ficheros adjuntos pueden contener información
>> confidencial destinada solamente a la(s) persona(s) mencionadas
>> anteriormente y pueden estar protegidos por secreto profesional.
>> Si usted recibe este correo electrónico por error, gracias por
>> informar inmediatamente al remitente y destruir el mensaje.
>> Al no estar asegurada la integridad de este mensaje sobre la red,
>> Atos no se hace responsable por su contenido. Su contenido no
>> constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>> emisor no puede garantizar nada al respecto y no será responsable de
>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
> As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
>
> Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
> Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
Re: HBaseSink for Flume
Posted by Roshan Naik <ro...@hortonworks.com>.
The Embedded agent actually has some code in it to limit itself only to a
list of allowed sinks/sources/channels in EmbeddedAgentConfiguration.java
Below is the list
private static final String[] ALLOWED_SOURCES = {
SOURCE_TYPE_EMBEDDED_ALIAS,
SOURCE_TYPE_EMBEDDED,
};
private static final String[] ALLOWED_CHANNELS = {
CHANNEL_TYPE_MEMORY,
CHANNEL_TYPE_FILE
};
private static final String[] ALLOWED_SINKS = {
SINK_TYPE_AVRO
};
private static final String[] ALLOWED_SINK_PROCESSORS = {
SINK_PROCESSOR_TYPE_DEFAULT,
SINK_PROCESSOR_TYPE_FAILOVER,
SINK_PROCESSOR_TYPE_LOAD_BALANCE
};
-roshan
On 6/10/15 12:37 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>Adding HBase and HDFS to embedded agent is non-trivial, especially with
>changing versions and dependencies. It is unlikely your hack will
>actually work without having everything hbase-client needs in its
>classpath (take a look at how flume-ng script adds everything to its own
>classpath). I would suggest running hbase classpath and adding eveyrhing
>from that to your app¹s classpath and trying.
>
>Also, it looks like at some point the hbase sink is not started, not sure
>why. Frankly, no one has really tested this, neither is it really
>supported.
>
>Thanks,
>Hari Shreedharan
>
>
>
>
>> On Jun 10, 2015, at 3:13 AM, Tinte garcia, Miguel Angel
>><mi...@atos.net> wrote:
>>
>> Hi,
>> Yes, it looks it works including flume-ng-auth in the classpath.
>>However, after configuring agent with properties it fails when tries to
>>start:
>> ...
>> properties.put("hbaseSink.table", "Models");
>> ...
>> agent.configure(properties);
>> agent.start();
>> ...
>> org.apache.flume.FlumeException: Could not load table, Models
>>from Hbase
>> SEVERE: ...Error while sending data to
>>HBasejava.lang.IllegalStateException: Cannot put events unless started
>> java.lang.IllegalStateException: Cannot put events unless started
>> ...
>>
>> It looks it can't access to Hbase table defined in properties. Should I
>>configure some property in the agent to access to HBase?
>>
>> Thanks again
>>
>>
>> -----Original Message-----
>> From: Hari Shreedharan [mailto:hshreedharan@apache.org]
>> Sent: Saturday, June 06, 2015 12:08 AM
>> To: dev@flume.apache.org
>> Subject: Re: HBaseSink for Flume
>>
>> No, you need to add flume-ng-auth module to your classpath.
>>
>> On Fri, Jun 5, 2015 at 2:38 AM, Tinte garcia, Miguel Angel <
>>miguel.tinte@atos.net> wrote:
>>
>>> Hi Hari,
>>> Thanks for the answer. I've changed it to "Hbase" and this error has
>>> disappeared but it seems that there is a problem with sink factory
>>> when looking for HBase sink:
>>>
>>> Jun 05, 2015 11:25:47 AM
>>> com.sun.jersey.spi.container.ContainerResponse
>>> mapMappableContainerException
>>> SEVERE: The exception contained within MappableContainerException
>>> could not be mapped to a response, re-throwing to the HTTP container
>>> java.lang.NoClassDefFoundError:
>>>org/apache/flume/auth/PrivilegedExecutor
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:191)
>>> at
>>>
>>>org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.jav
>>>a:67)
>>> at
>>>
>>>org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:
>>>41)
>>> at
>>>
>>>org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractCo
>>>nfigurationProvider.java:415)
>>> at
>>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(A
>>> bstractConfigurationProvider.java:103)
>>>
>>> Should I modify some class on org.apache.flume.sink package as well?
>>>
>>> Thanks
>>>
>>>
>>> -----Original Message-----
>>> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
>>> Sent: Thursday, June 04, 2015 12:45 AM
>>> To: dev@flume.apache.org
>>> Subject: Re: HBaseSink for Flume
>>>
>>> Did you simply try ³hbase² rather than HBaseSink?
>>>
>>> Thanks,
>>> Hari Shreedharan
>>>
>>>
>>>
>>>
>>>> On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <
>>> miguel.tinte@atos.net> wrote:
>>>>
>>>> Hi devs,
>>>> Sorry for asking in developer list but I¹d like to ask you about a
>>>> very
>>> technical issue and probably you know better than anyone about flume
>>>code.
>>> I want to include HBaseSink in EmbeddedAgent configuration but
>>> currently seems not to be included, despite I¹ve changed the code and
>>> compiled it
>>> again:
>>>> org.apache.flume.FlumeException: Component type of HBaseSink is not
>>>> in
>>> allowed types of [AVRO, HBASE, HDFS]
>>>>
>>>> Could you please let me know if there is somehow to add this
>>> functionality to EmbeddedAgent class?
>>>>
>>>> Thanks in advance
>>>>
>>>> This e-mail and the documents attached are confidential and intended
>>> solely for the addressee; it may also be privileged. If you receive
>>> this e-mail in error, please notify the sender immediately and destroy
>>>it.
>>>> As its integrity cannot be secured on the Internet, the Atos group
>>> liability cannot be triggered for the message content. Although the
>>> sender endeavors to maintain a computer virus-free network, the sender
>>> does not warrant that this transmission is virus-free and will not be
>>> liable for any damages resulting from any virus transmitted.
>>>>
>>>> Este mensaje y los ficheros adjuntos pueden contener información
>>> confidencial destinada solamente a la(s) persona(s) mencionadas
>>> anteriormente y pueden estar protegidos por secreto profesional.
>>>> Si usted recibe este correo electrónico por error, gracias por
>>>> informar
>>> inmediatamente al remitente y destruir el mensaje.
>>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>>> Atos
>>> no se hace responsable por su contenido. Su contenido no constituye
>>> ningún compromiso para el grupo Atos, salvo ratificación escrita por
>>>ambas partes.
>>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>> emisor no puede garantizar nada al respecto y no será responsable de
>>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>>
>>> This e-mail and the documents attached are confidential and intended
>>> solely for the addressee; it may also be privileged. If you receive
>>> this e-mail in error, please notify the sender immediately and destroy
>>>it.
>>> As its integrity cannot be secured on the Internet, the Atos group
>>> liability cannot be triggered for the message content. Although the
>>> sender endeavors to maintain a computer virus-free network, the sender
>>> does not warrant that this transmission is virus-free and will not be
>>> liable for any damages resulting from any virus transmitted.
>>>
>>> Este mensaje y los ficheros adjuntos pueden contener información
>>> confidencial destinada solamente a la(s) persona(s) mencionadas
>>> anteriormente y pueden estar protegidos por secreto profesional.
>>> Si usted recibe este correo electrónico por error, gracias por
>>> informar inmediatamente al remitente y destruir el mensaje.
>>> Al no estar asegurada la integridad de este mensaje sobre la red, Atos
>>> no se hace responsable por su contenido. Su contenido no constituye
>>> ningún compromiso para el grupo Atos, salvo ratificación escrita por
>>>ambas partes.
>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>> emisor no puede garantizar nada al respecto y no será responsable de
>>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>>
>> This e-mail and the documents attached are confidential and intended
>>solely for the addressee; it may also be privileged. If you receive this
>>e-mail in error, please notify the sender immediately and destroy it.
>> As its integrity cannot be secured on the Internet, the Atos group
>>liability cannot be triggered for the message content. Although the
>>sender endeavors to maintain a computer virus-free network, the sender
>>does not warrant that this transmission is virus-free and will not be
>>liable for any damages resulting from any virus transmitted.
>>
>> Este mensaje y los ficheros adjuntos pueden contener información
>>confidencial destinada solamente a la(s) persona(s) mencionadas
>>anteriormente y pueden estar protegidos por secreto profesional.
>> Si usted recibe este correo electrónico por error, gracias por informar
>>inmediatamente al remitente y destruir el mensaje.
>> Al no estar asegurada la integridad de este mensaje sobre la red, Atos
>>no se hace responsable por su contenido. Su contenido no constituye
>>ningún compromiso para el grupo Atos, salvo ratificación escrita por
>>ambas partes.
>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>emisor no puede garantizar nada al respecto y no será responsable de
>>cualesquiera daños que puedan resultar de una transmisión de virus.
>
Re: HBaseSink for Flume
Posted by Hari Shreedharan <hs...@cloudera.com>.
Adding HBase and HDFS to embedded agent is non-trivial, especially with changing versions and dependencies. It is unlikely your hack will actually work without having everything hbase-client needs in its classpath (take a look at how flume-ng script adds everything to its own classpath). I would suggest running hbase classpath and adding eveyrhing from that to your app’s classpath and trying.
Also, it looks like at some point the hbase sink is not started, not sure why. Frankly, no one has really tested this, neither is it really supported.
Thanks,
Hari Shreedharan
> On Jun 10, 2015, at 3:13 AM, Tinte garcia, Miguel Angel <mi...@atos.net> wrote:
>
> Hi,
> Yes, it looks it works including flume-ng-auth in the classpath. However, after configuring agent with properties it fails when tries to start:
> ...
> properties.put("hbaseSink.table", "Models");
> ...
> agent.configure(properties);
> agent.start();
> ...
> org.apache.flume.FlumeException: Could not load table, Models from Hbase
> SEVERE: ...Error while sending data to HBasejava.lang.IllegalStateException: Cannot put events unless started
> java.lang.IllegalStateException: Cannot put events unless started
> ...
>
> It looks it can't access to Hbase table defined in properties. Should I configure some property in the agent to access to HBase?
>
> Thanks again
>
>
> -----Original Message-----
> From: Hari Shreedharan [mailto:hshreedharan@apache.org]
> Sent: Saturday, June 06, 2015 12:08 AM
> To: dev@flume.apache.org
> Subject: Re: HBaseSink for Flume
>
> No, you need to add flume-ng-auth module to your classpath.
>
> On Fri, Jun 5, 2015 at 2:38 AM, Tinte garcia, Miguel Angel < miguel.tinte@atos.net> wrote:
>
>> Hi Hari,
>> Thanks for the answer. I've changed it to "Hbase" and this error has
>> disappeared but it seems that there is a problem with sink factory
>> when looking for HBase sink:
>>
>> Jun 05, 2015 11:25:47 AM
>> com.sun.jersey.spi.container.ContainerResponse
>> mapMappableContainerException
>> SEVERE: The exception contained within MappableContainerException
>> could not be mapped to a response, re-throwing to the HTTP container
>> java.lang.NoClassDefFoundError: org/apache/flume/auth/PrivilegedExecutor
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:191)
>> at
>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
>> at
>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
>> at
>> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
>> at
>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(A
>> bstractConfigurationProvider.java:103)
>>
>> Should I modify some class on org.apache.flume.sink package as well?
>>
>> Thanks
>>
>>
>> -----Original Message-----
>> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
>> Sent: Thursday, June 04, 2015 12:45 AM
>> To: dev@flume.apache.org
>> Subject: Re: HBaseSink for Flume
>>
>> Did you simply try “hbase” rather than HBaseSink?
>>
>> Thanks,
>> Hari Shreedharan
>>
>>
>>
>>
>>> On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <
>> miguel.tinte@atos.net> wrote:
>>>
>>> Hi devs,
>>> Sorry for asking in developer list but I’d like to ask you about a
>>> very
>> technical issue and probably you know better than anyone about flume code.
>> I want to include HBaseSink in EmbeddedAgent configuration but
>> currently seems not to be included, despite I’ve changed the code and
>> compiled it
>> again:
>>> org.apache.flume.FlumeException: Component type of HBaseSink is not
>>> in
>> allowed types of [AVRO, HBASE, HDFS]
>>>
>>> Could you please let me know if there is somehow to add this
>> functionality to EmbeddedAgent class?
>>>
>>> Thanks in advance
>>>
>>> This e-mail and the documents attached are confidential and intended
>> solely for the addressee; it may also be privileged. If you receive
>> this e-mail in error, please notify the sender immediately and destroy it.
>>> As its integrity cannot be secured on the Internet, the Atos group
>> liability cannot be triggered for the message content. Although the
>> sender endeavors to maintain a computer virus-free network, the sender
>> does not warrant that this transmission is virus-free and will not be
>> liable for any damages resulting from any virus transmitted.
>>>
>>> Este mensaje y los ficheros adjuntos pueden contener información
>> confidencial destinada solamente a la(s) persona(s) mencionadas
>> anteriormente y pueden estar protegidos por secreto profesional.
>>> Si usted recibe este correo electrónico por error, gracias por
>>> informar
>> inmediatamente al remitente y destruir el mensaje.
>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>> Atos
>> no se hace responsable por su contenido. Su contenido no constituye
>> ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>> emisor no puede garantizar nada al respecto y no será responsable de
>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>
>> This e-mail and the documents attached are confidential and intended
>> solely for the addressee; it may also be privileged. If you receive
>> this e-mail in error, please notify the sender immediately and destroy it.
>> As its integrity cannot be secured on the Internet, the Atos group
>> liability cannot be triggered for the message content. Although the
>> sender endeavors to maintain a computer virus-free network, the sender
>> does not warrant that this transmission is virus-free and will not be
>> liable for any damages resulting from any virus transmitted.
>>
>> Este mensaje y los ficheros adjuntos pueden contener información
>> confidencial destinada solamente a la(s) persona(s) mencionadas
>> anteriormente y pueden estar protegidos por secreto profesional.
>> Si usted recibe este correo electrónico por error, gracias por
>> informar inmediatamente al remitente y destruir el mensaje.
>> Al no estar asegurada la integridad de este mensaje sobre la red, Atos
>> no se hace responsable por su contenido. Su contenido no constituye
>> ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>> emisor no puede garantizar nada al respecto y no será responsable de
>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
> As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
>
> Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
> Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
RE: HBaseSink for Flume
Posted by "Tinte garcia, Miguel Angel" <mi...@atos.net>.
Hi,
Yes, it looks it works including flume-ng-auth in the classpath. However, after configuring agent with properties it fails when tries to start:
...
properties.put("hbaseSink.table", "Models");
...
agent.configure(properties);
agent.start();
...
org.apache.flume.FlumeException: Could not load table, Models from Hbase
SEVERE: ...Error while sending data to HBasejava.lang.IllegalStateException: Cannot put events unless started
java.lang.IllegalStateException: Cannot put events unless started
...
It looks it can't access to Hbase table defined in properties. Should I configure some property in the agent to access to HBase?
Thanks again
-----Original Message-----
From: Hari Shreedharan [mailto:hshreedharan@apache.org]
Sent: Saturday, June 06, 2015 12:08 AM
To: dev@flume.apache.org
Subject: Re: HBaseSink for Flume
No, you need to add flume-ng-auth module to your classpath.
On Fri, Jun 5, 2015 at 2:38 AM, Tinte garcia, Miguel Angel < miguel.tinte@atos.net> wrote:
> Hi Hari,
> Thanks for the answer. I've changed it to "Hbase" and this error has
> disappeared but it seems that there is a problem with sink factory
> when looking for HBase sink:
>
> Jun 05, 2015 11:25:47 AM
> com.sun.jersey.spi.container.ContainerResponse
> mapMappableContainerException
> SEVERE: The exception contained within MappableContainerException
> could not be mapped to a response, re-throwing to the HTTP container
> java.lang.NoClassDefFoundError: org/apache/flume/auth/PrivilegedExecutor
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:191)
> at
> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
> at
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
> at
> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
> at
> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(A
> bstractConfigurationProvider.java:103)
>
> Should I modify some class on org.apache.flume.sink package as well?
>
> Thanks
>
>
> -----Original Message-----
> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
> Sent: Thursday, June 04, 2015 12:45 AM
> To: dev@flume.apache.org
> Subject: Re: HBaseSink for Flume
>
> Did you simply try “hbase” rather than HBaseSink?
>
> Thanks,
> Hari Shreedharan
>
>
>
>
> > On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <
> miguel.tinte@atos.net> wrote:
> >
> > Hi devs,
> > Sorry for asking in developer list but I’d like to ask you about a
> > very
> technical issue and probably you know better than anyone about flume code.
> I want to include HBaseSink in EmbeddedAgent configuration but
> currently seems not to be included, despite I’ve changed the code and
> compiled it
> again:
> > org.apache.flume.FlumeException: Component type of HBaseSink is not
> > in
> allowed types of [AVRO, HBASE, HDFS]
> >
> > Could you please let me know if there is somehow to add this
> functionality to EmbeddedAgent class?
> >
> > Thanks in advance
> >
> > This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive
> this e-mail in error, please notify the sender immediately and destroy it.
> > As its integrity cannot be secured on the Internet, the Atos group
> liability cannot be triggered for the message content. Although the
> sender endeavors to maintain a computer virus-free network, the sender
> does not warrant that this transmission is virus-free and will not be
> liable for any damages resulting from any virus transmitted.
> >
> > Este mensaje y los ficheros adjuntos pueden contener información
> confidencial destinada solamente a la(s) persona(s) mencionadas
> anteriormente y pueden estar protegidos por secreto profesional.
> > Si usted recibe este correo electrónico por error, gracias por
> > informar
> inmediatamente al remitente y destruir el mensaje.
> > Al no estar asegurada la integridad de este mensaje sobre la red,
> > Atos
> no se hace responsable por su contenido. Su contenido no constituye
> ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
> > Aunque se esfuerza al máximo por mantener su red libre de virus, el
> emisor no puede garantizar nada al respecto y no será responsable de
> cualesquiera daños que puedan resultar de una transmisión de virus.
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive
> this e-mail in error, please notify the sender immediately and destroy it.
> As its integrity cannot be secured on the Internet, the Atos group
> liability cannot be triggered for the message content. Although the
> sender endeavors to maintain a computer virus-free network, the sender
> does not warrant that this transmission is virus-free and will not be
> liable for any damages resulting from any virus transmitted.
>
> Este mensaje y los ficheros adjuntos pueden contener información
> confidencial destinada solamente a la(s) persona(s) mencionadas
> anteriormente y pueden estar protegidos por secreto profesional.
> Si usted recibe este correo electrónico por error, gracias por
> informar inmediatamente al remitente y destruir el mensaje.
> Al no estar asegurada la integridad de este mensaje sobre la red, Atos
> no se hace responsable por su contenido. Su contenido no constituye
> ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
> Aunque se esfuerza al máximo por mantener su red libre de virus, el
> emisor no puede garantizar nada al respecto y no será responsable de
> cualesquiera daños que puedan resultar de una transmisión de virus.
>
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
Re: HBaseSink for Flume
Posted by Hari Shreedharan <hs...@apache.org>.
No, you need to add flume-ng-auth module to your classpath.
On Fri, Jun 5, 2015 at 2:38 AM, Tinte garcia, Miguel Angel <
miguel.tinte@atos.net> wrote:
> Hi Hari,
> Thanks for the answer. I've changed it to "Hbase" and this error has
> disappeared but it seems that there is a problem with sink factory when
> looking for HBase sink:
>
> Jun 05, 2015 11:25:47 AM com.sun.jersey.spi.container.ContainerResponse
> mapMappableContainerException
> SEVERE: The exception contained within MappableContainerException could
> not be mapped to a response, re-throwing to the HTTP container
> java.lang.NoClassDefFoundError: org/apache/flume/auth/PrivilegedExecutor
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:191)
> at
> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
> at
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
> at
> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
> at
> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
>
> Should I modify some class on org.apache.flume.sink package as well?
>
> Thanks
>
>
> -----Original Message-----
> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
> Sent: Thursday, June 04, 2015 12:45 AM
> To: dev@flume.apache.org
> Subject: Re: HBaseSink for Flume
>
> Did you simply try “hbase” rather than HBaseSink?
>
> Thanks,
> Hari Shreedharan
>
>
>
>
> > On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <
> miguel.tinte@atos.net> wrote:
> >
> > Hi devs,
> > Sorry for asking in developer list but I’d like to ask you about a very
> technical issue and probably you know better than anyone about flume code.
> I want to include HBaseSink in EmbeddedAgent configuration but currently
> seems not to be included, despite I’ve changed the code and compiled it
> again:
> > org.apache.flume.FlumeException: Component type of HBaseSink is not in
> allowed types of [AVRO, HBASE, HDFS]
> >
> > Could you please let me know if there is somehow to add this
> functionality to EmbeddedAgent class?
> >
> > Thanks in advance
> >
> > This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive this
> e-mail in error, please notify the sender immediately and destroy it.
> > As its integrity cannot be secured on the Internet, the Atos group
> liability cannot be triggered for the message content. Although the sender
> endeavors to maintain a computer virus-free network, the sender does not
> warrant that this transmission is virus-free and will not be liable for any
> damages resulting from any virus transmitted.
> >
> > Este mensaje y los ficheros adjuntos pueden contener información
> confidencial destinada solamente a la(s) persona(s) mencionadas
> anteriormente y pueden estar protegidos por secreto profesional.
> > Si usted recibe este correo electrónico por error, gracias por informar
> inmediatamente al remitente y destruir el mensaje.
> > Al no estar asegurada la integridad de este mensaje sobre la red, Atos
> no se hace responsable por su contenido. Su contenido no constituye ningún
> compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
> > Aunque se esfuerza al máximo por mantener su red libre de virus, el
> emisor no puede garantizar nada al respecto y no será responsable de
> cualesquiera daños que puedan resultar de una transmisión de virus.
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive this
> e-mail in error, please notify the sender immediately and destroy it.
> As its integrity cannot be secured on the Internet, the Atos group
> liability cannot be triggered for the message content. Although the sender
> endeavors to maintain a computer virus-free network, the sender does not
> warrant that this transmission is virus-free and will not be liable for any
> damages resulting from any virus transmitted.
>
> Este mensaje y los ficheros adjuntos pueden contener información
> confidencial destinada solamente a la(s) persona(s) mencionadas
> anteriormente y pueden estar protegidos por secreto profesional.
> Si usted recibe este correo electrónico por error, gracias por informar
> inmediatamente al remitente y destruir el mensaje.
> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no
> se hace responsable por su contenido. Su contenido no constituye ningún
> compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor
> no puede garantizar nada al respecto y no será responsable de cualesquiera
> daños que puedan resultar de una transmisión de virus.
>
RE: HBaseSink for Flume
Posted by "Tinte garcia, Miguel Angel" <mi...@atos.net>.
Hi Hari,
Thanks for the answer. I've changed it to "Hbase" and this error has disappeared but it seems that there is a problem with sink factory when looking for HBase sink:
Jun 05, 2015 11:25:47 AM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException
SEVERE: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.lang.NoClassDefFoundError: org/apache/flume/auth/PrivilegedExecutor
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
Should I modify some class on org.apache.flume.sink package as well?
Thanks
-----Original Message-----
From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Thursday, June 04, 2015 12:45 AM
To: dev@flume.apache.org
Subject: Re: HBaseSink for Flume
Did you simply try “hbase” rather than HBaseSink?
Thanks,
Hari Shreedharan
> On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <mi...@atos.net> wrote:
>
> Hi devs,
> Sorry for asking in developer list but I’d like to ask you about a very technical issue and probably you know better than anyone about flume code. I want to include HBaseSink in EmbeddedAgent configuration but currently seems not to be included, despite I’ve changed the code and compiled it again:
> org.apache.flume.FlumeException: Component type of HBaseSink is not in allowed types of [AVRO, HBASE, HDFS]
>
> Could you please let me know if there is somehow to add this functionality to EmbeddedAgent class?
>
> Thanks in advance
>
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
> As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
>
> Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
> Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
Re: HBaseSink for Flume
Posted by Hari Shreedharan <hs...@cloudera.com>.
Did you simply try “hbase” rather than HBaseSink?
Thanks,
Hari Shreedharan
> On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <mi...@atos.net> wrote:
>
> Hi devs,
> Sorry for asking in developer list but I’d like to ask you about a very technical issue and probably you know better than anyone about flume code. I want to include HBaseSink in EmbeddedAgent configuration but currently seems not to be included, despite I’ve changed the code and compiled it again:
> org.apache.flume.FlumeException: Component type of HBaseSink is not in allowed types of [AVRO, HBASE, HDFS]
>
> Could you please let me know if there is somehow to add this functionality to EmbeddedAgent class?
>
> Thanks in advance
>
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it.
> As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
>
> Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
> Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje.
> Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes.
> Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.