You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Otto Fowler <ot...@gmail.com> on 2017/04/12 11:49:16 UTC
[HELP!]Writing to HDFS from Storm
I am trying to write to HDFS from ParserBolt, but I’m getting the following
exception:
Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
user=storm, access=WRITE,
inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1827)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1811)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1794)
at
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4011)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1102)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:630)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
The HDFS directory is created as such:
self.__params.HdfsResource(self.__params.hdfs_metron_apps_extensions_working,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
mode=0775)
As the hdfs write handlers I am logging in as such:
HdfsSecurityUtil.login(stormConfig, fsConf);
FileSystem fileSystem = FileSystem.get(fsConf);
I am not sure what is different from the indexing hdfs writer setup here,
but what I’m doing obviously is not working.
Any ideas?
- the branch:
https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
I am not up to date with master.
Re: [HELP!]Writing to HDFS from Storm
Posted by Otto Fowler <ot...@gmail.com>.
Thanks Justin,
I’m working on rebasing right now, then I’ll be able to try it out.
On April 12, 2017 at 08:26:08, Justin Leet (justinjleet@gmail.com) wrote:
We changed the HDFS owner of /apps/metron/indexing/indexed a bit ago to be
metron:hadoop, specifically because Storm wasn't writing to HDFS (and had
perms issues). If you look into the indexing output directories, my
expectation is that you aren't getting any data out (and the Storm indexing
topology is throwing permissions errors).
PR for that change: https://github.com/apache/incubator-metron/pull/488
Your branch has the old code:
https://github.com/ottobackwards/incubator-metron/blob/parser_deploy/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py#L98
New code:
https://github.com/apache/incubator-metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
To set up ownership on startup with the hadoop group after you pull in
master (or at least make the changes from that PR)
self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
group=self.__params.hadoop_group,
mode=0775,
)
Justin
On Wed, Apr 12, 2017 at 8:24 AM, Otto Fowler <ot...@gmail.com>
wrote:
> "The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and
are
> deployed
> if required. So the writes for HDFS are the unpacking the bundles ( with
> the unpacked jars being loaded into a classloader ).
>
> If the unbundled is the same as the bundle, there is no op. In this case,
> it is first run.
>
> So this is the parser bolt using the HDFS backed extension loader.”
>
>
> This is part of the parser side loading, also feeds into the stellar load
> from hdfs stuff.
>
> * metron has extensions
> * extensions are packaged with their configurations and their ‘bundle’ (
> the nar part )
> * extension bundles are deployed to hdfs *lib directory
> * parts of the system that use extensions discover and load them, with
> class loaders setup for the bundle
> * part of that, the bundles are unpackaged into a ‘working’ area if
> required ( so if you drop a new version into the lib, the next app will
> unpack the new version )
> * thus, the bundle system needs to be able to write to the working areas
in
> hdfs
> * ???
> * profit
>
>
> On April 12, 2017 at 08:11:28, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> I’m curious: Otto, what’s your use case for writing to HDFS from a
parsers?
>
> Simon
>
> > On 12 Apr 2017, at 13:04, Justin Leet <ju...@gmail.com> wrote:
> >
> > Chown it to metron:hadoop and it'll work. Storm is in the hadoop group
> and
> > with 775 will be able to write.
> >
> > On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dl...@gmail.com>
> wrote:
> >
> >> It's curious to me that you're writing directly from parsing, but I
> suspect
> >> that your parsing topology is running as the storm user and it can't
> write
> >> to those directories.
> >>
> >> -D...
> >>
> >> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ot...@gmail.com>
> >> wrote:
> >>
> >>> The indexing dir is created:
> >>>
> >>>
self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
> >>> type="directory",
> >>> action="create_on_execute",
> >>> owner=self.__params.metron_user,
> >>> group=self.__params.metron_group,
> >>> mode=0775,
> >>> )
> >>>
> >>>
> >>>
> >>>
> >>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
> >>> wrote:
> >>>
> >>>
> >>> I am trying to write to HDFS from ParserBolt, but I’m getting the
> >> following
> >>> exception:
> >>>
> >>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
> >>> user=storm, access=WRITE,
> >>> inode="/apps/metron/extension_working/framework":metron:
> hdfs:drwxrwxr-x
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> check(FSPermissionChecker.java:319)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> check(FSPermissionChecker.java:292)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> checkPermission(FSPermissionChecker.java:213)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> checkPermission(FSPermissionChecker.java:190)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> >>> checkPermission(FSDirectory.java:1827)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> >>> checkPermission(FSDirectory.java:1811)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> checkAncestorAccess(
> >>> FSDirectory.java:1794)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
> >>> FSDirMkdirOp.java:71)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
> >>> FSNamesystem.java:4011)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> >>> mkdirs(NameNodeRpcServer.java:1102)
> >>> at
> >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
> >>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
> >>> deTranslatorPB.java:630)
> >>> at
> >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
> >>> ClientNamenodeProtocol$2.callBlockingMethod(
> >> ClientNamenodeProtocolProtos.
> >>> java)
> >>> at
> >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> ProtoBufRpcInvoker.call(
> >>> ProtobufRpcEngine.java:640)
> >>>
> >>>
> >>> The HDFS directory is created as such:
> >>>
> >>> self.__params.HdfsResource(self.__params.hdfs_metron_
> >>> apps_extensions_working,
> >>> type="directory",
> >>> action="create_on_execute",
> >>> owner=self.__params.metron_user,
> >>> mode=0775)
> >>>
> >>>
> >>> As the hdfs write handlers I am logging in as such:
> >>>
> >>> HdfsSecurityUtil.login(stormConfig, fsConf);
> >>> FileSystem fileSystem = FileSystem.get(fsConf);
> >>>
> >>> I am not sure what is different from the indexing hdfs writer setup
> here,
> >>> but what I’m doing obviously is not working.
> >>>
> >>> Any ideas?
> >>>
> >>>
> >>> - the branch:
> >>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
> >>>
> >>> I am not up to date with master.
> >>>
> >>
>
Re: [HELP!]Writing to HDFS from Storm
Posted by Justin Leet <ju...@gmail.com>.
We changed the HDFS owner of /apps/metron/indexing/indexed a bit ago to be
metron:hadoop, specifically because Storm wasn't writing to HDFS (and had
perms issues). If you look into the indexing output directories, my
expectation is that you aren't getting any data out (and the Storm indexing
topology is throwing permissions errors).
PR for that change: https://github.com/apache/incubator-metron/pull/488
Your branch has the old code:
https://github.com/ottobackwards/incubator-metron/blob/parser_deploy/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py#L98
New code:
https://github.com/apache/incubator-metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
To set up ownership on startup with the hadoop group after you pull in
master (or at least make the changes from that PR)
self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
group=self.__params.hadoop_group,
mode=0775,
)
Justin
On Wed, Apr 12, 2017 at 8:24 AM, Otto Fowler <ot...@gmail.com>
wrote:
> "The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are
> deployed
> if required. So the writes for HDFS are the unpacking the bundles ( with
> the unpacked jars being loaded into a classloader ).
>
> If the unbundled is the same as the bundle, there is no op. In this case,
> it is first run.
>
> So this is the parser bolt using the HDFS backed extension loader.”
>
>
> This is part of the parser side loading, also feeds into the stellar load
> from hdfs stuff.
>
> * metron has extensions
> * extensions are packaged with their configurations and their ‘bundle’ (
> the nar part )
> * extension bundles are deployed to hdfs *lib directory
> * parts of the system that use extensions discover and load them, with
> class loaders setup for the bundle
> * part of that, the bundles are unpackaged into a ‘working’ area if
> required ( so if you drop a new version into the lib, the next app will
> unpack the new version )
> * thus, the bundle system needs to be able to write to the working areas in
> hdfs
> * ???
> * profit
>
>
> On April 12, 2017 at 08:11:28, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> I’m curious: Otto, what’s your use case for writing to HDFS from a parsers?
>
> Simon
>
> > On 12 Apr 2017, at 13:04, Justin Leet <ju...@gmail.com> wrote:
> >
> > Chown it to metron:hadoop and it'll work. Storm is in the hadoop group
> and
> > with 775 will be able to write.
> >
> > On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dl...@gmail.com>
> wrote:
> >
> >> It's curious to me that you're writing directly from parsing, but I
> suspect
> >> that your parsing topology is running as the storm user and it can't
> write
> >> to those directories.
> >>
> >> -D...
> >>
> >> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ot...@gmail.com>
> >> wrote:
> >>
> >>> The indexing dir is created:
> >>>
> >>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
> >>> type="directory",
> >>> action="create_on_execute",
> >>> owner=self.__params.metron_user,
> >>> group=self.__params.metron_group,
> >>> mode=0775,
> >>> )
> >>>
> >>>
> >>>
> >>>
> >>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
> >>> wrote:
> >>>
> >>>
> >>> I am trying to write to HDFS from ParserBolt, but I’m getting the
> >> following
> >>> exception:
> >>>
> >>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
> >>> user=storm, access=WRITE,
> >>> inode="/apps/metron/extension_working/framework":metron:
> hdfs:drwxrwxr-x
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> check(FSPermissionChecker.java:319)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> check(FSPermissionChecker.java:292)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> checkPermission(FSPermissionChecker.java:213)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> checkPermission(FSPermissionChecker.java:190)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> >>> checkPermission(FSDirectory.java:1827)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> >>> checkPermission(FSDirectory.java:1811)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> checkAncestorAccess(
> >>> FSDirectory.java:1794)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
> >>> FSDirMkdirOp.java:71)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
> >>> FSNamesystem.java:4011)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> >>> mkdirs(NameNodeRpcServer.java:1102)
> >>> at
> >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
> >>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
> >>> deTranslatorPB.java:630)
> >>> at
> >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
> >>> ClientNamenodeProtocol$2.callBlockingMethod(
> >> ClientNamenodeProtocolProtos.
> >>> java)
> >>> at
> >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> ProtoBufRpcInvoker.call(
> >>> ProtobufRpcEngine.java:640)
> >>>
> >>>
> >>> The HDFS directory is created as such:
> >>>
> >>> self.__params.HdfsResource(self.__params.hdfs_metron_
> >>> apps_extensions_working,
> >>> type="directory",
> >>> action="create_on_execute",
> >>> owner=self.__params.metron_user,
> >>> mode=0775)
> >>>
> >>>
> >>> As the hdfs write handlers I am logging in as such:
> >>>
> >>> HdfsSecurityUtil.login(stormConfig, fsConf);
> >>> FileSystem fileSystem = FileSystem.get(fsConf);
> >>>
> >>> I am not sure what is different from the indexing hdfs writer setup
> here,
> >>> but what I’m doing obviously is not working.
> >>>
> >>> Any ideas?
> >>>
> >>>
> >>> - the branch:
> >>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
> >>>
> >>> I am not up to date with master.
> >>>
> >>
>
Re: [HELP!]Writing to HDFS from Storm
Posted by Otto Fowler <ot...@gmail.com>.
The VFSClassloader pulls it in locally
On April 12, 2017 at 13:59:35, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
Right, I would certainly agree about a single deployment of the package,
but when it's pulling in and unpacked, I would expect we'd pull that
locally, though of course that adds complexity, maybe some of storms local
file caching could help us there. Still that I think is a discussion for
another thread!
Simon
On 12 Apr 2017, at 18:48, Otto Fowler <ot...@gmail.com> wrote:
What Casey and I had talked about ( and he actually implemented first with
stellar ) is having a single shared repository, without having to tie the
deployment
to the machine.
This all still has to stand review so we’ll see.
Maybe Casey can comment.
On April 12, 2017 at 08:30:31, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
Makes sense, however, would it make sense to unpack to local file system,
which would then avoid further HDFS ops?
On 12 Apr 2017, at 13:24, Otto Fowler <ot...@gmail.com> wrote:
"The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are
deployed
if required. So the writes for HDFS are the unpacking the bundles ( with
the unpacked jars being loaded into a classloader ).
If the unbundled is the same as the bundle, there is no op. In this case,
it is first run.
So this is the parser bolt using the HDFS backed extension loader.”
This is part of the parser side loading, also feeds into the stellar load
from hdfs stuff.
* metron has extensions
* extensions are packaged with their configurations and their ‘bundle’ (
the nar part )
* extension bundles are deployed to hdfs *lib directory
* parts of the system that use extensions discover and load them, with
class loaders setup for the bundle
* part of that, the bundles are unpackaged into a ‘working’ area if
required ( so if you drop a new version into the lib, the next app will
unpack the new version )
* thus, the bundle system needs to be able to write to the working areas in
hdfs
* ???
* profit
On April 12, 2017 at 08:11:28, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
I’m curious: Otto, what’s your use case for writing to HDFS from a parsers?
Simon
> On 12 Apr 2017, at 13:04, Justin Leet <ju...@gmail.com> wrote:
>
> Chown it to metron:hadoop and it'll work. Storm is in the hadoop group and
> with 775 will be able to write.
>
> On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dl...@gmail.com> wrote:
>
>> It's curious to me that you're writing directly from parsing, but I
suspect
>> that your parsing topology is running as the storm user and it can't
write
>> to those directories.
>>
>> -D...
>>
>> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> The indexing dir is created:
>>>
>>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
>>> type="directory",
>>> action="create_on_execute",
>>> owner=self.__params.metron_user,
>>> group=self.__params.metron_group,
>>> mode=0775,
>>> )
>>>
>>>
>>>
>>>
>>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
>>> wrote:
>>>
>>>
>>> I am trying to write to HDFS from ParserBolt, but I’m getting the
>> following
>>> exception:
>>>
>>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
>>> user=storm, access=WRITE,
>>> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> check(FSPermissionChecker.java:319)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> check(FSPermissionChecker.java:292)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> checkPermission(FSPermissionChecker.java:213)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> checkPermission(FSPermissionChecker.java:190)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>>> checkPermission(FSDirectory.java:1827)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>>> checkPermission(FSDirectory.java:1811)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(
>>> FSDirectory.java:1794)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
>>> FSDirMkdirOp.java:71)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
>>> FSNamesystem.java:4011)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
>>> mkdirs(NameNodeRpcServer.java:1102)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
>>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
>>> deTranslatorPB.java:630)
>>> at
>>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
>>> ClientNamenodeProtocol$2.callBlockingMethod(
>> ClientNamenodeProtocolProtos.
>>> java)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(
>>> ProtobufRpcEngine.java:640)
>>>
>>>
>>> The HDFS directory is created as such:
>>>
>>> self.__params.HdfsResource(self.__params.hdfs_metron_
>>> apps_extensions_working,
>>> type="directory",
>>> action="create_on_execute",
>>> owner=self.__params.metron_user,
>>> mode=0775)
>>>
>>>
>>> As the hdfs write handlers I am logging in as such:
>>>
>>> HdfsSecurityUtil.login(stormConfig, fsConf);
>>> FileSystem fileSystem = FileSystem.get(fsConf);
>>>
>>> I am not sure what is different from the indexing hdfs writer setup
here,
>>> but what I’m doing obviously is not working.
>>>
>>> Any ideas?
>>>
>>>
>>> - the branch:
>>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
>>>
>>> I am not up to date with master.
>>>
>>
Re: [HELP!]Writing to HDFS from Storm
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Right, I would certainly agree about a single deployment of the package, but when it's pulling in and unpacked, I would expect we'd pull that locally, though of course that adds complexity, maybe some of storms local file caching could help us there. Still that I think is a discussion for another thread!
Simon
> On 12 Apr 2017, at 18:48, Otto Fowler <ot...@gmail.com> wrote:
>
> What Casey and I had talked about ( and he actually implemented first with stellar ) is having a single shared repository, without having to tie the deployment
> to the machine.
>
> This all still has to stand review so we’ll see.
> Maybe Casey can comment.
>
>
>
>> On April 12, 2017 at 08:30:31, Simon Elliston Ball (simon@simonellistonball.com) wrote:
>>
>> Makes sense, however, would it make sense to unpack to local file system, which would then avoid further HDFS ops?
>>
>>
>>> On 12 Apr 2017, at 13:24, Otto Fowler <ot...@gmail.com> wrote:
>>>
>>> "The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are deployed
>>> if required. So the writes for HDFS are the unpacking the bundles ( with the unpacked jars being loaded into a classloader ).
>>>
>>> If the unbundled is the same as the bundle, there is no op. In this case, it is first run.
>>>
>>> So this is the parser bolt using the HDFS backed extension loader.”
>>>
>>>
>>> This is part of the parser side loading, also feeds into the stellar load from hdfs stuff.
>>>
>>> * metron has extensions
>>> * extensions are packaged with their configurations and their ‘bundle’ ( the nar part )
>>> * extension bundles are deployed to hdfs *lib directory
>>> * parts of the system that use extensions discover and load them, with class loaders setup for the bundle
>>> * part of that, the bundles are unpackaged into a ‘working’ area if required ( so if you drop a new version into the lib, the next app will unpack the new version )
>>> * thus, the bundle system needs to be able to write to the working areas in hdfs
>>> * ???
>>> * profit
>>>
>>>
>>>> On April 12, 2017 at 08:11:28, Simon Elliston Ball (simon@simonellistonball.com) wrote:
>>>>
>>>> I’m curious: Otto, what’s your use case for writing to HDFS from a parsers?
>>>>
>>>> Simon
>>>>
>>>> > On 12 Apr 2017, at 13:04, Justin Leet <ju...@gmail.com> wrote:
>>>> >
>>>> > Chown it to metron:hadoop and it'll work. Storm is in the hadoop group and
>>>> > with 775 will be able to write.
>>>> >
>>>> > On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dl...@gmail.com> wrote:
>>>> >
>>>> >> It's curious to me that you're writing directly from parsing, but I suspect
>>>> >> that your parsing topology is running as the storm user and it can't write
>>>> >> to those directories.
>>>> >>
>>>> >> -D...
>>>> >>
>>>> >> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ot...@gmail.com>
>>>> >> wrote:
>>>> >>
>>>> >>> The indexing dir is created:
>>>> >>>
>>>> >>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
>>>> >>> type="directory",
>>>> >>> action="create_on_execute",
>>>> >>> owner=self.__params.metron_user,
>>>> >>> group=self.__params.metron_group,
>>>> >>> mode=0775,
>>>> >>> )
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
>>>> >>> wrote:
>>>> >>>
>>>> >>>
>>>> >>> I am trying to write to HDFS from ParserBolt, but I’m getting the
>>>> >> following
>>>> >>> exception:
>>>> >>>
>>>> >>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
>>>> >>> user=storm, access=WRITE,
>>>> >>> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>>> >>> check(FSPermissionChecker.java:319)
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>>> >>> check(FSPermissionChecker.java:292)
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>>> >>> checkPermission(FSPermissionChecker.java:213)
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>>> >>> checkPermission(FSPermissionChecker.java:190)
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>>>> >>> checkPermission(FSDirectory.java:1827)
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>>>> >>> checkPermission(FSDirectory.java:1811)
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(
>>>> >>> FSDirectory.java:1794)
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
>>>> >>> FSDirMkdirOp.java:71)
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
>>>> >>> FSNamesystem.java:4011)
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
>>>> >>> mkdirs(NameNodeRpcServer.java:1102)
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
>>>> >>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
>>>> >>> deTranslatorPB.java:630)
>>>> >>> at
>>>> >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
>>>> >>> ClientNamenodeProtocol$2.callBlockingMethod(
>>>> >> ClientNamenodeProtocolProtos.
>>>> >>> java)
>>>> >>> at
>>>> >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(
>>>> >>> ProtobufRpcEngine.java:640)
>>>> >>>
>>>> >>>
>>>> >>> The HDFS directory is created as such:
>>>> >>>
>>>> >>> self.__params.HdfsResource(self.__params.hdfs_metron_
>>>> >>> apps_extensions_working,
>>>> >>> type="directory",
>>>> >>> action="create_on_execute",
>>>> >>> owner=self.__params.metron_user,
>>>> >>> mode=0775)
>>>> >>>
>>>> >>>
>>>> >>> As the hdfs write handlers I am logging in as such:
>>>> >>>
>>>> >>> HdfsSecurityUtil.login(stormConfig, fsConf);
>>>> >>> FileSystem fileSystem = FileSystem.get(fsConf);
>>>> >>>
>>>> >>> I am not sure what is different from the indexing hdfs writer setup here,
>>>> >>> but what I’m doing obviously is not working.
>>>> >>>
>>>> >>> Any ideas?
>>>> >>>
>>>> >>>
>>>> >>> - the branch:
>>>> >>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
>>>> >>>
>>>> >>> I am not up to date with master.
>>>> >>>
>>>> >>
>>
Re: [HELP!]Writing to HDFS from Storm
Posted by Otto Fowler <ot...@gmail.com>.
What Casey and I had talked about ( and he actually implemented first with
stellar ) is having a single shared repository, without having to tie the
deployment
to the machine.
This all still has to stand review so we’ll see.
Maybe Casey can comment.
On April 12, 2017 at 08:30:31, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
Makes sense, however, would it make sense to unpack to local file system,
which would then avoid further HDFS ops?
On 12 Apr 2017, at 13:24, Otto Fowler <ot...@gmail.com> wrote:
"The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are
deployed
if required. So the writes for HDFS are the unpacking the bundles ( with
the unpacked jars being loaded into a classloader ).
If the unbundled is the same as the bundle, there is no op. In this case,
it is first run.
So this is the parser bolt using the HDFS backed extension loader.”
This is part of the parser side loading, also feeds into the stellar load
from hdfs stuff.
* metron has extensions
* extensions are packaged with their configurations and their ‘bundle’ (
the nar part )
* extension bundles are deployed to hdfs *lib directory
* parts of the system that use extensions discover and load them, with
class loaders setup for the bundle
* part of that, the bundles are unpackaged into a ‘working’ area if
required ( so if you drop a new version into the lib, the next app will
unpack the new version )
* thus, the bundle system needs to be able to write to the working areas in
hdfs
* ???
* profit
On April 12, 2017 at 08:11:28, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
I’m curious: Otto, what’s your use case for writing to HDFS from a parsers?
Simon
> On 12 Apr 2017, at 13:04, Justin Leet <ju...@gmail.com> wrote:
>
> Chown it to metron:hadoop and it'll work. Storm is in the hadoop group and
> with 775 will be able to write.
>
> On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dl...@gmail.com> wrote:
>
>> It's curious to me that you're writing directly from parsing, but I
suspect
>> that your parsing topology is running as the storm user and it can't
write
>> to those directories.
>>
>> -D...
>>
>> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> The indexing dir is created:
>>>
>>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
>>> type="directory",
>>> action="create_on_execute",
>>> owner=self.__params.metron_user,
>>> group=self.__params.metron_group,
>>> mode=0775,
>>> )
>>>
>>>
>>>
>>>
>>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
>>> wrote:
>>>
>>>
>>> I am trying to write to HDFS from ParserBolt, but I’m getting the
>> following
>>> exception:
>>>
>>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
>>> user=storm, access=WRITE,
>>> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> check(FSPermissionChecker.java:319)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> check(FSPermissionChecker.java:292)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> checkPermission(FSPermissionChecker.java:213)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> checkPermission(FSPermissionChecker.java:190)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>>> checkPermission(FSDirectory.java:1827)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>>> checkPermission(FSDirectory.java:1811)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(
>>> FSDirectory.java:1794)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
>>> FSDirMkdirOp.java:71)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
>>> FSNamesystem.java:4011)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
>>> mkdirs(NameNodeRpcServer.java:1102)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
>>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
>>> deTranslatorPB.java:630)
>>> at
>>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
>>> ClientNamenodeProtocol$2.callBlockingMethod(
>> ClientNamenodeProtocolProtos.
>>> java)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(
>>> ProtobufRpcEngine.java:640)
>>>
>>>
>>> The HDFS directory is created as such:
>>>
>>> self.__params.HdfsResource(self.__params.hdfs_metron_
>>> apps_extensions_working,
>>> type="directory",
>>> action="create_on_execute",
>>> owner=self.__params.metron_user,
>>> mode=0775)
>>>
>>>
>>> As the hdfs write handlers I am logging in as such:
>>>
>>> HdfsSecurityUtil.login(stormConfig, fsConf);
>>> FileSystem fileSystem = FileSystem.get(fsConf);
>>>
>>> I am not sure what is different from the indexing hdfs writer setup
here,
>>> but what I’m doing obviously is not working.
>>>
>>> Any ideas?
>>>
>>>
>>> - the branch:
>>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
>>>
>>> I am not up to date with master.
>>>
>>
Re: [HELP!]Writing to HDFS from Storm
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Makes sense, however, would it make sense to unpack to local file system, which would then avoid further HDFS ops?
> On 12 Apr 2017, at 13:24, Otto Fowler <ot...@gmail.com> wrote:
>
> "The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are deployed
> if required. So the writes for HDFS are the unpacking the bundles ( with the unpacked jars being loaded into a classloader ).
>
> If the unbundled is the same as the bundle, there is no op. In this case, it is first run.
>
> So this is the parser bolt using the HDFS backed extension loader.”
>
>
> This is part of the parser side loading, also feeds into the stellar load from hdfs stuff.
>
> * metron has extensions
> * extensions are packaged with their configurations and their ‘bundle’ ( the nar part )
> * extension bundles are deployed to hdfs *lib directory
> * parts of the system that use extensions discover and load them, with class loaders setup for the bundle
> * part of that, the bundles are unpackaged into a ‘working’ area if required ( so if you drop a new version into the lib, the next app will unpack the new version )
> * thus, the bundle system needs to be able to write to the working areas in hdfs
> * ???
> * profit
>
>
> On April 12, 2017 at 08:11:28, Simon Elliston Ball (simon@simonellistonball.com <ma...@simonellistonball.com>) wrote:
>
>> I’m curious: Otto, what’s your use case for writing to HDFS from a parsers?
>>
>> Simon
>>
>> > On 12 Apr 2017, at 13:04, Justin Leet <justinjleet@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > Chown it to metron:hadoop and it'll work. Storm is in the hadoop group and
>> > with 775 will be able to write.
>> >
>> > On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dlyle65535@gmail.com <ma...@gmail.com>> wrote:
>> >
>> >> It's curious to me that you're writing directly from parsing, but I suspect
>> >> that your parsing topology is running as the storm user and it can't write
>> >> to those directories.
>> >>
>> >> -D...
>> >>
>> >> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ottobackwards@gmail.com <ma...@gmail.com>>
>> >> wrote:
>> >>
>> >>> The indexing dir is created:
>> >>>
>> >>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
>> >>> type="directory",
>> >>> action="create_on_execute",
>> >>> owner=self.__params.metron_user,
>> >>> group=self.__params.metron_group,
>> >>> mode=0775,
>> >>> )
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com <ma...@gmail.com>)
>> >>> wrote:
>> >>>
>> >>>
>> >>> I am trying to write to HDFS from ParserBolt, but I’m getting the
>> >> following
>> >>> exception:
>> >>>
>> >>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
>> >>> user=storm, access=WRITE,
>> >>> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
>> >>> at
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>> >>> check(FSPermissionChecker.java:319)
>> >>> at
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>> >>> check(FSPermissionChecker.java:292)
>> >>> at
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>> >>> checkPermission(FSPermissionChecker.java:213)
>> >>> at
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>> >>> checkPermission(FSPermissionChecker.java:190)
>> >>> at
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>> >>> checkPermission(FSDirectory.java:1827)
>> >>> at
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>> >>> checkPermission(FSDirectory.java:1811)
>> >>> at
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(
>> >>> FSDirectory.java:1794)
>> >>> at
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
>> >>> FSDirMkdirOp.java:71)
>> >>> at
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
>> >>> FSNamesystem.java:4011)
>> >>> at
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
>> >>> mkdirs(NameNodeRpcServer.java:1102)
>> >>> at
>> >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
>> >>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
>> >>> deTranslatorPB.java:630)
>> >>> at
>> >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
>> >>> ClientNamenodeProtocol$2.callBlockingMethod(
>> >> ClientNamenodeProtocolProtos.
>> >>> java)
>> >>> at
>> >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(
>> >>> ProtobufRpcEngine.java:640)
>> >>>
>> >>>
>> >>> The HDFS directory is created as such:
>> >>>
>> >>> self.__params.HdfsResource(self.__params.hdfs_metron_
>> >>> apps_extensions_working,
>> >>> type="directory",
>> >>> action="create_on_execute",
>> >>> owner=self.__params.metron_user,
>> >>> mode=0775)
>> >>>
>> >>>
>> >>> As the hdfs write handlers I am logging in as such:
>> >>>
>> >>> HdfsSecurityUtil.login(stormConfig, fsConf);
>> >>> FileSystem fileSystem = FileSystem.get(fsConf);
>> >>>
>> >>> I am not sure what is different from the indexing hdfs writer setup here,
>> >>> but what I’m doing obviously is not working.
>> >>>
>> >>> Any ideas?
>> >>>
>> >>>
>> >>> - the branch:
>> >>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy <https://github.com/ottobackwards/incubator-metron/tree/parser_deploy>
>> >>>
>> >>> I am not up to date with master.
>> >>>
>> >>
Re: [HELP!]Writing to HDFS from Storm
Posted by Otto Fowler <ot...@gmail.com>.
"The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are
deployed
if required. So the writes for HDFS are the unpacking the bundles ( with
the unpacked jars being loaded into a classloader ).
If the unbundled is the same as the bundle, there is no op. In this case,
it is first run.
So this is the parser bolt using the HDFS backed extension loader.”
This is part of the parser side loading, also feeds into the stellar load
from hdfs stuff.
* metron has extensions
* extensions are packaged with their configurations and their ‘bundle’ (
the nar part )
* extension bundles are deployed to hdfs *lib directory
* parts of the system that use extensions discover and load them, with
class loaders setup for the bundle
* part of that, the bundles are unpackaged into a ‘working’ area if
required ( so if you drop a new version into the lib, the next app will
unpack the new version )
* thus, the bundle system needs to be able to write to the working areas in
hdfs
* ???
* profit
On April 12, 2017 at 08:11:28, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
I’m curious: Otto, what’s your use case for writing to HDFS from a parsers?
Simon
> On 12 Apr 2017, at 13:04, Justin Leet <ju...@gmail.com> wrote:
>
> Chown it to metron:hadoop and it'll work. Storm is in the hadoop group
and
> with 775 will be able to write.
>
> On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dl...@gmail.com> wrote:
>
>> It's curious to me that you're writing directly from parsing, but I
suspect
>> that your parsing topology is running as the storm user and it can't
write
>> to those directories.
>>
>> -D...
>>
>> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> The indexing dir is created:
>>>
>>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
>>> type="directory",
>>> action="create_on_execute",
>>> owner=self.__params.metron_user,
>>> group=self.__params.metron_group,
>>> mode=0775,
>>> )
>>>
>>>
>>>
>>>
>>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
>>> wrote:
>>>
>>>
>>> I am trying to write to HDFS from ParserBolt, but I’m getting the
>> following
>>> exception:
>>>
>>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
>>> user=storm, access=WRITE,
>>> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> check(FSPermissionChecker.java:319)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> check(FSPermissionChecker.java:292)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> checkPermission(FSPermissionChecker.java:213)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> checkPermission(FSPermissionChecker.java:190)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>>> checkPermission(FSDirectory.java:1827)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>>> checkPermission(FSDirectory.java:1811)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(
>>> FSDirectory.java:1794)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
>>> FSDirMkdirOp.java:71)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
>>> FSNamesystem.java:4011)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
>>> mkdirs(NameNodeRpcServer.java:1102)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
>>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
>>> deTranslatorPB.java:630)
>>> at
>>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
>>> ClientNamenodeProtocol$2.callBlockingMethod(
>> ClientNamenodeProtocolProtos.
>>> java)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(
>>> ProtobufRpcEngine.java:640)
>>>
>>>
>>> The HDFS directory is created as such:
>>>
>>> self.__params.HdfsResource(self.__params.hdfs_metron_
>>> apps_extensions_working,
>>> type="directory",
>>> action="create_on_execute",
>>> owner=self.__params.metron_user,
>>> mode=0775)
>>>
>>>
>>> As the hdfs write handlers I am logging in as such:
>>>
>>> HdfsSecurityUtil.login(stormConfig, fsConf);
>>> FileSystem fileSystem = FileSystem.get(fsConf);
>>>
>>> I am not sure what is different from the indexing hdfs writer setup
here,
>>> but what I’m doing obviously is not working.
>>>
>>> Any ideas?
>>>
>>>
>>> - the branch:
>>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
>>>
>>> I am not up to date with master.
>>>
>>
Re: [HELP!]Writing to HDFS from Storm
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
I’m curious: Otto, what’s your use case for writing to HDFS from a parsers?
Simon
> On 12 Apr 2017, at 13:04, Justin Leet <ju...@gmail.com> wrote:
>
> Chown it to metron:hadoop and it'll work. Storm is in the hadoop group and
> with 775 will be able to write.
>
> On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dl...@gmail.com> wrote:
>
>> It's curious to me that you're writing directly from parsing, but I suspect
>> that your parsing topology is running as the storm user and it can't write
>> to those directories.
>>
>> -D...
>>
>> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> The indexing dir is created:
>>>
>>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
>>> type="directory",
>>> action="create_on_execute",
>>> owner=self.__params.metron_user,
>>> group=self.__params.metron_group,
>>> mode=0775,
>>> )
>>>
>>>
>>>
>>>
>>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
>>> wrote:
>>>
>>>
>>> I am trying to write to HDFS from ParserBolt, but I’m getting the
>> following
>>> exception:
>>>
>>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
>>> user=storm, access=WRITE,
>>> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> check(FSPermissionChecker.java:319)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> check(FSPermissionChecker.java:292)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> checkPermission(FSPermissionChecker.java:213)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
>>> checkPermission(FSPermissionChecker.java:190)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>>> checkPermission(FSDirectory.java:1827)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
>>> checkPermission(FSDirectory.java:1811)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(
>>> FSDirectory.java:1794)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
>>> FSDirMkdirOp.java:71)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
>>> FSNamesystem.java:4011)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
>>> mkdirs(NameNodeRpcServer.java:1102)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
>>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
>>> deTranslatorPB.java:630)
>>> at
>>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
>>> ClientNamenodeProtocol$2.callBlockingMethod(
>> ClientNamenodeProtocolProtos.
>>> java)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(
>>> ProtobufRpcEngine.java:640)
>>>
>>>
>>> The HDFS directory is created as such:
>>>
>>> self.__params.HdfsResource(self.__params.hdfs_metron_
>>> apps_extensions_working,
>>> type="directory",
>>> action="create_on_execute",
>>> owner=self.__params.metron_user,
>>> mode=0775)
>>>
>>>
>>> As the hdfs write handlers I am logging in as such:
>>>
>>> HdfsSecurityUtil.login(stormConfig, fsConf);
>>> FileSystem fileSystem = FileSystem.get(fsConf);
>>>
>>> I am not sure what is different from the indexing hdfs writer setup here,
>>> but what I’m doing obviously is not working.
>>>
>>> Any ideas?
>>>
>>>
>>> - the branch:
>>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
>>>
>>> I am not up to date with master.
>>>
>>
Re: [HELP!]Writing to HDFS from Storm
Posted by Otto Fowler <ot...@gmail.com>.
So group=hadoop?
How does the indexing bolt work? it is metron:metron
On April 12, 2017 at 08:04:47, Justin Leet (justinjleet@gmail.com) wrote:
Chown it to metron:hadoop and it'll work. Storm is in the hadoop group and
with 775 will be able to write.
On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dl...@gmail.com> wrote:
> It's curious to me that you're writing directly from parsing, but I
suspect
> that your parsing topology is running as the storm user and it can't
write
> to those directories.
>
> -D...
>
> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > The indexing dir is created:
> >
> > self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
> > type="directory",
> > action="create_on_execute",
> > owner=self.__params.metron_user,
> > group=self.__params.metron_group,
> > mode=0775,
> > )
> >
> >
> >
> >
> > On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
> > wrote:
> >
> >
> > I am trying to write to HDFS from ParserBolt, but I’m getting the
> following
> > exception:
> >
> > Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
> > user=storm, access=WRITE,
> > inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> > check(FSPermissionChecker.java:319)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> > check(FSPermissionChecker.java:292)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> > checkPermission(FSPermissionChecker.java:213)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> > checkPermission(FSPermissionChecker.java:190)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> > checkPermission(FSDirectory.java:1827)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> > checkPermission(FSDirectory.java:1811)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(
> > FSDirectory.java:1794)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
> > FSDirMkdirOp.java:71)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
> > FSNamesystem.java:4011)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> > mkdirs(NameNodeRpcServer.java:1102)
> > at
> > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
> > deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
> > deTranslatorPB.java:630)
> > at
> > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
> > ClientNamenodeProtocol$2.callBlockingMethod(
> ClientNamenodeProtocolProtos.
> > java)
> > at
> > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(
> > ProtobufRpcEngine.java:640)
> >
> >
> > The HDFS directory is created as such:
> >
> > self.__params.HdfsResource(self.__params.hdfs_metron_
> > apps_extensions_working,
> > type="directory",
> > action="create_on_execute",
> > owner=self.__params.metron_user,
> > mode=0775)
> >
> >
> > As the hdfs write handlers I am logging in as such:
> >
> > HdfsSecurityUtil.login(stormConfig, fsConf);
> > FileSystem fileSystem = FileSystem.get(fsConf);
> >
> > I am not sure what is different from the indexing hdfs writer setup
here,
> > but what I’m doing obviously is not working.
> >
> > Any ideas?
> >
> >
> > - the branch:
> > https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
> >
> > I am not up to date with master.
> >
>
Re: [HELP!]Writing to HDFS from Storm
Posted by Justin Leet <ju...@gmail.com>.
Chown it to metron:hadoop and it'll work. Storm is in the hadoop group and
with 775 will be able to write.
On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dl...@gmail.com> wrote:
> It's curious to me that you're writing directly from parsing, but I suspect
> that your parsing topology is running as the storm user and it can't write
> to those directories.
>
> -D...
>
> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > The indexing dir is created:
> >
> > self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
> > type="directory",
> > action="create_on_execute",
> > owner=self.__params.metron_user,
> > group=self.__params.metron_group,
> > mode=0775,
> > )
> >
> >
> >
> >
> > On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
> > wrote:
> >
> >
> > I am trying to write to HDFS from ParserBolt, but I’m getting the
> following
> > exception:
> >
> > Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
> > user=storm, access=WRITE,
> > inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> > check(FSPermissionChecker.java:319)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> > check(FSPermissionChecker.java:292)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> > checkPermission(FSPermissionChecker.java:213)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> > checkPermission(FSPermissionChecker.java:190)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> > checkPermission(FSDirectory.java:1827)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> > checkPermission(FSDirectory.java:1811)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(
> > FSDirectory.java:1794)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
> > FSDirMkdirOp.java:71)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
> > FSNamesystem.java:4011)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> > mkdirs(NameNodeRpcServer.java:1102)
> > at
> > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
> > deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
> > deTranslatorPB.java:630)
> > at
> > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
> > ClientNamenodeProtocol$2.callBlockingMethod(
> ClientNamenodeProtocolProtos.
> > java)
> > at
> > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(
> > ProtobufRpcEngine.java:640)
> >
> >
> > The HDFS directory is created as such:
> >
> > self.__params.HdfsResource(self.__params.hdfs_metron_
> > apps_extensions_working,
> > type="directory",
> > action="create_on_execute",
> > owner=self.__params.metron_user,
> > mode=0775)
> >
> >
> > As the hdfs write handlers I am logging in as such:
> >
> > HdfsSecurityUtil.login(stormConfig, fsConf);
> > FileSystem fileSystem = FileSystem.get(fsConf);
> >
> > I am not sure what is different from the indexing hdfs writer setup here,
> > but what I’m doing obviously is not working.
> >
> > Any ideas?
> >
> >
> > - the branch:
> > https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
> >
> > I am not up to date with master.
> >
>
Re: [HELP!]Writing to HDFS from Storm
Posted by Otto Fowler <ot...@gmail.com>.
The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are deployed
if required. So the writes for HDFS are the unpacking the bundles ( with the unpacked jars being loaded into a classloader ).
If the unbundled is the same as the bundle, there is no op. In this case, it is first run.
So this is the parser bolt using the HDFS backed extension loader.
On April 12, 2017 at 07:56:35, David Lyle (dlyle65535@gmail.com) wrote:
It's curious to me that you're writing directly from parsing, but I suspect
that your parsing topology is running as the storm user and it can't write
to those directories.
-D...
On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ot...@gmail.com>
wrote:
> The indexing dir is created:
>
> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
> type="directory",
> action="create_on_execute",
> owner=self.__params.metron_user,
> group=self.__params.metron_group,
> mode=0775,
> )
>
>
>
>
> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
>
> I am trying to write to HDFS from ParserBolt, but I’m getting the following
> exception:
>
> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
> user=storm, access=WRITE,
> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> check(FSPermissionChecker.java:319)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> check(FSPermissionChecker.java:292)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> checkPermission(FSPermissionChecker.java:213)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> checkPermission(FSPermissionChecker.java:190)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> checkPermission(FSDirectory.java:1827)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> checkPermission(FSDirectory.java:1811)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(
> FSDirectory.java:1794)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
> FSDirMkdirOp.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
> FSNamesystem.java:4011)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> mkdirs(NameNodeRpcServer.java:1102)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
> deTranslatorPB.java:630)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
> ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.
> java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(
> ProtobufRpcEngine.java:640)
>
>
> The HDFS directory is created as such:
>
> self.__params.HdfsResource(self.__params.hdfs_metron_
> apps_extensions_working,
> type="directory",
> action="create_on_execute",
> owner=self.__params.metron_user,
> mode=0775)
>
>
> As the hdfs write handlers I am logging in as such:
>
> HdfsSecurityUtil.login(stormConfig, fsConf);
> FileSystem fileSystem = FileSystem.get(fsConf);
>
> I am not sure what is different from the indexing hdfs writer setup here,
> but what I’m doing obviously is not working.
>
> Any ideas?
>
>
> - the branch:
> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
>
> I am not up to date with master.
>
Re: [HELP!]Writing to HDFS from Storm
Posted by David Lyle <dl...@gmail.com>.
It's curious to me that you're writing directly from parsing, but I suspect
that your parsing topology is running as the storm user and it can't write
to those directories.
-D...
On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ot...@gmail.com>
wrote:
> The indexing dir is created:
>
> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
> type="directory",
> action="create_on_execute",
> owner=self.__params.metron_user,
> group=self.__params.metron_group,
> mode=0775,
> )
>
>
>
>
> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
>
> I am trying to write to HDFS from ParserBolt, but I’m getting the following
> exception:
>
> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
> user=storm, access=WRITE,
> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> check(FSPermissionChecker.java:319)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> check(FSPermissionChecker.java:292)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> checkPermission(FSPermissionChecker.java:213)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> checkPermission(FSPermissionChecker.java:190)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> checkPermission(FSDirectory.java:1827)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> checkPermission(FSDirectory.java:1811)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(
> FSDirectory.java:1794)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
> FSDirMkdirOp.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
> FSNamesystem.java:4011)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> mkdirs(NameNodeRpcServer.java:1102)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
> deTranslatorPB.java:630)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
> ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.
> java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(
> ProtobufRpcEngine.java:640)
>
>
> The HDFS directory is created as such:
>
> self.__params.HdfsResource(self.__params.hdfs_metron_
> apps_extensions_working,
> type="directory",
> action="create_on_execute",
> owner=self.__params.metron_user,
> mode=0775)
>
>
> As the hdfs write handlers I am logging in as such:
>
> HdfsSecurityUtil.login(stormConfig, fsConf);
> FileSystem fileSystem = FileSystem.get(fsConf);
>
> I am not sure what is different from the indexing hdfs writer setup here,
> but what I’m doing obviously is not working.
>
> Any ideas?
>
>
> - the branch:
> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
>
> I am not up to date with master.
>
Re: [HELP!]Writing to HDFS from Storm
Posted by Otto Fowler <ot...@gmail.com>.
This is in FULL_DEV vagrant
On April 12, 2017 at 07:51:22, Otto Fowler (ottobackwards@gmail.com) wrote:
The indexing dir is created:
self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
group=self.__params.metron_group,
mode=0775,
)
On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com) wrote:
I am trying to write to HDFS from ParserBolt, but I’m getting the following
exception:
Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
user=storm, access=WRITE,
inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1827)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1811)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1794)
at
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4011)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1102)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:630)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
The HDFS directory is created as such:
self.__params.HdfsResource(self.__params.hdfs_metron_apps_extensions_working,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
mode=0775)
As the hdfs write handlers I am logging in as such:
HdfsSecurityUtil.login(stormConfig, fsConf);
FileSystem fileSystem = FileSystem.get(fsConf);
I am not sure what is different from the indexing hdfs writer setup here,
but what I’m doing obviously is not working.
Any ideas?
- the branch:
https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
I am not up to date with master.
Re: [HELP!]Writing to HDFS from Storm
Posted by Otto Fowler <ot...@gmail.com>.
The indexing dir is created:
self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
group=self.__params.metron_group,
mode=0775,
)
On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com) wrote:
I am trying to write to HDFS from ParserBolt, but I’m getting the following
exception:
Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
user=storm, access=WRITE,
inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1827)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1811)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1794)
at
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4011)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1102)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:630)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
The HDFS directory is created as such:
self.__params.HdfsResource(self.__params.hdfs_metron_apps_extensions_working,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
mode=0775)
As the hdfs write handlers I am logging in as such:
HdfsSecurityUtil.login(stormConfig, fsConf);
FileSystem fileSystem = FileSystem.get(fsConf);
I am not sure what is different from the indexing hdfs writer setup here,
but what I’m doing obviously is not working.
Any ideas?
- the branch:
https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
I am not up to date with master.