You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "McCullough, Alex" <Al...@capitalone.com> on 2016/04/03 16:28:57 UTC
Running Local Mode Test Application Connected to HDFS Cluster
Hello All,
I am trying to get my local test application to run but when I try I get an error. This test application is running on my local machine and attempting to connect to a remote HDFS cluster.
I have made sure I have the same version of hadoop that is on the cluster installed locally and with the same config. I can run local HDFS commands that connect just fine to cluster to read the same files my application should be reading/writing.
Thanks,
Alex
Error:
java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: hdfs
at com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:334)
at com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:71)
at com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:22)
at com.datatorrent.stram.engine.Node.setup(Node.java:182)
at com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1290)
at com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:129)
at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1363)
Caused by: java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2455)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:414)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:422)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:404)
at com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.getFSInstance(HdfsFileOutputOperator.java:90)
at com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:332)
... 6 more
Application Test Class:
@Category(IntegrationTest.class)
public class ApplicationTest {
@Test
public void testApplication() throws IOException, Exception {
try {
LocalMode lma = LocalMode.newInstance();
Configuration conf = new Configuration(false);
conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));
lma.prepareDAG(new Application(), conf);
LocalMode.Controller lc = lma.getController();
lc.run(200000); // runs for 10 seconds and quits
lc.shutdown();
} catch (ConstraintViolationException e) {
Assert.fail("constraint violations: " + e.getConstraintViolations());
}
}
}
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Running Local Mode Test Application Connected to HDFS Cluster
Posted by Ashwin Chandra Putta <as...@gmail.com>.
Alex,
Since you are trying to interact with remote fs you also need to provide
the remote fs endpoint details. You can do it in two ways.
1. include remote fs url in the file path eg:
hdfs://<host>:<port>/user/alex/output/
2. add remote fs to the config used by the operator. For this, you can
override the getFSInstance method to config the remote fs as follows. In
this case, you can give the filePath as you would normally give.
@Override
protected FileSystem getFSInstance() throws IOException
{
Configuration config = new Configuration();
if (remoteFs != null && !remoteFs.trim().isEmpty()) {
config.set(FileSystem.FS_DEFAULT_NAME_KEY, remoteFs);
}
FileSystem tempFS = FileSystem.newInstance(new Path(filePath).toUri(),
config);
if (tempFS instanceof LocalFileSystem) {
tempFS = ((LocalFileSystem)tempFS).getRaw();
}
return tempFS;
}
Where, remoteFs is the fs name of the remote server, you can find it as
fs.default.name in core-site.xml in remote hadoop.
Regards,
Ashwin.
On Sun, Apr 3, 2016 at 12:33 PM, Timothy Farkas <ti...@datatorrent.com> wrote:
> I suspect since the hdfs jar is marked provided it is not in the test
> classpath. So the hdfs files stem implementation is not found. If you mark
> it with the test scope it will be included. There may be other issues with
> the path as chandni
> On Apr 3, 2016 11:58 AM, "Chandni Singh" <ch...@datatorrent.com> wrote:
>
> > Since this is happening with the FileOutputOperator can you please let me
> > know what is the value of property 'filePath' in the configuration?
> >
> > Thanks,
> > Chandni
> >
> > On Sun, Apr 3, 2016 at 11:26 AM, Chandni Singh <ch...@datatorrent.com>
> > wrote:
> >
> > > Hi Alex,
> > >
> > > What is the input directory path that you have specified in the
> > > configuration?
> > > On Apr 3, 2016 7:57 AM, "McCullough, Alex" <
> > Alex.McCullough@capitalone.com>
> > > wrote:
> > >
> > >> I have the dependencies defined in the POM but get the same error when
> > >> running in the IDE.
> > >>
> > >> <dependency>
> > >> <groupId>org.apache.hadoop</groupId>
> > >> <artifactId>hadoop-hdfs</artifactId>
> > >> <version>2.7.2</version>
> > >> <scope>provided</scope>
> > >> </dependency>
> > >>
> > >> <dependency>
> > >> <groupId>org.apache.hadoop</groupId>
> > >> <artifactId>hadoop-client</artifactId>
> > >> <version>2.7.2</version>
> > >> <scope>provided</scope>
> > >> </dependency>
> > >>
> > >> <dependency>
> > >> <groupId>org.apache.hadoop</groupId>
> > >> <artifactId>hadoop-common</artifactId>
> > >> <version>2.7.2</version>
> > >> <scope>provided</scope>
> > >> </dependency>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On 4/3/16, 10:33 AM, "Thomas Weise" <th...@datatorrent.com> wrote:
> > >>
> > >> >Alex,
> > >> >
> > >> >Local mode in the IDE is fully embedded. It does not use any external
> > >> >Hadoop install. Please try launching the app through the CLI (in
> local
> > >> >mode) or you need to include the HDFS dependencies in your
> application
> > >> pom
> > >> >(with provided scope).
> > >> >
> > >> >Thanks
> > >> >
> > >> >--
> > >> >sent from mobile
> > >> >On Apr 3, 2016 7:29 AM, "McCullough, Alex" <
> > >> Alex.McCullough@capitalone.com>
> > >> >wrote:
> > >> >
> > >> >> Hello All,
> > >> >>
> > >> >> I am trying to get my local test application to run but when I try
> I
> > >> get
> > >> >> an error. This test application is running on my local machine and
> > >> >> attempting to connect to a remote HDFS cluster.
> > >> >>
> > >> >> I have made sure I have the same version of hadoop that is on the
> > >> cluster
> > >> >> installed locally and with the same config. I can run local HDFS
> > >> commands
> > >> >> that connect just fine to cluster to read the same files my
> > >> application
> > >> >> should be reading/writing.
> > >> >>
> > >> >> Thanks,
> > >> >> Alex
> > >> >>
> > >> >>
> > >> >>
> > >> >> Error:
> > >> >>
> > >> >> java.lang.RuntimeException: java.io.IOException: No FileSystem for
> > >> scheme:
> > >> >> hdfs
> > >> >> at
> > >> >>
> > >>
> >
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:334)
> > >> >> at
> > >> >>
> > >>
> >
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:71)
> > >> >> at
> > >> >>
> > >>
> >
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:22)
> > >> >> at com.datatorrent.stram.engine.Node.setup(Node.java:182)
> > >> >> at
> > >> >>
> > >>
> >
> com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1290)
> > >> >> at
> > >> >>
> > >>
> >
> com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:129)
> > >> >> at
> > >> >>
> > >>
> >
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1363)
> > >> >> Caused by: java.io.IOException: No FileSystem for scheme: hdfs
> > >> >> at
> > >>
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
> > >> >> at
> > >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
> > >> >> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
> > >> >> at
> > >>
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
> > >> >> at
> > >> org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2455)
> > >> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:414)
> > >> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:422)
> > >> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:404)
> > >> >> at
> > >> >>
> > >>
> >
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.getFSInstance(HdfsFileOutputOperator.java:90)
> > >> >> at
> > >> >>
> > >>
> >
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:332)
> > >> >> ... 6 more
> > >> >>
> > >> >>
> > >> >> Application Test Class:
> > >> >>
> > >> >>
> > >> >> @Category(IntegrationTest.class)
> > >> >> public class ApplicationTest {
> > >> >>
> > >> >> @Test
> > >> >> public void testApplication() throws IOException, Exception {
> > >> >> try {
> > >> >> LocalMode lma = LocalMode.newInstance();
> > >> >> Configuration conf = new Configuration(false);
> > >> >>
> > >> >>
> > >>
> >
> conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));
> > >> >> lma.prepareDAG(new Application(), conf);
> > >> >> LocalMode.Controller lc = lma.getController();
> > >> >> lc.run(200000); // runs for 10 seconds and quits
> > >> >> lc.shutdown();
> > >> >> } catch (ConstraintViolationException e) {
> > >> >> Assert.fail("constraint violations: " +
> > >> e.getConstraintViolations());
> > >> >> }
> > >> >> }
> > >> >> }
> > >> >> ________________________________________________________
> > >> >>
> > >> >> The information contained in this e-mail is confidential and/or
> > >> >> proprietary to Capital One and/or its affiliates and may only be
> used
> > >> >> solely in performance of work or services for Capital One. The
> > >> information
> > >> >> transmitted herewith is intended only for use by the individual or
> > >> entity
> > >> >> to which it is addressed. If the reader of this message is not the
> > >> intended
> > >> >> recipient, you are hereby notified that any review, retransmission,
> > >> >> dissemination, distribution, copying or other use of, or taking of
> > any
> > >> >> action in reliance upon this information is strictly prohibited. If
> > you
> > >> >> have received this communication in error, please contact the
> sender
> > >> and
> > >> >> delete the material from your computer.
> > >> >>
> > >> ________________________________________________________
> > >>
> > >> The information contained in this e-mail is confidential and/or
> > >> proprietary to Capital One and/or its affiliates and may only be used
> > >> solely in performance of work or services for Capital One. The
> > information
> > >> transmitted herewith is intended only for use by the individual or
> > entity
> > >> to which it is addressed. If the reader of this message is not the
> > intended
> > >> recipient, you are hereby notified that any review, retransmission,
> > >> dissemination, distribution, copying or other use of, or taking of any
> > >> action in reliance upon this information is strictly prohibited. If
> you
> > >> have received this communication in error, please contact the sender
> and
> > >> delete the material from your computer.
> > >>
> > >
> >
>
--
Regards,
Ashwin.
Re: Running Local Mode Test Application Connected to HDFS Cluster
Posted by Timothy Farkas <ti...@datatorrent.com>.
I suspect since the hdfs jar is marked provided it is not in the test
classpath. So the hdfs files stem implementation is not found. If you mark
it with the test scope it will be included. There may be other issues with
the path as chandni
On Apr 3, 2016 11:58 AM, "Chandni Singh" <ch...@datatorrent.com> wrote:
> Since this is happening with the FileOutputOperator can you please let me
> know what is the value of property 'filePath' in the configuration?
>
> Thanks,
> Chandni
>
> On Sun, Apr 3, 2016 at 11:26 AM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > Hi Alex,
> >
> > What is the input directory path that you have specified in the
> > configuration?
> > On Apr 3, 2016 7:57 AM, "McCullough, Alex" <
> Alex.McCullough@capitalone.com>
> > wrote:
> >
> >> I have the dependencies defined in the POM but get the same error when
> >> running in the IDE.
> >>
> >> <dependency>
> >> <groupId>org.apache.hadoop</groupId>
> >> <artifactId>hadoop-hdfs</artifactId>
> >> <version>2.7.2</version>
> >> <scope>provided</scope>
> >> </dependency>
> >>
> >> <dependency>
> >> <groupId>org.apache.hadoop</groupId>
> >> <artifactId>hadoop-client</artifactId>
> >> <version>2.7.2</version>
> >> <scope>provided</scope>
> >> </dependency>
> >>
> >> <dependency>
> >> <groupId>org.apache.hadoop</groupId>
> >> <artifactId>hadoop-common</artifactId>
> >> <version>2.7.2</version>
> >> <scope>provided</scope>
> >> </dependency>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 4/3/16, 10:33 AM, "Thomas Weise" <th...@datatorrent.com> wrote:
> >>
> >> >Alex,
> >> >
> >> >Local mode in the IDE is fully embedded. It does not use any external
> >> >Hadoop install. Please try launching the app through the CLI (in local
> >> >mode) or you need to include the HDFS dependencies in your application
> >> pom
> >> >(with provided scope).
> >> >
> >> >Thanks
> >> >
> >> >--
> >> >sent from mobile
> >> >On Apr 3, 2016 7:29 AM, "McCullough, Alex" <
> >> Alex.McCullough@capitalone.com>
> >> >wrote:
> >> >
> >> >> Hello All,
> >> >>
> >> >> I am trying to get my local test application to run but when I try I
> >> get
> >> >> an error. This test application is running on my local machine and
> >> >> attempting to connect to a remote HDFS cluster.
> >> >>
> >> >> I have made sure I have the same version of hadoop that is on the
> >> cluster
> >> >> installed locally and with the same config. I can run local HDFS
> >> commands
> >> >> that connect just fine to cluster to read the same files my
> >> application
> >> >> should be reading/writing.
> >> >>
> >> >> Thanks,
> >> >> Alex
> >> >>
> >> >>
> >> >>
> >> >> Error:
> >> >>
> >> >> java.lang.RuntimeException: java.io.IOException: No FileSystem for
> >> scheme:
> >> >> hdfs
> >> >> at
> >> >>
> >>
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:334)
> >> >> at
> >> >>
> >>
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:71)
> >> >> at
> >> >>
> >>
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:22)
> >> >> at com.datatorrent.stram.engine.Node.setup(Node.java:182)
> >> >> at
> >> >>
> >>
> com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1290)
> >> >> at
> >> >>
> >>
> com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:129)
> >> >> at
> >> >>
> >>
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1363)
> >> >> Caused by: java.io.IOException: No FileSystem for scheme: hdfs
> >> >> at
> >> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
> >> >> at
> >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
> >> >> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
> >> >> at
> >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
> >> >> at
> >> org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2455)
> >> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:414)
> >> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:422)
> >> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:404)
> >> >> at
> >> >>
> >>
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.getFSInstance(HdfsFileOutputOperator.java:90)
> >> >> at
> >> >>
> >>
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:332)
> >> >> ... 6 more
> >> >>
> >> >>
> >> >> Application Test Class:
> >> >>
> >> >>
> >> >> @Category(IntegrationTest.class)
> >> >> public class ApplicationTest {
> >> >>
> >> >> @Test
> >> >> public void testApplication() throws IOException, Exception {
> >> >> try {
> >> >> LocalMode lma = LocalMode.newInstance();
> >> >> Configuration conf = new Configuration(false);
> >> >>
> >> >>
> >>
> conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));
> >> >> lma.prepareDAG(new Application(), conf);
> >> >> LocalMode.Controller lc = lma.getController();
> >> >> lc.run(200000); // runs for 10 seconds and quits
> >> >> lc.shutdown();
> >> >> } catch (ConstraintViolationException e) {
> >> >> Assert.fail("constraint violations: " +
> >> e.getConstraintViolations());
> >> >> }
> >> >> }
> >> >> }
> >> >> ________________________________________________________
> >> >>
> >> >> The information contained in this e-mail is confidential and/or
> >> >> proprietary to Capital One and/or its affiliates and may only be used
> >> >> solely in performance of work or services for Capital One. The
> >> information
> >> >> transmitted herewith is intended only for use by the individual or
> >> entity
> >> >> to which it is addressed. If the reader of this message is not the
> >> intended
> >> >> recipient, you are hereby notified that any review, retransmission,
> >> >> dissemination, distribution, copying or other use of, or taking of
> any
> >> >> action in reliance upon this information is strictly prohibited. If
> you
> >> >> have received this communication in error, please contact the sender
> >> and
> >> >> delete the material from your computer.
> >> >>
> >> ________________________________________________________
> >>
> >> The information contained in this e-mail is confidential and/or
> >> proprietary to Capital One and/or its affiliates and may only be used
> >> solely in performance of work or services for Capital One. The
> information
> >> transmitted herewith is intended only for use by the individual or
> entity
> >> to which it is addressed. If the reader of this message is not the
> intended
> >> recipient, you are hereby notified that any review, retransmission,
> >> dissemination, distribution, copying or other use of, or taking of any
> >> action in reliance upon this information is strictly prohibited. If you
> >> have received this communication in error, please contact the sender and
> >> delete the material from your computer.
> >>
> >
>
Re: Running Local Mode Test Application Connected to HDFS Cluster
Posted by Chandni Singh <ch...@datatorrent.com>.
Since this is happening with the FileOutputOperator can you please let me
know what is the value of property 'filePath' in the configuration?
Thanks,
Chandni
On Sun, Apr 3, 2016 at 11:26 AM, Chandni Singh <ch...@datatorrent.com>
wrote:
> Hi Alex,
>
> What is the input directory path that you have specified in the
> configuration?
> On Apr 3, 2016 7:57 AM, "McCullough, Alex" <Al...@capitalone.com>
> wrote:
>
>> I have the dependencies defined in the POM but get the same error when
>> running in the IDE.
>>
>> <dependency>
>> <groupId>org.apache.hadoop</groupId>
>> <artifactId>hadoop-hdfs</artifactId>
>> <version>2.7.2</version>
>> <scope>provided</scope>
>> </dependency>
>>
>> <dependency>
>> <groupId>org.apache.hadoop</groupId>
>> <artifactId>hadoop-client</artifactId>
>> <version>2.7.2</version>
>> <scope>provided</scope>
>> </dependency>
>>
>> <dependency>
>> <groupId>org.apache.hadoop</groupId>
>> <artifactId>hadoop-common</artifactId>
>> <version>2.7.2</version>
>> <scope>provided</scope>
>> </dependency>
>>
>>
>>
>>
>>
>>
>>
>> On 4/3/16, 10:33 AM, "Thomas Weise" <th...@datatorrent.com> wrote:
>>
>> >Alex,
>> >
>> >Local mode in the IDE is fully embedded. It does not use any external
>> >Hadoop install. Please try launching the app through the CLI (in local
>> >mode) or you need to include the HDFS dependencies in your application
>> pom
>> >(with provided scope).
>> >
>> >Thanks
>> >
>> >--
>> >sent from mobile
>> >On Apr 3, 2016 7:29 AM, "McCullough, Alex" <
>> Alex.McCullough@capitalone.com>
>> >wrote:
>> >
>> >> Hello All,
>> >>
>> >> I am trying to get my local test application to run but when I try I
>> get
>> >> an error. This test application is running on my local machine and
>> >> attempting to connect to a remote HDFS cluster.
>> >>
>> >> I have made sure I have the same version of hadoop that is on the
>> cluster
>> >> installed locally and with the same config. I can run local HDFS
>> commands
>> >> that connect just fine to cluster to read the same files my
>> application
>> >> should be reading/writing.
>> >>
>> >> Thanks,
>> >> Alex
>> >>
>> >>
>> >>
>> >> Error:
>> >>
>> >> java.lang.RuntimeException: java.io.IOException: No FileSystem for
>> scheme:
>> >> hdfs
>> >> at
>> >>
>> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:334)
>> >> at
>> >>
>> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:71)
>> >> at
>> >>
>> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:22)
>> >> at com.datatorrent.stram.engine.Node.setup(Node.java:182)
>> >> at
>> >>
>> com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1290)
>> >> at
>> >>
>> com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:129)
>> >> at
>> >>
>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1363)
>> >> Caused by: java.io.IOException: No FileSystem for scheme: hdfs
>> >> at
>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
>> >> at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
>> >> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
>> >> at
>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
>> >> at
>> org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2455)
>> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:414)
>> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:422)
>> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:404)
>> >> at
>> >>
>> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.getFSInstance(HdfsFileOutputOperator.java:90)
>> >> at
>> >>
>> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:332)
>> >> ... 6 more
>> >>
>> >>
>> >> Application Test Class:
>> >>
>> >>
>> >> @Category(IntegrationTest.class)
>> >> public class ApplicationTest {
>> >>
>> >> @Test
>> >> public void testApplication() throws IOException, Exception {
>> >> try {
>> >> LocalMode lma = LocalMode.newInstance();
>> >> Configuration conf = new Configuration(false);
>> >>
>> >>
>> conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));
>> >> lma.prepareDAG(new Application(), conf);
>> >> LocalMode.Controller lc = lma.getController();
>> >> lc.run(200000); // runs for 10 seconds and quits
>> >> lc.shutdown();
>> >> } catch (ConstraintViolationException e) {
>> >> Assert.fail("constraint violations: " +
>> e.getConstraintViolations());
>> >> }
>> >> }
>> >> }
>> >> ________________________________________________________
>> >>
>> >> The information contained in this e-mail is confidential and/or
>> >> proprietary to Capital One and/or its affiliates and may only be used
>> >> solely in performance of work or services for Capital One. The
>> information
>> >> transmitted herewith is intended only for use by the individual or
>> entity
>> >> to which it is addressed. If the reader of this message is not the
>> intended
>> >> recipient, you are hereby notified that any review, retransmission,
>> >> dissemination, distribution, copying or other use of, or taking of any
>> >> action in reliance upon this information is strictly prohibited. If you
>> >> have received this communication in error, please contact the sender
>> and
>> >> delete the material from your computer.
>> >>
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>
Re: Running Local Mode Test Application Connected to HDFS Cluster
Posted by Chandni Singh <ch...@datatorrent.com>.
Hi Alex,
What is the input directory path that you have specified in the
configuration?
On Apr 3, 2016 7:57 AM, "McCullough, Alex" <Al...@capitalone.com>
wrote:
> I have the dependencies defined in the POM but get the same error when
> running in the IDE.
>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-hdfs</artifactId>
> <version>2.7.2</version>
> <scope>provided</scope>
> </dependency>
>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>2.7.2</version>
> <scope>provided</scope>
> </dependency>
>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-common</artifactId>
> <version>2.7.2</version>
> <scope>provided</scope>
> </dependency>
>
>
>
>
>
>
>
> On 4/3/16, 10:33 AM, "Thomas Weise" <th...@datatorrent.com> wrote:
>
> >Alex,
> >
> >Local mode in the IDE is fully embedded. It does not use any external
> >Hadoop install. Please try launching the app through the CLI (in local
> >mode) or you need to include the HDFS dependencies in your application pom
> >(with provided scope).
> >
> >Thanks
> >
> >--
> >sent from mobile
> >On Apr 3, 2016 7:29 AM, "McCullough, Alex" <
> Alex.McCullough@capitalone.com>
> >wrote:
> >
> >> Hello All,
> >>
> >> I am trying to get my local test application to run but when I try I get
> >> an error. This test application is running on my local machine and
> >> attempting to connect to a remote HDFS cluster.
> >>
> >> I have made sure I have the same version of hadoop that is on the
> cluster
> >> installed locally and with the same config. I can run local HDFS
> commands
> >> that connect just fine to cluster to read the same files my application
> >> should be reading/writing.
> >>
> >> Thanks,
> >> Alex
> >>
> >>
> >>
> >> Error:
> >>
> >> java.lang.RuntimeException: java.io.IOException: No FileSystem for
> scheme:
> >> hdfs
> >> at
> >>
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:334)
> >> at
> >>
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:71)
> >> at
> >>
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:22)
> >> at com.datatorrent.stram.engine.Node.setup(Node.java:182)
> >> at
> >>
> com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1290)
> >> at
> >>
> com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:129)
> >> at
> >>
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1363)
> >> Caused by: java.io.IOException: No FileSystem for scheme: hdfs
> >> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
> >> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
> >> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
> >> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
> >> at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2455)
> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:414)
> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:422)
> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:404)
> >> at
> >>
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.getFSInstance(HdfsFileOutputOperator.java:90)
> >> at
> >>
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:332)
> >> ... 6 more
> >>
> >>
> >> Application Test Class:
> >>
> >>
> >> @Category(IntegrationTest.class)
> >> public class ApplicationTest {
> >>
> >> @Test
> >> public void testApplication() throws IOException, Exception {
> >> try {
> >> LocalMode lma = LocalMode.newInstance();
> >> Configuration conf = new Configuration(false);
> >>
> >>
> conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));
> >> lma.prepareDAG(new Application(), conf);
> >> LocalMode.Controller lc = lma.getController();
> >> lc.run(200000); // runs for 10 seconds and quits
> >> lc.shutdown();
> >> } catch (ConstraintViolationException e) {
> >> Assert.fail("constraint violations: " +
> e.getConstraintViolations());
> >> }
> >> }
> >> }
> >> ________________________________________________________
> >>
> >> The information contained in this e-mail is confidential and/or
> >> proprietary to Capital One and/or its affiliates and may only be used
> >> solely in performance of work or services for Capital One. The
> information
> >> transmitted herewith is intended only for use by the individual or
> entity
> >> to which it is addressed. If the reader of this message is not the
> intended
> >> recipient, you are hereby notified that any review, retransmission,
> >> dissemination, distribution, copying or other use of, or taking of any
> >> action in reliance upon this information is strictly prohibited. If you
> >> have received this communication in error, please contact the sender and
> >> delete the material from your computer.
> >>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
Re: Running Local Mode Test Application Connected to HDFS Cluster
Posted by "McCullough, Alex" <Al...@capitalone.com>.
I have the dependencies defined in the POM but get the same error when running in the IDE.
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.2</version>
<scope>provided</scope>
</dependency>
On 4/3/16, 10:33 AM, "Thomas Weise" <th...@datatorrent.com> wrote:
>Alex,
>
>Local mode in the IDE is fully embedded. It does not use any external
>Hadoop install. Please try launching the app through the CLI (in local
>mode) or you need to include the HDFS dependencies in your application pom
>(with provided scope).
>
>Thanks
>
>--
>sent from mobile
>On Apr 3, 2016 7:29 AM, "McCullough, Alex" <Al...@capitalone.com>
>wrote:
>
>> Hello All,
>>
>> I am trying to get my local test application to run but when I try I get
>> an error. This test application is running on my local machine and
>> attempting to connect to a remote HDFS cluster.
>>
>> I have made sure I have the same version of hadoop that is on the cluster
>> installed locally and with the same config. I can run local HDFS commands
>> that connect just fine to cluster to read the same files my application
>> should be reading/writing.
>>
>> Thanks,
>> Alex
>>
>>
>>
>> Error:
>>
>> java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme:
>> hdfs
>> at
>> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:334)
>> at
>> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:71)
>> at
>> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:22)
>> at com.datatorrent.stram.engine.Node.setup(Node.java:182)
>> at
>> com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1290)
>> at
>> com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:129)
>> at
>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1363)
>> Caused by: java.io.IOException: No FileSystem for scheme: hdfs
>> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
>> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
>> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
>> at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2455)
>> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:414)
>> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:422)
>> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:404)
>> at
>> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.getFSInstance(HdfsFileOutputOperator.java:90)
>> at
>> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:332)
>> ... 6 more
>>
>>
>> Application Test Class:
>>
>>
>> @Category(IntegrationTest.class)
>> public class ApplicationTest {
>>
>> @Test
>> public void testApplication() throws IOException, Exception {
>> try {
>> LocalMode lma = LocalMode.newInstance();
>> Configuration conf = new Configuration(false);
>>
>> conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));
>> lma.prepareDAG(new Application(), conf);
>> LocalMode.Controller lc = lma.getController();
>> lc.run(200000); // runs for 10 seconds and quits
>> lc.shutdown();
>> } catch (ConstraintViolationException e) {
>> Assert.fail("constraint violations: " + e.getConstraintViolations());
>> }
>> }
>> }
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Running Local Mode Test Application Connected to HDFS Cluster
Posted by Thomas Weise <th...@datatorrent.com>.
Alex,
Local mode in the IDE is fully embedded. It does not use any external
Hadoop install. Please try launching the app through the CLI (in local
mode) or you need to include the HDFS dependencies in your application pom
(with provided scope).
Thanks
--
sent from mobile
On Apr 3, 2016 7:29 AM, "McCullough, Alex" <Al...@capitalone.com>
wrote:
> Hello All,
>
> I am trying to get my local test application to run but when I try I get
> an error. This test application is running on my local machine and
> attempting to connect to a remote HDFS cluster.
>
> I have made sure I have the same version of hadoop that is on the cluster
> installed locally and with the same config. I can run local HDFS commands
> that connect just fine to cluster to read the same files my application
> should be reading/writing.
>
> Thanks,
> Alex
>
>
>
> Error:
>
> java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme:
> hdfs
> at
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:334)
> at
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:71)
> at
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:22)
> at com.datatorrent.stram.engine.Node.setup(Node.java:182)
> at
> com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1290)
> at
> com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:129)
> at
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1363)
> Caused by: java.io.IOException: No FileSystem for scheme: hdfs
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
> at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2455)
> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:414)
> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:422)
> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:404)
> at
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.getFSInstance(HdfsFileOutputOperator.java:90)
> at
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:332)
> ... 6 more
>
>
> Application Test Class:
>
>
> @Category(IntegrationTest.class)
> public class ApplicationTest {
>
> @Test
> public void testApplication() throws IOException, Exception {
> try {
> LocalMode lma = LocalMode.newInstance();
> Configuration conf = new Configuration(false);
>
> conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));
> lma.prepareDAG(new Application(), conf);
> LocalMode.Controller lc = lma.getController();
> lc.run(200000); // runs for 10 seconds and quits
> lc.shutdown();
> } catch (ConstraintViolationException e) {
> Assert.fail("constraint violations: " + e.getConstraintViolations());
> }
> }
> }
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>