You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Munagala Ramanath <ra...@datatorrent.com> on 2016/11/15 17:16:56 UTC
Re: SortedWordCount example is not running
Couple of questions:
Could you outline the steps you followed to run this application ?
Do you have any custom code that is creating or modifying a Configuration
object ?
Can you try building and running some of the example programs at:
https://github.com/DataTorrent/examples/tree/master/tutorials
for instance *fileOutput* or *fileIO*; do they run into the same issue ?
Ram
On Tue, Nov 15, 2016 at 2:06 AM, prakashduttaru@hotmail.com <
prakashduttaru@hotmail.com> wrote:
> Hi Support team
>
> Not able to start SortedWordCount example because the apex console is
> showing failed everytime when using list-apps command after putting the apa
> file name with its part.
> My hadoop configuration is :
> It is single node cluster
> hadoop 2.2
> Java 1.7
> OS:CentOS
>
> The dt.log file is showing below error:
>
> rent.stram.ResourceRequestHandler: Found host null
> 2016-11-15 14:24:27,473 INFO com.datatorrent.stram.ResourceRequestHandler:
> Strict anti-affinity = [] for container with operators
> PTOperator[id=1,name=lineReader]
> 2016-11-15 14:24:27,473 INFO com.datatorrent.stram.ResourceRequestHandler:
> Found host null
> 2016-11-15 14:24:27,474 INFO com.datatorrent.stram.ResourceRequestHandler:
> Strict anti-affinity = [] for container with operators
> PTOperator[id=3,name=windowWordCount]
> 2016-11-15 14:24:27,474 INFO com.datatorrent.stram.ResourceRequestHandler:
> Found host null
> 2016-11-15 14:24:27,474 INFO com.datatorrent.stram.ResourceRequestHandler:
> Strict anti-affinity = [] for container with operators
> PTOperator[id=5,name=wcWriter]
> 2016-11-15 14:24:27,474 INFO com.datatorrent.stram.ResourceRequestHandler:
> Found host null
> 2016-11-15 14:24:27,474 INFO
> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers:
> [Capability[<memory:1024, vCores:1>]Priority[0], Capability[<memory:1024,
> vCores:1>]Priority[1], Capability[<memory:1024, vCores:1>]Priority[2],
> Capability[<memory:1024, vCores:1>]Priority[3], Capability[<memory:1024,
> vCores:1>]Priority[4]]
> 2016-11-15 14:24:27,474 INFO
> com.datatorrent.stram.StreamingAppMasterService: Requested container:
> Capability[<memory:1024, vCores:1>]Priority[0] on host: [null]
> 2016-11-15 14:24:27,486 INFO
> com.datatorrent.stram.StreamingAppMasterService: Requested container:
> Capability[<memory:1024, vCores:1>]Priority[1] on host: [null]
> 2016-11-15 14:24:27,486 INFO
> com.datatorrent.stram.StreamingAppMasterService: Requested container:
> Capability[<memory:1024, vCores:1>]Priority[2] on host: [null]
> 2016-11-15 14:24:27,486 INFO
> com.datatorrent.stram.StreamingAppMasterService: Requested container:
> Capability[<memory:1024, vCores:1>]Priority[3] on host: [null]
> 2016-11-15 14:24:27,486 INFO
> com.datatorrent.stram.StreamingAppMasterService: Requested container:
> Capability[<memory:1024, vCores:1>]Priority[4] on host: [null]
> 2016-11-15 14:24:28,538 INFO
> com.datatorrent.stram.StreamingAppMasterService: Got new container.,
> containerId=container_1479174466882_0005_01_000002,
> containerNode=localhost:43913, containerNodeURI=localhost:8042,
> containerResourceMemory1024, priority0
> 2016-11-15 14:24:28,555 INFO com.datatorrent.stram.
> LaunchContainerRunnable:
> Setting up container launch context for
> containerid=container_1479174466882_0005_01_000002
> 2016-11-15 14:24:28,565 INFO com.datatorrent.stram.
> LaunchContainerRunnable:
> CLASSPATH:
> ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/
> common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$
> HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/
> share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/
> yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:.
> 2016-11-15 14:24:28,950 ERROR com.datatorrent.stram.StreamingAppMaster:
> Exiting Application Master
> java.lang.IllegalArgumentException: Wrong FS:
> file:/root/datatorrent/apps/application_1479174466882_
> 0005/myapexapp-1.0-SNAPSHOT.jar,
> expected: hdfs://localhost:8020
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:642)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(
> DistributedFileSystem.java:181)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$
> 000(DistributedFileSystem.java:92)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.
> doCall(DistributedFileSystem.java:1106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.
> doCall(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
> FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(
> DistributedFileSystem.java:1102)
> at
> com.datatorrent.stram.LaunchContainerRunnable.addFilesToLocalResources(
> LaunchContainerRunnable.java:139)
> at
> com.datatorrent.stram.LaunchContainerRunnable.run(
> LaunchContainerRunnable.java:170)
> at
> com.datatorrent.stram.StreamingAppMasterService.execute(
> StreamingAppMasterService.java:917)
> at
> com.datatorrent.stram.StreamingAppMasterService.run(
> StreamingAppMasterService.java:652)
> at
> com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:104)
>
>
> Please suggest me the solutions.
>
> Thanks in advance.
> Prakash
>
>
>
> --
> View this message in context: http://apache-apex-users-list.
> 78494.x6.nabble.com/SortedWordCount-example-is-not-running-tp1126.html
> Sent from the Apache Apex Users list mailing list archive at Nabble.com.
>
Re: SortedWordCount example is not running
Posted by "prakashduttaru@hotmail.com" <pr...@hotmail.com>.
hduser@edureka-VirtualBox:/work/apex-core/engine/src/main/scripts$ sudo bash
apex
[sudo] password for hduser:
Warning: hadoop executable not found. Running standalone with java.
Apex CLI 3.5.0-SNAPSHOT 25.11.2016 @ 11:35:45 CST rev: a54e0b7 branch:
master
apex> list-apps
{"apps": [{
"startTime": "2016-11-25 19:47:15 +0800",
"id": 1,
"name": "SortedWordCount",
"state": "FAILED",
"trackingUrl":
"http:\/\/edureka-VirtualBox:8088\/cluster\/app\/application_1480074384820_0001",
"finalStatus": "FAILED",
"tags": []
}]}
0 active, total 1 applications.
apex> exit
hduser@edureka-VirtualBox:/work/apex-core/engine/src/main/scripts$ bash apex
Error: A JNI error has occurred, please check your installation and try
again
Exception in thread "main" java.lang.NoClassDefFoundError:
jline/console/ConsoleReader
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: jline.console.ConsoleReader
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/SortedWordCount-example-is-not-running-tp1126p1182.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: SortedWordCount example is not running
Posted by Munagala Ramanath <ra...@datatorrent.com>.
The .apa file content can be listed with:* jar tvf ...../target/*.apa*
Could you post the output of "*mvn dependency:tree*" ?
Ram
On Fri, Nov 25, 2016 at 4:15 AM, prakashduttaru@hotmail.com <
prakashduttaru@hotmail.com> wrote:
> The apa lists message is showing as below
>
> apex> launch /work/myapexapp/target/*.apa
> 1. MyFirstApplication
> 2. SortedWordCount
> Choose application: 2
> {"appId": "application_1480074384820_0001"}
> apex (application_1480074384820_0001) > list-apps
> {"apps": [{
> "startTime": "2016-11-25 19:47:15 +0800",
> "id": 1,
> "name": "SortedWordCount",
> "state": "FAILED",
> "trackingUrl":
> "http:\/\/edureka-VirtualBox:8088\/cluster\/app\/
> application_1480074384820_0001",
> "finalStatus": "FAILED",
> "tags": []
> }]}
> 0 active, total 1 applications.
> apex (application_1480074384820_0001) >
>
>
>
>
> --
> View this message in context: http://apache-apex-users-list.
> 78494.x6.nabble.com/SortedWordCount-example-is-
> not-running-tp1126p1181.html
> Sent from the Apache Apex Users list mailing list archive at Nabble.com.
>
Re: SortedWordCount example is not running
Posted by "prakashduttaru@hotmail.com" <pr...@hotmail.com>.
The apa lists message is showing as below
apex> launch /work/myapexapp/target/*.apa
1. MyFirstApplication
2. SortedWordCount
Choose application: 2
{"appId": "application_1480074384820_0001"}
apex (application_1480074384820_0001) > list-apps
{"apps": [{
"startTime": "2016-11-25 19:47:15 +0800",
"id": 1,
"name": "SortedWordCount",
"state": "FAILED",
"trackingUrl":
"http:\/\/edureka-VirtualBox:8088\/cluster\/app\/application_1480074384820_0001",
"finalStatus": "FAILED",
"tags": []
}]}
0 active, total 1 applications.
apex (application_1480074384820_0001) >
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/SortedWordCount-example-is-not-running-tp1126p1181.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: SortedWordCount example is not running
Posted by "prakashduttaru@hotmail.com" <pr...@hotmail.com>.
Hi
After reinstalling all the necessary tools on hadoop 2.6, when I am trying
to execute SortedWordCount application, the below issue is showing. All the
maven build is successful when building the process. Please suggest how to
solve this issue
Application application_1480074384820_0001 failed 2 times due to AM
Container for appattempt_1480074384820_0001_000002 exited with exitCode:
-1000
For more detailed output, check application tracking
page:http://edureka-VirtualBox:8088/proxy/application_1480074384820_0001/Then,
click on links to logs of each attempt.
Diagnostics: File
file:/root/datatorrent/apps/application_1480074384820_0001/mbassador-1.1.9.jar
does not exist
java.io.FileNotFoundException: File
file:/root/datatorrent/apps/application_1480074384820_0001/mbassador-1.1.9.jar
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:539)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:752)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:529)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Failing this attempt. Failing the application.
Let me know if you need any more details.
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/SortedWordCount-example-is-not-running-tp1126p1180.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: SortedWordCount example is not running
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Hi Prakash,
It seems that you have hadoop dependencies included into apa file. can
you exclude hadoop dependencies from app package and try again.
Take a look at following documentation for removing these dependencies
from application package.
http://docs.datatorrent.com/troubleshooting/#hadoop-dependencies-conflicts
Also make sure that your hadoop version is above 2.6. Apex may not
work with hadoop 2.2 after APEXCORE-536.
Regards,
-Tushar
On Wed, Nov 16, 2016 at 7:39 AM, prakashduttaru@hotmail.com
<pr...@hotmail.com> wrote:
> Hi Munagala
>
> I tried with fileIO program copied from examples. It is showing the below
> error when executing the apa file.
>
> Please find the below messages which is occurring when executing the apa
> file.
> =========================================================
> [edureka@localhost work]$ cd apex-core/
> [edureka@localhost apex-core]$ cd engine/
> engine-checkstyle-suppressions.xml src/
> pom.xml target/
> [edureka@localhost apex-core]$ cd engine/src/
> main/ test/
> [edureka@localhost apex-core]$ cd engine/src/main/
> appended-resources/ java/ resources/ scripts/
> [edureka@localhost apex-core]$ cd engine/src/main/scripts/
> [edureka@localhost scripts]$ ls
> apex target
> [edureka@localhost scripts]$ bash apex
> Apex CLI 3.5.0-SNAPSHOT 14.11.2016 @ 08:46:21 IST rev: 2c024cd branch:
> master
> apex> launch /home/edureka/work
> workspace work
> apex> launch /home/edureka/work/fileIO/target/*.apa
> 1. FileIO
> 2. ThroughputBasedFileIO
> Choose application: 1
> java.lang.NoSuchMethodError:
> org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext.setApplicationTags(Ljava/util/Set;)V
> at com.datatorrent.stram.StramClient.startApplication(StramClient.java:612)
> at
> com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:640)
> at
> com.datatorrent.stram.cli.ApexCli$LaunchCommand.execute(ApexCli.java:2065)
> at com.datatorrent.stram.cli.ApexCli.launchAppPackage(ApexCli.java:3464)
> at com.datatorrent.stram.cli.ApexCli.access$7400(ApexCli.java:153)
> at
> com.datatorrent.stram.cli.ApexCli$LaunchCommand.execute(ApexCli.java:1909)
> at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1468)
>
> Fatal error encountered
> [edureka@localhost scripts]$ javac -version
> javac 1.7.0_67
> [edureka@localhost scripts]$ java -version
> java version "1.7.0_67"
> Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
> Java HotSpot(TM) Client VM (build 24.65-b04, mixed mode)
> [edureka@localhost scripts]$ mvn --version
> Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19
> 19:21:28+0530)
> Maven home: /home/edureka/apache-maven-3.0.5
> Java version: 1.7.0_67, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/jdk1.7.0_67/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "2.6.32-431.29.2.el6.i686", arch: "i386", family:
> "unix"
> [edureka@localhost scripts]$
> =============================================================
>
> I can execute any other java project smoothly. I already have the input-dir
> & output-dir folder exists in the hdfs and also in local.
>
> Please suggest me the solution so that I can kick start with apex.
>
> Thanks a lot in advance.
>
> Regards,
> Prakash
>
>
>
> --
> View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/SortedWordCount-example-is-not-running-tp1126p1128.html
> Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: SortedWordCount example is not running
Posted by "prakashduttaru@hotmail.com" <pr...@hotmail.com>.
Hi Munagala
I tried with fileIO program copied from examples. It is showing the below
error when executing the apa file.
Please find the below messages which is occurring when executing the apa
file.
=========================================================
[edureka@localhost work]$ cd apex-core/
[edureka@localhost apex-core]$ cd engine/
engine-checkstyle-suppressions.xml src/
pom.xml target/
[edureka@localhost apex-core]$ cd engine/src/
main/ test/
[edureka@localhost apex-core]$ cd engine/src/main/
appended-resources/ java/ resources/ scripts/
[edureka@localhost apex-core]$ cd engine/src/main/scripts/
[edureka@localhost scripts]$ ls
apex target
[edureka@localhost scripts]$ bash apex
Apex CLI 3.5.0-SNAPSHOT 14.11.2016 @ 08:46:21 IST rev: 2c024cd branch:
master
apex> launch /home/edureka/work
workspace work
apex> launch /home/edureka/work/fileIO/target/*.apa
1. FileIO
2. ThroughputBasedFileIO
Choose application: 1
java.lang.NoSuchMethodError:
org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext.setApplicationTags(Ljava/util/Set;)V
at com.datatorrent.stram.StramClient.startApplication(StramClient.java:612)
at
com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:640)
at
com.datatorrent.stram.cli.ApexCli$LaunchCommand.execute(ApexCli.java:2065)
at com.datatorrent.stram.cli.ApexCli.launchAppPackage(ApexCli.java:3464)
at com.datatorrent.stram.cli.ApexCli.access$7400(ApexCli.java:153)
at
com.datatorrent.stram.cli.ApexCli$LaunchCommand.execute(ApexCli.java:1909)
at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1468)
Fatal error encountered
[edureka@localhost scripts]$ javac -version
javac 1.7.0_67
[edureka@localhost scripts]$ java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) Client VM (build 24.65-b04, mixed mode)
[edureka@localhost scripts]$ mvn --version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19
19:21:28+0530)
Maven home: /home/edureka/apache-maven-3.0.5
Java version: 1.7.0_67, vendor: Oracle Corporation
Java home: /usr/lib/jvm/jdk1.7.0_67/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.29.2.el6.i686", arch: "i386", family:
"unix"
[edureka@localhost scripts]$
=============================================================
I can execute any other java project smoothly. I already have the input-dir
& output-dir folder exists in the hdfs and also in local.
Please suggest me the solution so that I can kick start with apex.
Thanks a lot in advance.
Regards,
Prakash
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/SortedWordCount-example-is-not-running-tp1126p1128.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.