You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2018/11/13 23:56:00 UTC
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[ https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685913#comment-16685913 ]
Hadoop QA commented on YARN-9016:
---------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 1s{color} | {color:green} The patch appears to include 12 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 8s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m 26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 44s{color} | {color:orange} root: The patch generated 71 new + 0 unchanged - 0 fixed = 71 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 45s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 28s{color} | {color:red} hadoop-yarn-server-timelineservice-documentstore in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 23s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}146m 8s{color} | {color:red} hadoop-yarn-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 20s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 42s{color} | {color:green} hadoop-yarn-server-timelineservice-documentstore in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 42s{color} | {color:red} The patch generated 9 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}284m 32s{color} | {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.rmapp.TestApplicationLifetimeMonitor |
| | hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9016 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12948015/YARN-9016.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux f5045019849f 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 671fd65 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/22525/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/22525/artifact/out/whitespace-eol.txt |
| whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/22525/artifact/out/whitespace-tabs.txt |
| findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/22525/artifact/out/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore.txt |
| javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/22525/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt |
| javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/22525/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore.txt |
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/22525/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt |
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22525/testReport/ |
| asflicense | https://builds.apache.org/job/PreCommit-YARN-Build/22525/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 882 (vs. ulimit of 10000) |
| modules | C: hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore U: . |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22525/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> DocumentStore as a backend for ATSv2
> ------------------------------------
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
> Reporter: Sushil Ks
> Assignee: Sushil Ks
> Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in any Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, MongoDB etc by just adding new configurations properties and writing Document Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the store either if the document buffer gets full or periodically at every flush interval in background without adding any additional latency to the running jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> <!-- config required for ATSv2 to use DocumentStore-->
> <property>
> <name>yarn.timeline-service.writer.class&lt;/name>
> <value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl</value>
> </property>
> <property>
> <name>yarn.timeline-service.reader.class&lt;/name> <value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl</value>
> </property>
> <property>
> <name>yarn.timeline-service.documentstore.db-name</name>
> <value>YOUR_DATABASE_NAME</value> <!-- default is timeline_service -->
> </property>{code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the following command needs to be executed once for setting up the database and collections for storing documents.
> {code:java}
> hadoop org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional properties under *yarn-site.xml* is required..
> {code:java}
> <!-- config required for using Azure CosmosDB as a DocumentStore for ATSv2 -->
> <property>
> <name>yarn.timeline-service.store-type</name>
> <value>COSMOS_DB</value>
> </property>
> <property>
> <name>yarn.timeline-service.cosmos-db.endpoint</name>
> <value>http://YOUR_AZURE_COSMOS_DB_URL:443/</value>
> </property>
> <property>
> <name>yarn.timeline-service.cosmos-db.masterkey</name>
> <value>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL</value>
> </property>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore locally, install the emulator from [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and start it locally. Set the endpoint and master key under *yarn-site.xml* as mentioned above and run any example job like DistributedShell etc. Later you can check the data explorer UI of Azure CosmosDB locally to query the documents or even launch the *TimelineReader* locally and fetch/query the data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org