You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@eagle.apache.org by koone <gi...@git.apache.org> on 2016/12/02 10:14:11 UTC

[GitHub] incubator-eagle pull request #710: [EAGLE-820] add unit test for eagle-jpm-m...

GitHub user koone opened a pull request:

    https://github.com/apache/incubator-eagle/pull/710

    [EAGLE-820] add unit test for eagle-jpm-mr-history

    add unit test for eagle-jpm-mr-history
    https://issues.apache.org/jira/browse/EAGLE-820

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/koone/incubator-eagle EAGLE-775

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-eagle/pull/710.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #710
    
----
commit 2b7fc14393663f3fb08d6da00b64a9cfb8c16f16
Author: koone <lu...@126.com>
Date:   2016-11-28T04:55:48Z

    add unit test for eagle-service-base

commit 42880b056784fcb2a3c45097ed7eb6a39b95c372
Author: koone <lu...@126.com>
Date:   2016-12-02T10:09:15Z

    [EAGLE-820] add unit test for eagle-jpm-mr-history

commit 77f08ee322ff68c4edea4d89e76aa83df8f0f4b1
Author: koone <lu...@126.com>
Date:   2016-12-02T10:09:31Z

    Merge branch 'master' of https://github.com/apache/incubator-eagle into EAGLE-775

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle issue #710: [EAGLE-820] add unit test for eagle-jpm-mr-histo...

Posted by wujinhu <gi...@git.apache.org>.
Github user wujinhu commented on the issue:

    https://github.com/apache/incubator-eagle/pull/710
  
    LGTM, but pls remove code like 'e.printStackTrace();'


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #710: [EAGLE-820] add unit test for eagle-jpm-m...

Posted by wujinhu <gi...@git.apache.org>.
Github user wujinhu commented on a diff in the pull request:

    https://github.com/apache/incubator-eagle/pull/710#discussion_r90795468
  
    --- Diff: eagle-jpm/eagle-jpm-mr-history/src/test/java/org/apache/eagle/jpm/mr/history/storm/JobHistorySpoutTest.java ---
    @@ -0,0 +1,176 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + * <p/>
    + * http://www.apache.org/licenses/LICENSE-2.0
    + * <p/>
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.eagle.jpm.mr.history.storm;
    +
    +import backtype.storm.spout.ISpoutOutputCollector;
    +import backtype.storm.spout.SpoutOutputCollector;
    +import backtype.storm.task.TopologyContext;
    +import com.google.common.collect.Lists;
    +import com.typesafe.config.ConfigFactory;
    +import org.apache.commons.io.FileUtils;
    +import org.apache.curator.framework.CuratorFramework;
    +import org.apache.curator.framework.CuratorFrameworkFactory;
    +import org.apache.curator.framework.imps.CuratorFrameworkState;
    +import org.apache.curator.retry.ExponentialBackoffRetry;
    +import org.apache.curator.retry.RetryNTimes;
    +import org.apache.curator.test.TestingServer;
    +import org.apache.eagle.jpm.mr.history.MRHistoryJobConfig;
    +import org.apache.eagle.jpm.mr.history.crawler.JobHistoryContentFilter;
    +import org.apache.eagle.jpm.mr.history.crawler.JobHistoryContentFilterBuilder;
    +import org.apache.eagle.jpm.util.Constants;
    +import org.apache.eagle.jpm.util.HDFSUtil;
    +import org.apache.hadoop.conf.Configuration;
    +import org.junit.After;
    +import org.junit.Assert;
    +import org.junit.Before;
    +import org.junit.Test;
    +import org.junit.runner.RunWith;
    +import org.powermock.api.mockito.PowerMockito;
    +import org.powermock.core.classloader.annotations.PowerMockIgnore;
    +import org.powermock.core.classloader.annotations.PrepareForTest;
    +import org.powermock.modules.junit4.PowerMockRunner;
    +
    +import java.io.File;
    +import java.io.IOException;
    +import java.lang.reflect.Field;
    +import java.lang.reflect.Method;
    +import java.util.ArrayList;
    +import java.util.Date;
    +import java.util.List;
    +import java.util.regex.Pattern;
    +
    +import static org.mockito.Mockito.*;
    +
    +/**
    + * Created by luokun on 12/1/16.
    + */
    +@RunWith(PowerMockRunner.class)
    +@PrepareForTest({CuratorFrameworkFactory.class, HDFSUtil.class})
    +@PowerMockIgnore({"javax.*", "com.sun.org.*","org.apache.hadoop.conf.*"})
    +public class JobHistorySpoutTest {
    +
    +    private TestingServer server;
    +    private CuratorFramework zookeeper;
    +    private MRHistoryJobConfig appConfig;
    +
    +    @Before
    +    public void setUp() throws Exception {
    +        this.appConfig = MRHistoryJobConfig.newInstance(ConfigFactory.load());
    +        createZk();
    +    }
    +
    +    @After
    +    public void tearDown() throws Exception {
    +        try {
    +            if (zookeeper != null) {
    +                if (!zookeeper.getState().equals(CuratorFrameworkState.STOPPED)) {
    +                    zookeeper.close();
    +                }
    +            }
    +        } catch (Throwable e) {
    +            e.printStackTrace();
    +        } finally {
    +            try {
    +                if (server != null) {
    +                    server.close();
    +                }
    +            } catch (IOException e) {
    +                e.printStackTrace();
    +            }
    +        }
    +    }
    +
    +    @Test
    +    public void testSpout() throws Exception {
    --- End diff --
    
    Is this unit test meaningful? There is no assert about the test data


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #710: [EAGLE-820] add unit test for eagle-jpm-m...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-eagle/pull/710


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle issue #710: [EAGLE-820] add unit test for eagle-jpm-mr-histo...

Posted by koone <gi...@git.apache.org>.
Github user koone commented on the issue:

    https://github.com/apache/incubator-eagle/pull/710
  
    @wujinhu sorry , last week I was on a busines strip.  I have changed the test case with test data\uff0cplease check again


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #710: [EAGLE-820] add unit test for eagle-jpm-m...

Posted by wujinhu <gi...@git.apache.org>.
Github user wujinhu commented on a diff in the pull request:

    https://github.com/apache/incubator-eagle/pull/710#discussion_r92535638
  
    --- Diff: eagle-jpm/eagle-jpm-mr-history/src/test/resources/job_1479206441898_508949_conf.xml ---
    @@ -0,0 +1,1337 @@
    +<!--
    +   Licensed to the Apache Software Foundation (ASF) under one or more
    +   contributor license agreements.  See the NOTICE file distributed with
    +   this work for additional information regarding copyright ownership.
    +   The ASF licenses this file to You under the Apache License, Version 2.0
    +   (the "License"); you may not use this file except in compliance with
    +   the License.  You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +-->
    +<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
    +<property><name>hadoop.proxyuser.hive.groups</name><value>*</value><source>core-site.xml</source><source>job.xml</source></property>
    +<property><name>dfs.block.access.token.lifetime</name><value>600</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.rpc-address.nameservice2.namenode309</name><value>yhd-jqhadoop180.int.yihaodian.com:8020</value><source>hdfs-site.xml</source><source>job.xml</source></property>
    +<property><name>hive.skewjoin.key</name><value>100000</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>mapreduce.job.heap.memory-mb.ratio</name><value>0.8</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.rpc-address.nameservice2.namenode307</name><value>yhd-jqhadoop175.int.yihaodian.com:8020</value><source>hdfs-site.xml</source><source>job.xml</source></property>
    +<property><name>hive.index.compact.binary.search</name><value>true</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>mapreduce.map.log.level</name><value>INFO</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.lazypersist.file.scrub.interval.sec</name><value>300</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.admin.user.env</name><value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH</value><source>mapred-site.xml</source><source>job.xml</source></property>
    +<property><name>file.bytes-per-checksum</name><value>512</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.client.completion.pollinterval</name><value>5000</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage</name><value>false</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.log-aggregation-enable</name><value>false</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.edit.log.autoroll.check.interval.ms</name><value>300000</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.job.speculative.retry-after-speculate</name><value>15000</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>ipc.client.fallback-to-simple-auth-allowed</name><value>false</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.client.failover.connection.retries</name><value>0</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.jobtracker.system.dir</name><value>${hadoop.tmp.dir}/mapred/system</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.metastore.event.db.listener.timetolive</name><value>86400s</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>yarn.scheduler.minimum-allocation-mb</name><value>1024</value><source>yarn-site.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.task.profile.map.params</name><value>${mapreduce.task.profile.params}</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.map.memory.mb</name><value>2048</value><source>mapred-site.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.tasktracker.dns.interface</name><value>default</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.datanode.failed.volumes.tolerated</name><value>0</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.server2.authentication</name><value>NONE</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>_hive.tmp_table_space</name><value>/tmp/warehouse/bi_etl/9cbc3c63-c8f9-48cc-94a3-43ae6191ddf1/_tmp_space.db</value><source>programatically</source><source>job.xml</source></property>
    +<property><name>stream.stderr.reporter.prefix</name><value>reporter:</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>dfs.client.slow.io.warning.threshold.ms</name><value>30000</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>hadoop.security.groups.cache.secs</name><value>300</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.nodemanager.env-whitelist</name><value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.top.window.num.buckets</name><value>10</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.metastore.authorization.storage.checks</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>map.sort.class</name><value>org.apache.hadoop.util.QuickSort</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.safemode.threshold-pct</name><value>0.999f</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.jobtracker.jobhistory.task.numberprogresssplits</name><value>12</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>datanucleus.storeManagerType</name><value>rdbms</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>dfs.short.circuit.shared.memory.watcher.interrupt.check.ms</name><value>60000</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.metastore.aggregate.stats.cache.max.writer.wait</name><value>5000ms</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>fs.s3n.block.size</name><value>67108864</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.resourcemanager.client.thread-count</name><value>50</value><source>yarn-site.xml</source><source>job.xml</source></property>
    +<property><name>dfs.client.read.shortcircuit</name><value>false</value><source>hdfs-site.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.job.end-notification.max.retry.interval</name><value>5000</value><source>mapred-default.xml</source></property>
    +<property><name>hadoop.security.authentication</name><value>simple</value><source>core-site.xml</source><source>job.xml</source></property>
    +<property><name>dfs.client.mmap.retry.timeout.ms</name><value>300000</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.datanode.readahead.bytes</name><value>4193404</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.jobhistory.max-age-ms</name><value>604800000</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.app.mapreduce.client-am.ipc.max-retries</name><value>3</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.mapjoin.followby.map.aggr.hash.percentmemory</name><value>0.3</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.script.recordwriter</name><value>org.apache.hadoop.hive.ql.exec.TextRecordWriter</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.multigroupby.singlereducer</name><value>true</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>yarn.nodemanager.sleep-delay-before-sigkill.ms</name><value>250</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.vectorized.execution.reduce.groupby.enabled</name><value>true</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hadoop.shell.missing.defaultFs.warning</name><value>true</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>fs.trash.interval</name><value>1</value><source>core-site.xml</source><source>job.xml</source></property>
    +<property><name>dfs.datanode.max.locked.memory</name><value>0</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>hadoop.http.filter.initializers</name><value>org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer</value><source>programatically</source><source>job.xml</source></property>
    +<property><name>fs.har.impl</name><value>org.apache.hadoop.hive.shims.HiveHarFileSystem</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>dfs.datanode.ipc.address</name><value>0.0.0.0:50020</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.delegation.token.renew-interval</name><value>86400000</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.resourcemanager.webapp.address</name><value>${yarn.resourcemanager.hostname}:8088</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.spark.client.rpc.sasl.mechanisms</name><value>DIGEST-MD5</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.current.database</name><value>default</value><source>programatically</source><source>job.xml</source></property>
    +<property><name>hive.optimize.index.groupby</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>mapreduce.shuffle.max.connections</name><value>80</value><source>mapred-site.xml</source><source>job.xml</source></property>
    +<property><name>yarn.nodemanager.local-cache.max-files-per-directory</name><value>8192</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.replication.work.multiplier.per.iteration</name><value>2</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapred.mapper.new-api</name><value>false</value><source>programatically</source><source>job.xml</source></property>
    +<property><name>mapreduce.jobtracker.address</name><value>local</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.datanode.slow.io.warning.threshold.ms</name><value>300</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.client.max-cached-nodemanagers-proxies</name><value>0</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.job.working.dir</name><value>viewfs://cluster9/user/bi_etl</value><source>programatically</source><source>job.xml</source></property>
    +<property><name>mapreduce.tasktracker.instrumentation</name><value>org.apache.hadoop.mapred.TaskTrackerMetricsInst</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>hadoop.registry.zk.quorum</name><value>localhost:2181</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.app.mapreduce.am.job.committer.commit-window</name><value>10000</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>hadoop.registry.zk.session.timeout.ms</name><value>60000</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.tez.exec.print.summary</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>nfs.exports.allowed.hosts</name><value>* rw</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.metastore.filter.hook</name><value>org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.metastore.integral.jdo.pushdown</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hadoop.util.hash.type</name><value>murmur</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.replication.min</name><value>1</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.job.user.name</name><value>bi_etl</value><source>programatically</source></property>
    +<property><name>hive.cli.print.current.db</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.debug.localtask</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.security.authorization.enabled</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.warehouse.subdir.inherit.perms</name><value>true</value><source>file:/data/opt/hive1.2.1/apache-hive-1.2.1-bin/conf/hive-site.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.jobtracker.retiredjobs.cache.size</name><value>1000</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>s3.client-write-packet-size</name><value>65536</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.resourcemanager.scheduler.address</name><value>${yarn.resourcemanager.hostname}:8030</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.metastore.try.direct.sql</name><value>true</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>fs.s3.block.size</name><value>67108864</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>hadoop.security.group.mapping.ldap.search.filter.user</name><value>(&amp;(objectClass=user)(sAMAccountName={0}))</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.permissions.superusergroup</name><value>supergroup</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.resourcemanager.resource-tracker.address</name><value>${yarn.resourcemanager.hostname}:8031</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.resourcemanager.resource-tracker.address.rm405</name><value>yhd-jqhadoop168.int.yihaodian.com:8031</value><source>yarn-site.xml</source><source>job.xml</source></property>
    --- End diff --
    
    Pls remove sensitive information.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #710: [EAGLE-820] add unit test for eagle-jpm-m...

Posted by wujinhu <gi...@git.apache.org>.
Github user wujinhu commented on a diff in the pull request:

    https://github.com/apache/incubator-eagle/pull/710#discussion_r92535708
  
    --- Diff: eagle-jpm/eagle-jpm-mr-history/src/test/resources/job_1479206441898_508949_conf.xml ---
    @@ -0,0 +1,1337 @@
    +<!--
    +   Licensed to the Apache Software Foundation (ASF) under one or more
    +   contributor license agreements.  See the NOTICE file distributed with
    +   this work for additional information regarding copyright ownership.
    +   The ASF licenses this file to You under the Apache License, Version 2.0
    +   (the "License"); you may not use this file except in compliance with
    +   the License.  You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +-->
    +<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
    +<property><name>hadoop.proxyuser.hive.groups</name><value>*</value><source>core-site.xml</source><source>job.xml</source></property>
    +<property><name>dfs.block.access.token.lifetime</name><value>600</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.rpc-address.nameservice2.namenode309</name><value>yhd-jqhadoop180.int.yihaodian.com:8020</value><source>hdfs-site.xml</source><source>job.xml</source></property>
    +<property><name>hive.skewjoin.key</name><value>100000</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>mapreduce.job.heap.memory-mb.ratio</name><value>0.8</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.rpc-address.nameservice2.namenode307</name><value>yhd-jqhadoop175.int.yihaodian.com:8020</value><source>hdfs-site.xml</source><source>job.xml</source></property>
    +<property><name>hive.index.compact.binary.search</name><value>true</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>mapreduce.map.log.level</name><value>INFO</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.lazypersist.file.scrub.interval.sec</name><value>300</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.admin.user.env</name><value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH</value><source>mapred-site.xml</source><source>job.xml</source></property>
    +<property><name>file.bytes-per-checksum</name><value>512</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.client.completion.pollinterval</name><value>5000</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage</name><value>false</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.log-aggregation-enable</name><value>false</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.edit.log.autoroll.check.interval.ms</name><value>300000</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.job.speculative.retry-after-speculate</name><value>15000</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>ipc.client.fallback-to-simple-auth-allowed</name><value>false</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.client.failover.connection.retries</name><value>0</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.jobtracker.system.dir</name><value>${hadoop.tmp.dir}/mapred/system</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.metastore.event.db.listener.timetolive</name><value>86400s</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>yarn.scheduler.minimum-allocation-mb</name><value>1024</value><source>yarn-site.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.task.profile.map.params</name><value>${mapreduce.task.profile.params}</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.map.memory.mb</name><value>2048</value><source>mapred-site.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.tasktracker.dns.interface</name><value>default</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.datanode.failed.volumes.tolerated</name><value>0</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.server2.authentication</name><value>NONE</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>_hive.tmp_table_space</name><value>/tmp/warehouse/bi_etl/9cbc3c63-c8f9-48cc-94a3-43ae6191ddf1/_tmp_space.db</value><source>programatically</source><source>job.xml</source></property>
    +<property><name>stream.stderr.reporter.prefix</name><value>reporter:</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>dfs.client.slow.io.warning.threshold.ms</name><value>30000</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>hadoop.security.groups.cache.secs</name><value>300</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.nodemanager.env-whitelist</name><value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.top.window.num.buckets</name><value>10</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.metastore.authorization.storage.checks</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>map.sort.class</name><value>org.apache.hadoop.util.QuickSort</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.safemode.threshold-pct</name><value>0.999f</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.jobtracker.jobhistory.task.numberprogresssplits</name><value>12</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>datanucleus.storeManagerType</name><value>rdbms</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>dfs.short.circuit.shared.memory.watcher.interrupt.check.ms</name><value>60000</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.metastore.aggregate.stats.cache.max.writer.wait</name><value>5000ms</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>fs.s3n.block.size</name><value>67108864</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.resourcemanager.client.thread-count</name><value>50</value><source>yarn-site.xml</source><source>job.xml</source></property>
    +<property><name>dfs.client.read.shortcircuit</name><value>false</value><source>hdfs-site.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.job.end-notification.max.retry.interval</name><value>5000</value><source>mapred-default.xml</source></property>
    +<property><name>hadoop.security.authentication</name><value>simple</value><source>core-site.xml</source><source>job.xml</source></property>
    +<property><name>dfs.client.mmap.retry.timeout.ms</name><value>300000</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.datanode.readahead.bytes</name><value>4193404</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.jobhistory.max-age-ms</name><value>604800000</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.app.mapreduce.client-am.ipc.max-retries</name><value>3</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.mapjoin.followby.map.aggr.hash.percentmemory</name><value>0.3</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.script.recordwriter</name><value>org.apache.hadoop.hive.ql.exec.TextRecordWriter</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.multigroupby.singlereducer</name><value>true</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>yarn.nodemanager.sleep-delay-before-sigkill.ms</name><value>250</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.vectorized.execution.reduce.groupby.enabled</name><value>true</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hadoop.shell.missing.defaultFs.warning</name><value>true</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>fs.trash.interval</name><value>1</value><source>core-site.xml</source><source>job.xml</source></property>
    +<property><name>dfs.datanode.max.locked.memory</name><value>0</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>hadoop.http.filter.initializers</name><value>org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer</value><source>programatically</source><source>job.xml</source></property>
    +<property><name>fs.har.impl</name><value>org.apache.hadoop.hive.shims.HiveHarFileSystem</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>dfs.datanode.ipc.address</name><value>0.0.0.0:50020</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.delegation.token.renew-interval</name><value>86400000</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.resourcemanager.webapp.address</name><value>${yarn.resourcemanager.hostname}:8088</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.spark.client.rpc.sasl.mechanisms</name><value>DIGEST-MD5</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.current.database</name><value>default</value><source>programatically</source><source>job.xml</source></property>
    +<property><name>hive.optimize.index.groupby</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>mapreduce.shuffle.max.connections</name><value>80</value><source>mapred-site.xml</source><source>job.xml</source></property>
    +<property><name>yarn.nodemanager.local-cache.max-files-per-directory</name><value>8192</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.replication.work.multiplier.per.iteration</name><value>2</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapred.mapper.new-api</name><value>false</value><source>programatically</source><source>job.xml</source></property>
    +<property><name>mapreduce.jobtracker.address</name><value>local</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.datanode.slow.io.warning.threshold.ms</name><value>300</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.client.max-cached-nodemanagers-proxies</name><value>0</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.job.working.dir</name><value>viewfs://cluster9/user/bi_etl</value><source>programatically</source><source>job.xml</source></property>
    +<property><name>mapreduce.tasktracker.instrumentation</name><value>org.apache.hadoop.mapred.TaskTrackerMetricsInst</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>hadoop.registry.zk.quorum</name><value>localhost:2181</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.app.mapreduce.am.job.committer.commit-window</name><value>10000</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>hadoop.registry.zk.session.timeout.ms</name><value>60000</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.tez.exec.print.summary</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>nfs.exports.allowed.hosts</name><value>* rw</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.metastore.filter.hook</name><value>org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.metastore.integral.jdo.pushdown</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hadoop.util.hash.type</name><value>murmur</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.namenode.replication.min</name><value>1</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.job.user.name</name><value>bi_etl</value><source>programatically</source></property>
    +<property><name>hive.cli.print.current.db</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.debug.localtask</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.security.authorization.enabled</name><value>false</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>hive.warehouse.subdir.inherit.perms</name><value>true</value><source>file:/data/opt/hive1.2.1/apache-hive-1.2.1-bin/conf/hive-site.xml</source><source>job.xml</source></property>
    +<property><name>mapreduce.jobtracker.retiredjobs.cache.size</name><value>1000</value><source>mapred-default.xml</source><source>job.xml</source></property>
    +<property><name>s3.client-write-packet-size</name><value>65536</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.resourcemanager.scheduler.address</name><value>${yarn.resourcemanager.hostname}:8030</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>hive.metastore.try.direct.sql</name><value>true</value><source>programatically</source><source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@78dfcf46</source><source>job.xml</source></property>
    +<property><name>fs.s3.block.size</name><value>67108864</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>hadoop.security.group.mapping.ldap.search.filter.user</name><value>(&amp;(objectClass=user)(sAMAccountName={0}))</value><source>core-default.xml</source><source>job.xml</source></property>
    +<property><name>dfs.permissions.superusergroup</name><value>supergroup</value><source>hdfs-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.resourcemanager.resource-tracker.address</name><value>${yarn.resourcemanager.hostname}:8031</value><source>yarn-default.xml</source><source>job.xml</source></property>
    +<property><name>yarn.resourcemanager.resource-tracker.address.rm405</name><value>yhd-jqhadoop168.int.yihaodian.com:8031</value><source>yarn-site.xml</source><source>job.xml</source></property>
    --- End diff --
    
    Can we reduce the file content since our tests do not need so much test data?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---