[jira] [Created] (YARN-11208) SampleContainerLogAggregationPolicy is inconsistent with expectations

["chuanjie.duan (Jira)" <ji...@apache.org>]

chuanjie.duan created YARN-11208:
------------------------------------

             Summary: SampleContainerLogAggregationPolicy is inconsistent with expectations
                 Key: YARN-11208
                 URL: https://issues.apache.org/jira/browse/YARN-11208
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: log-aggregation, nodemanager
    Affects Versions: 3.2.2
            Reporter: chuanjie.duan


repro steps:

Configure the following parameters in yarn-site.xml

  <property>
    <name>yarn.nodemanager.log-aggregation.policy.class</name>
 <value>org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.SampleContainerLogAggregationPolicy</value>
  </property>
  <property>
    <name>yarn.nodemanager.log-aggregation.policy.parameters</name>
    <value>SR:0.3,MIN:20</value>
  </property>

 

run pi:

hadoop  org.apache.hadoop.examples.QuasiMonteCarlo 100 5

 

three nodemanagers in my cluster

 

Expected quantity：20+(100-20)*0.3+1=45 container logs uploaded

Actual quantity: 21 container logs uploaded

 

So why are there fewer logs uploaded?

 

I print debug log in SampleContainerLogAggregationPolicy.

shouldDoLogAggregation()

 

logContext.getContainerId().hashCode():-1455756646

logContext.getContainerId().hashCode() % (1/sampleRate):-0.041277885

 

So in most cases, it will not be uploaded

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org