You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Rakesh Radhakrishnan (Jira)" <ji...@apache.org> on 2020/06/15 10:12:00 UTC
[jira] [Created] (HDDS-3801) Config being reloaded with hdfs as
'fs.defaultFS'
Rakesh Radhakrishnan created HDDS-3801:
------------------------------------------
Summary: Config being reloaded with hdfs as 'fs.defaultFS'
Key: HDDS-3801
URL: https://issues.apache.org/jira/browse/HDDS-3801
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Rakesh Radhakrishnan
Assignee: Rakesh Radhakrishnan
Noticed performance degradation while running {{SynteticLoadGenerator}} benchmark test with {{fs.defaultFS}}=HDFS *Vs* {{fs.defaultFS}}=O3FS.
{code:java}
Running LoadGenerator against fileSystem: hdfs://vb0929.halxg.cloudera.com:8020
*Vs*
Running LoadGenerator against fileSystem: o3fs://bucket2.vol2/
{code}
Command to run SyntheticLoadGenerator:-
{code:java}
yarn jar /opt/cloudera/parcels/CDH-7.2.0-1.cdh7.2.0.p0.3738720/jars/hadoop-mapreduce-client-jobclient-3.1.1.7.2.0.0-236-tests.jar NNloadGenerator -writeProbability 1 -readProbability 0.00 -elapsedTime 120 -numOfThreads 300 -root o3fs://bucket2.vol2/fsperf-Jun-11-2020/
{code}
With HDFS as 'fs.defaultFS', I could see that configuration is reloading resources always for each FileContext#create API call and is causing delay in submitting requests to OM server.
{code:java}
"Thread-304" #327 prio=5 os_prio=0 tid=0x00007f1609d42000 nid=0x2990b waiting for monitor entry [0x00007f15cb990000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:479)
- waiting to lock <0x00000000f01097e8> (a sun.misc.URLClassPath)
at sun.misc.URLClassPath.findResource(URLClassPath.java:224)
at java.net.URLClassLoader$2.run(URLClassLoader.java:572)
at java.net.URLClassLoader$2.run(URLClassLoader.java:570)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findResource(URLClassLoader.java:569)
at java.lang.ClassLoader.getResource(ClassLoader.java:1089)
at java.lang.ClassLoader.getResource(ClassLoader.java:1084)
at org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2803)
at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3059)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3018)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2991)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2871)
- locked <0x00000000f1da5f10> (a org.apache.hadoop.hdds.conf.OzoneConfiguration)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1223)
at org.apache.hadoop.ozone.OmUtils.isServiceIdsDefined(OmUtils.java:175)
at org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.<init>(BasicOzoneClientAdapterImpl.java:125)
at org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.<init>(OzoneClientAdapterImpl.java:51)
at org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:103)
at org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:167)
at org.apache.hadoop.fs.DelegateToFileSystem.<init>(DelegateToFileSystem.java:54)
at org.apache.hadoop.fs.ozone.OzFs.<init>(OzFs.java:41)
at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:142)
at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:180)
at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:265)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:341)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:338)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:338)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:330)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:85)
at org.apache.hadoop.fs.FileContext.create(FileContext.java:703)
at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.genFile(LoadGenerator.java:330)
at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.write(LoadGenerator.java:304)
at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.nextOp(LoadGenerator.java:271)
at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.run(LoadGenerator.java:236)
{code}
With O3FS as 'fs.defaultFS', there is no config#loading.
{code:java}
"Thread-400" #424 prio=5 os_prio=0 tid=0x00007f1988fe6800 nid=0x58441 in Object.wait() [0x00007f1942080000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1541)
- locked <0x00000000f60c3ab8> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1499)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy10.submitRequest(Unknown Source)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
- locked <0x00000000f60a1690> (a org.apache.hadoop.io.retry.RetryInvocationHandler$Call)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy10.submitRequest(Unknown Source)
at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:424)
at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createFile(OzoneManagerProtocolClientSideTranslatorPB.java:1632)
at org.apache.hadoop.ozone.client.rpc.RpcClient.createFile(RpcClient.java:1088)
at org.apache.hadoop.ozone.client.OzoneBucket.createFile(OzoneBucket.java:538)
at org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.createFile(BasicOzoneClientAdapterImpl.java:227)
at org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.createOutputStream(BasicOzoneFileSystem.java:270)
at org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.create(BasicOzoneFileSystem.java:250)
at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1263)
at org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:102)
at org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:615)
at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:701)
at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:697)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.create(FileContext.java:703)
at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.genFile(LoadGenerator.java:330)
at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.write(LoadGenerator.java:304)
at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.nextOp(LoadGenerator.java:271)
at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.run(LoadGenerator.java:236)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org