You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by hash-X <gi...@git.apache.org> on 2015/11/05 04:26:06 UTC
[GitHub] hadoop pull request: AltFileInputStream.java replace FileInputStre...
GitHub user hash-X opened a pull request:
https://github.com/apache/hadoop/pull/42
AltFileInputStream.java replace FileInputStream.java in apache/hadoop/HDFS
A brief description
Long Stop-The-World GC pauses due to Final Reference processing are observed.
So, Where are those Final Reference come from ?
1 : `Finalizer`
2 : `FileInputStream`
How to solve this problem ?
Here is the detailed description,and I give a solution on this.
https://issues.apache.org/jira/browse/HDFS-8562
FileInputStream have a method of finalize , and it can cause GC pause for a long time.In our test,G1 as our GC. So,in AltFileInputStream , no finalize. A new design for a inputstream use in windows and non-windows.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hash-X/hadoop AltFileInputStream
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hadoop/pull/42.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #42
----
commit 8d64ef0feb8c8d8f5d5823ccaa428a1b58f6fd04
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-19T09:50:19Z
Add some code.
commit 3ccf4c70c40cf1ba921d76b949317b5fd6752e3c
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-19T09:56:49Z
I cannot replace FileInputStream to NewFileInputStream in a casual way,cause the act
of change can damage other part of the HDFS.For example,When I test my code
using a Single Node (psedo-distributed) Cluster."Failed to load an FSImage
file." will happen when I start HDFS Daemons.At start,I replace
many FileInputStream which happend as an arg or constructor to
NewFileInputStream,but it seems like wrong.So,I have to do this in another
way.
commit 4da55130586ee9803a09162f7e2482b533aa12d9
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-19T10:30:11Z
Replace FIS to NFIS( NewFileInputStream ) is not recommend I
think,although there is a man named Alan Bateman from
https://bugs.openjdk.java.net/browse/JDK-8080225
suggest that.But test shows it is not good.Some problem may happen.And
these test consume so long time.Every time I change the source code,I need to
build the whole project (maybe it is not needed).But I install the new
version hadoop on my computer.So,build the whole project is needed.Maybe
should have a good way to do it I think.
commit 06b1509e0ad6dd74cf7c903e6ed6f2ec74d9b341
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-19T11:06:37Z
Replace FIS to NFIS,If test success,just do these first.It is not as
simple as that.
commit 2a79cd9c3b012556af7db5bdbf96663a1c30dcc4
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-20T02:36:55Z
Add a LOG info in DataXceiver for test.
commit 436c998ae21b3fe843b2d5ba6506e37ff2a34ab2
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-20T06:01:41Z
Rename NewFileInputStream to AltFileInputStream.
commit 14de2788ea2407c6ee252a69cfd3b4f6132c6faa
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-20T06:16:32Z
replace License header to Apache.
commit 387f7624a96716abef2062986f05523199e1927e
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-20T07:16:25Z
Remove open method in AltFileInputStream.java.
commit 52b029fac56bc054add1eac836e6cf71a0735304
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-20T10:14:09Z
Performance between AltFileInputStream and FileInputStream is not do
from this commit.Important question I think whether
AltFileInputStream could convert
to FileInputStream safely.I define a frame plan to do it.But I don't know is
this correct for the problem ? In HDFS code,compulsory conversion to
FileInputStream is happend everywhere.
commit e76d5eb4bf0145a4b28c581ecec07dcee7bae4e5
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-20T13:11:24Z
I think the compulsory conversion is safety,cause the AltFileInputSteam is
the subclass of the InputStream.In the previous version of the HDFS,the
convert to FileInputStream I see is safety cause these method return
InputStream which is the superclass of the FileInputStream.In my version
of HDFS,InputStream is also the superclass of the
AltFileInputStream.So,AltFileInputStream is also a InputStream just like
the FileInputStream is a InputStream too.So,I think it is
safety.Everyone agree ? If not,please give your opinion and tell me What's
wrong with that ? thank you.
commit 959e91c05c11cc1445513d36ec083707f0bba4e1
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-20T14:08:51Z
channel.close() is not needed.Because closeing the stream will in turn
cause the channel to be closed.
commit aa7f82efb29d6ff457dcf6e5b2a74af663682106
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-20T15:57:36Z
Delete HDFS-8562.patch
old patch.
commit 5e8e15cecb6d159706270dafd6e71dc9816abf19
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-21T02:56:34Z
Revice AltFileInputStream.java
commit 1d0e7fb8f9cca59480d4a79d62cdf005a804912f
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-22T10:32:45Z
Make a test.
commit 5daf8730dbc3e94fe766acf83b3ace39e60bc730
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-22T14:32:02Z
update AltFileInputStream.java
commit f58060ce118d6991824c9c9f622fe54776c2d32d
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-23T06:22:49Z
Test class of AltFileInputStream.java
commit 76673377a73624ff60322c0ae4cdd61701ea79fc
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-23T08:05:52Z
new test file.
commit fea1be8a7b21d15b0b0020c12c313eff108acd56
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-23T13:48:51Z
The latest AltFileInputStream.java
commit 3ad1b9b25f54e68d6816dfbc477a3c4221f445bd
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-23T14:03:03Z
The latest new file of AFIS
commit 8805c8bd87985d840eb534cad6c5e891b7f6ca41
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-24T00:59:44Z
Latest file of AltFileInputStream.java
commit 5cdabc49963652e7bb65aa00a448a5c06793b421
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-24T01:47:16Z
update
commit 221d27dbb908cc606a495cd298a7717ad65733aa
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-24T02:01:17Z
Soga ....
commit 69d147d98f83ee564df73ae7f6ac6d4bdcf3c816
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-24T03:02:24Z
a patch
commit a6de350f5240554015a54e8a07e399aaf6e89b61
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-24T04:39:21Z
Latest file.
commit 0860e5da144ea8bfe51d378b212921f365904497
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-24T04:50:38Z
update AltFileInputStream.
commit a566a304650cc4b8b3b3e5f700c564169bfe6d5b
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-24T08:22:27Z
Soga...
commit 02d53c9227ad124290042129cdd872b48c24dd78
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-27T08:21:10Z
update some code.
commit 090a4c8dd1eee4e01b791e623775b2074f9e6afb
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-27T09:06:32Z
update some code.
commit 63047775b3b528bf1e831f6c5baa2b8ccdba0f24
Author: zhangminglei <mi...@intel.com>
Date: 2015-07-27T09:19:45Z
update some code.
commit 6d3deaa377b85a3f24bd8d7e7a4d7682e8949ec1
Author: zhangminglei <mi...@intel.com>
Date: 2015-09-25T05:06:17Z
Add some info
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---