You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Denis Zhdanov (JIRA)" <ji...@apache.org> on 2009/09/06 18:39:57 UTC
[jira] Created: (IO-218) Introduce new filter input stream with
replacement facilities
Introduce new filter input stream with replacement facilities
-------------------------------------------------------------
Key: IO-218
URL: https://issues.apache.org/jira/browse/IO-218
Project: Commons IO
Issue Type: Improvement
Components: Filters
Affects Versions: 1.4
Environment: all environments
Reporter: Denis Zhdanov
Fix For: 2.0, 1.4
It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
For example we may want to configure the following replacements:
{noformat}
{1, } -> {7, 8}
{1} -> {9}
{3, 2} -> {}
{noformat}
and apply them to the input like
{noformat}
{4, 3, 2, 1, 2, 1, 3}
{noformat}
in order to get a result like
{noformat}
{4, 7, 8, 9, 3}
{noformat}
I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Yandell updated IO-218:
-----------------------------
Fix Version/s: (was: 1.4)
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753497#action_12753497 ]
haruhiko nishi commented on IO-218:
-----------------------------------
Do you think the search/replace needs to happen on the fly,without buffering up entire page?
ByteArrayInputStream is not really a stream processing, but it still is part of io package and named InputStream.
byte[] processing relatively quick and I guess you can have the search target buffered so long as the JVM does not thow OutoutMemory Error.
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753115#action_12753115 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 8:04 AM:
-----------------------------------------------------------
Hi I'm also working on a Stream that can handle pattern replacement.
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
was (Author: hanishi):
Hi I'm also working on a Stream that can handle pattern replacement.
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code}
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753434#action_12753434 ]
haruhiko nishi commented on IO-218:
-----------------------------------
I've tried ReplaceFilterInputStream and got the following and got Exception.
all I did was
ReplaceFilterInputStream in=new ReplaceFilterInputStream(new FileInputStream("test.html"),map);
where map is
map.put("yahoo".getBytes(),"google".getBytes());
then I get
Exception in thread "main" java.io.IOException: Push back buffer is full
at java.io.PushbackInputStream.unread(PushbackInputStream.java:215)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.replaceWithExpand(ReplaceFilterInputStream.java:467)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.tryToReplace(ReplaceFilterInputStream.java:405)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.doRead(ReplaceFilterInputStream.java:340)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.read(ReplaceFilterInputStream.java:239)
at TestMain.main(TestMain.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Niall Pemberton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Niall Pemberton updated IO-218:
-------------------------------
Fix Version/s: (was: 2.0)
2.1
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 2.1
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denis Zhdanov updated IO-218:
-----------------------------
Attachment: ReplaceFilterInputStreamTest.java
Unit test class
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753115#action_12753115 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 8:30 AM:
-----------------------------------------------------------
Hi I'm also working on a Stream that can handle pattern replacement. (It's more like byte[] wrapper rather than actual
streaming process. If you can figure out how to do this in real streaming. please help.)
usage: Integer.MAX_VALUE is max occurrence of the pattern that is provided.
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayReplaceInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
// pos is zero based position of this patern's occurrence in the byte[]
}
{code}
{code:title=StringPatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=SrcHrefReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
was (Author: hanishi):
Hi I'm also working on a Stream that can handle pattern replacement.
usage: Integer.MAX_VALUE is max occurrence of the pattern that is provided.
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayReplaceInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
// pos is zero based position of this patern's occurrence in the byte[]
}
{code}
{code:title=StringPatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=SrcHrefReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753115#action_12753115 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 8:02 AM:
-----------------------------------------------------------
Hi I'm also working on a Stream that can handle pattern replacement.
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code}
was (Author: hanishi):
Hi I'm also working on a Stream that can handle pattern replacement.
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753434#action_12753434 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 10:50 PM:
------------------------------------------------------------
I've tried ReplaceFilterInputStream and got the following Exception.
all I did was
public class TestMain {
public static void main(String[] args)throws Exception{
Map<byte[],byte[]> map=new HashMap<byte[],byte[]>();
map.put("yahoo".getBytes(),"google".getBytes());
ReplaceFilterInputStream in=new ReplaceFilterInputStream(new FileInputStream("test.html"),map);
int bytesRead;
FileOutputStream fout=new FileOutputStream("test_mod_1.html");
byte[] tmp=new byte[4096];
while((bytesRead=in.read(tmp,0,tmp.length))!=-1)
fout.write(tmp,0,bytesRead);
in.close();
fout.close();
}
}
then I get
Exception in thread "main" java.io.IOException: Push back buffer is full
at java.io.PushbackInputStream.unread(PushbackInputStream.java:215)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.replaceWithExpand(ReplaceFilterInputStream.java:467)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.tryToReplace(ReplaceFilterInputStream.java:405)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.doRead(ReplaceFilterInputStream.java:340)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.read(ReplaceFilterInputStream.java:239)
at TestMain.main(TestMain.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
was (Author: hanishi):
I've tried ReplaceFilterInputStream and got the following Exception.
all I did was
ReplaceFilterInputStream in=new ReplaceFilterInputStream(new FileInputStream("test.html"),map);
where map is
map.put("yahoo".getBytes(),"google".getBytes());
then I get
Exception in thread "main" java.io.IOException: Push back buffer is full
at java.io.PushbackInputStream.unread(PushbackInputStream.java:215)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.replaceWithExpand(ReplaceFilterInputStream.java:467)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.tryToReplace(ReplaceFilterInputStream.java:405)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.doRead(ReplaceFilterInputStream.java:340)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.read(ReplaceFilterInputStream.java:239)
at TestMain.main(TestMain.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denis Zhdanov updated IO-218:
-----------------------------
Attachment: ReplaceFilterInputStreamTest.java
ReplaceFilterInputStream.java
There was a bug in the case of long 'from' rule that is partially matched by the input data and 'slow' underlying stream.
It's fixed now and the test suit is populated.
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751926#action_12751926 ]
Denis Zhdanov edited comment on IO-218 at 9/6/09 9:40 AM:
----------------------------------------------------------
Implementation class is added
was (Author: denis.zhdanov):
Implementation class
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753532#action_12753532 ]
haruhiko nishi commented on IO-218:
-----------------------------------
When I started making my version of the search/replace InputStream, the one I posted above, I made it to subclass of FilterInputStream and implemented such as
{code}
public ByteArrayReplaceInputStream(InputStream in,PatternList list) throws IOException {
super(in);
if(in==null)
throw new IllegalArgumentException("Input stream may not be null");
this.itr=list.iterator();
parse();
}
private void parse() throws IOException {
byte[] srcbuf=new byte[1024];
byte[] tmpbuf=new byte[1024];
int size;
while((size=in.read(tmpbuf))!=-1){
int newcount=count+size;
if(newcount>srcbuf.length){
byte newbuf[]=new byte[Math.max(srcbuf.length<<1,newcount)];
System.arraycopy(srcbuf,0,newbuf,0,count);
srcbuf=newbuf;
}
System.arraycopy(tmpbuf,0,srcbuf,count,size);
count=newcount;
}
buf=new byte[count];
ByteBuffer byteBuffer=ByteBuffer.wrap(srcbuf,0,count);
count=0;
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
{code}
but then I though the creation of this buffer at parse() method is so redundant and I decided to change it to be like ByteArrayInputStream.
Since your method does not buffer up the target byte sequence, I was hoping to get some help from you.
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751927#action_12751927 ]
Denis Zhdanov edited comment on IO-218 at 9/6/09 9:41 AM:
----------------------------------------------------------
Unit test class is attached
was (Author: denis.zhdanov):
Unit test class
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753307#action_12753307 ]
haruhiko nishi commented on IO-218:
-----------------------------------
There is no problem at all.
It's just some other way to solve search replace stream. Mine just needs to buffer up everything before the parse occurs.
I was just wondering if you know how not to buffer everything and filter the stream in my code, I appreciate it very much.
I'll try using yours. Does yours support surround match?
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753434#action_12753434 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 10:51 PM:
------------------------------------------------------------
I've tried ReplaceFilterInputStream and got the following Exception.
all I did was
{code}
public class TestMain {
public static void main(String[] args)throws Exception{
Map<byte[],byte[]> map=new HashMap<byte[],byte[]>();
map.put("yahoo".getBytes(),"google".getBytes());
ReplaceFilterInputStream in=new ReplaceFilterInputStream(new FileInputStream("test.html"),map);
int bytesRead;
FileOutputStream fout=new FileOutputStream("test_mod_1.html");
byte[] tmp=new byte[4096];
while((bytesRead=in.read(tmp,0,tmp.length))!=-1)
fout.write(tmp,0,bytesRead);
in.close();
fout.close();
}
}
{code}
then I get
Exception in thread "main" java.io.IOException: Push back buffer is full
at java.io.PushbackInputStream.unread(PushbackInputStream.java:215)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.replaceWithExpand(ReplaceFilterInputStream.java:467)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.tryToReplace(ReplaceFilterInputStream.java:405)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.doRead(ReplaceFilterInputStream.java:340)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.read(ReplaceFilterInputStream.java:239)
at TestMain.main(TestMain.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
was (Author: hanishi):
I've tried ReplaceFilterInputStream and got the following Exception.
all I did was
public class TestMain {
public static void main(String[] args)throws Exception{
Map<byte[],byte[]> map=new HashMap<byte[],byte[]>();
map.put("yahoo".getBytes(),"google".getBytes());
ReplaceFilterInputStream in=new ReplaceFilterInputStream(new FileInputStream("test.html"),map);
int bytesRead;
FileOutputStream fout=new FileOutputStream("test_mod_1.html");
byte[] tmp=new byte[4096];
while((bytesRead=in.read(tmp,0,tmp.length))!=-1)
fout.write(tmp,0,bytesRead);
in.close();
fout.close();
}
}
then I get
Exception in thread "main" java.io.IOException: Push back buffer is full
at java.io.PushbackInputStream.unread(PushbackInputStream.java:215)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.replaceWithExpand(ReplaceFilterInputStream.java:467)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.tryToReplace(ReplaceFilterInputStream.java:405)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.doRead(ReplaceFilterInputStream.java:340)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.read(ReplaceFilterInputStream.java:239)
at TestMain.main(TestMain.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denis Zhdanov updated IO-218:
-----------------------------
Attachment: ReplaceFilterInputStream.java
Implementation class
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denis Zhdanov updated IO-218:
-----------------------------
Attachment: (was: ReplaceFilterInputStream.java)
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753115#action_12753115 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 8:21 AM:
-----------------------------------------------------------
Hi I'm also working on a Stream that can handle pattern replacement.
usage: Integer.MAX_VALUE is max occurrence of the pattern that is provided.
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayReplaceInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
// pos is zero based position of this patern's occurrence in the byte[]
}
{code}
{code:title=StringPatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=SrcHrefReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
was (Author: hanishi):
Hi I'm also working on a Stream that can handle pattern replacement.
usage:
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayReplaceInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=StringPatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=SrcHrefReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753115#action_12753115 ]
haruhiko nishi commented on IO-218:
-----------------------------------
Hi I'm also working on a Stream that can handle pattern replacement.
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denis Zhdanov updated IO-218:
-----------------------------
Attachment: (was: ReplaceFilterInputStreamTest.java)
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denis Zhdanov updated IO-218:
-----------------------------
Attachment: (was: ReplaceFilterInputStream.java)
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denis Zhdanov updated IO-218:
-----------------------------
Attachment: ReplaceFilterInputStreamTest.java
ReplaceFilterInputStream.java
Buffer overflow error is fixed. Test suit is expanded accordingly.
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753115#action_12753115 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 8:12 AM:
-----------------------------------------------------------
Hi I'm also working on a Stream that can handle pattern replacement.
usage:
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=StringPatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=SrcHrefReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
was (Author: hanishi):
Hi I'm also working on a Stream that can handle pattern replacement.
usage:
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=SrcHrefReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755906#action_12755906 ]
Denis Zhdanov commented on IO-218:
----------------------------------
New files are reattached.
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751926#action_12751926 ]
Denis Zhdanov edited comment on IO-218 at 9/6/09 9:41 AM:
----------------------------------------------------------
Implementation class is attached
was (Author: denis.zhdanov):
Implementation class is added
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753115#action_12753115 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 8:10 AM:
-----------------------------------------------------------
Hi I'm also working on a Stream that can handle pattern replacement.
usage:
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
was (Author: hanishi):
Hi I'm also working on a Stream that can handle pattern replacement.
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753454#action_12753454 ]
Denis Zhdanov commented on IO-218:
----------------------------------
Haruhiko,
Ok, I see now your question. I'll check your code if I have free time (not sure when that can be done however).
About the 'surround replacements' - no, my class doesn't support that because it's not possible to determine if such a replacement occurs without variable-sized buffering that, in turn, doesn't correlate with streaming processing.
About the exception - you probably found a bug that evaded from me. Can you send me your input file (_"test.html"_ from your example)?
Regards, Denis
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753115#action_12753115 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 8:15 AM:
-----------------------------------------------------------
Hi I'm also working on a Stream that can handle pattern replacement.
usage:
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayReplaceInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=StringPatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=SrcHrefReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
was (Author: hanishi):
Hi I'm also working on a Stream that can handle pattern replacement.
usage:
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=StringPatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=SrcHrefReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753484#action_12753484 ]
Denis Zhdanov commented on IO-218:
----------------------------------
Haruhiko,
Thanks for the example, I was able to reproduce the problem at the local environment. I'll investigate the problem and provide the fixed stream class and expanded unit test here.
Regards, Denis
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753458#action_12753458 ]
haruhiko nishi commented on IO-218:
-----------------------------------
I appreciate your comment.
About the exception the "test.html" is http://www.yahoo.com is top page copied into a file named test.html.
It happened intermittently, does not seem to happen always.
OOPs the Exception has some package name shown....
The reason we need to use such InputStream is that we need to do search/replace on entire page of web sites.
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753434#action_12753434 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 10:49 PM:
------------------------------------------------------------
I've tried ReplaceFilterInputStream and got the following Exception.
all I did was
ReplaceFilterInputStream in=new ReplaceFilterInputStream(new FileInputStream("test.html"),map);
where map is
map.put("yahoo".getBytes(),"google".getBytes());
then I get
Exception in thread "main" java.io.IOException: Push back buffer is full
at java.io.PushbackInputStream.unread(PushbackInputStream.java:215)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.replaceWithExpand(ReplaceFilterInputStream.java:467)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.tryToReplace(ReplaceFilterInputStream.java:405)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.doRead(ReplaceFilterInputStream.java:340)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.read(ReplaceFilterInputStream.java:239)
at TestMain.main(TestMain.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
was (Author: hanishi):
I've tried ReplaceFilterInputStream and got the following and got Exception.
all I did was
ReplaceFilterInputStream in=new ReplaceFilterInputStream(new FileInputStream("test.html"),map);
where map is
map.put("yahoo".getBytes(),"google".getBytes());
then I get
Exception in thread "main" java.io.IOException: Push back buffer is full
at java.io.PushbackInputStream.unread(PushbackInputStream.java:215)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.replaceWithExpand(ReplaceFilterInputStream.java:467)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.tryToReplace(ReplaceFilterInputStream.java:405)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.doRead(ReplaceFilterInputStream.java:340)
at jp.co.alibaba.util.template.reverse.groovy.ReplaceFilterInputStream.read(ReplaceFilterInputStream.java:239)
at TestMain.main(TestMain.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denis Zhdanov updated IO-218:
-----------------------------
Attachment: (was: ReplaceFilterInputStreamTest.java)
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753524#action_12753524 ]
Denis Zhdanov commented on IO-218:
----------------------------------
Haruhiko,
I believe that stream-based replacement and replacement on the static data are two different tasks from the user point of view.
However, you can see that if you have a stream-based replacement you can reuse it with the static data as well (at least my implementation) because java streams widely use _Decorator_ pattern and replacement stream works on any stream that _IS-A InputStream_. I.e. you can do the following to perform replacements against particular byte array:
# Create _ByteArrayInputStream_ for the target array;
# Create new _ReplaceFilterInputStream_ that wraps the _ByteArrayInputStream_ created before;
# Create new _ByteArrayOutputStream_;
# Filter the data from the _ByteArrayInputStream_ to the _ByteArrayOutputStream_;
The only drawback of such approach is that another byte array will be created, i.e. replacements don't occur in-place.
Regards, Denis.
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753169#action_12753169 ]
Denis Zhdanov commented on IO-218:
----------------------------------
> Hi I'm also working on a Stream that can handle pattern replacement. (It's more like byte[] wrapper rather than actual
> streaming process. If you can figure out how to do this in real streaming. please help.)
Haruhiko,
I'm fraid I don't understand the problem. The class I attached to this ticket (ReplaceFilterInputStream) already does the job, so you can just check the source and use it to perform replacements in streaming mode. Did you mean somthing else? May be some questions on its contract or implementation details?
Regards, Denis
P.S. it's better to attach the sources to the ticket instead of plcaing their content directly at the comments since its produces inconvenient comments reading.
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753115#action_12753115 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 8:11 AM:
-----------------------------------------------------------
Hi I'm also working on a Stream that can handle pattern replacement.
usage:
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=SrcHrefReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
was (Author: hanishi):
Hi I'm also working on a Stream that can handle pattern replacement.
usage:
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753115#action_12753115 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 8:14 AM:
-----------------------------------------------------------
Hi I'm also working on a Stream that can handle pattern replacement.
usage:
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=StringPatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=SrcHrefReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
was (Author: hanishi):
Hi I'm also working on a Stream that can handle pattern replacement.
usage:
PatternList list=new PatternList();
list.add("src=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF8"),Integer.MAX_VALUE);
list.add("href=\"".getBytes("UTF-8"),"\"".getBytes("UTF-8"),new SrcHrefReplacer("UTF-8"),Integer.MAX_VALUE);
ByteArrayReplaceInputStream in=new ByteArrayReplaceInputStream(someByteArray,list);
int bytesRead;
byte[] buf=new byte[4096];
while((bytesRead=in.read(buf,0,buf.length))!=-1)
out.write(buf,0,bytesRead);
in.close();
{code:title=ByteArrayInputStream.java|borderStyle=solid}
public class ByteArrayReplaceInputStream extends InputStream {
private byte[] buf;
private int count;
private PatternList.PatternListIterator itr;
private Map<PatternList.PatternEntry,Integer> counter=new IdentityHashMap<PatternList.PatternEntry,Integer>();
private int pos;
private int mark=0;
public ByteArrayReplaceInputStream(byte[] buf,PatternList list) {
this.itr=list.iterator();
this.buf=new byte[buf.length];
ByteBuffer byteBuffer=ByteBuffer.wrap(buf,0,buf.length);
itr.readLock();
try{
match(byteBuffer,itr.next(),0,byteBuffer.limit());
}finally{
itr.readUnlock();
}
}
private void write(ByteBuffer buffer){
byte[] byteArray=new byte[buffer.remaining()];
buffer.get(byteArray);
write(byteArray);
}
private void write(byte[] b){
int newcount=count+b.length;
if(newcount > buf.length){
byte newbuf[]=new byte[Math.max(buf.length<<1,newcount)];
System.arraycopy(buf,0,newbuf,0,count);
buf=newbuf;
}
System.arraycopy(b,0,buf,count,b.length);
count=newcount;
}
private final void match(ByteBuffer src,PatternList.PatternEntry pattern,int start,int end) {
if(pattern.isSingle())
single(src,pattern,start,end);
else
around(src,pattern,start,end);
}
private int count(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
count=counter.put(pattern,1);
else
count=counter.put(pattern,count+1);
return count==null ? 0 : count;
}
private boolean isPatternValid(PatternList.PatternEntry pattern){
Integer count=counter.get(pattern);
if(count==null)
return 0<=pattern.getMaxOccurence();
else
return count<pattern.getMaxOccurence();
}
private final void around(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int mark=0;
boolean flag=false;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
if(flag=!flag){
j=mark=cur+pattern.length();
pattern=pattern.swap();
}else{
src.position(pos).limit(mark);
j=cur+pattern.length();
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
pattern=pattern.swap();
src.position(mark).limit(cur);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
array=patternEntry.replace(count(patternEntry),array);
write(array);
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
match(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
private void single(ByteBuffer src,PatternList.PatternEntry patternEntry,int start,int end){
if(start<0 || start>end || end>src.limit())
throw new IndexOutOfBoundsException("start:"+start+" end:"+end);
int pos=start;
int limit_org=end;
int j=pos;
Pattern pattern=patternEntry.getPattern();
while(isPatternValid(patternEntry) && j<=limit_org-pattern.length()) {
boolean found=true;
int cur=j;
for(int i=0;i<pattern.length();i++){
if(src.get(cur+i)!=pattern.get(i)){
found=false;
break;
}
}
if(found){
src.position(pos).limit(cur);
j=cur+pattern.length();
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
src.position(cur).limit(j);
if(src.remaining()>0){
ByteBuffer target=src.slice();
int size=target.remaining();
byte[] array=new byte[size];
target.get(array);
write(patternEntry.replace(count(patternEntry),array));
}
pos=src.position(src.limit()).position();
src.limit(limit_org);
}
if(!found){
int k=cur+pattern.length();
if(k>=src.limit())
break;
j +=pattern.skip(src.get(k) & 0xff);
}
}
src.position(pos);
if(itr.hasNext())
single(src,itr.next(),pos,src.limit());
else
write(src);
itr.previous();
}
@Override
public synchronized int read(){
return (pos < count) ? (buf[pos++] & 0xff) : -1;
}
@Override
public synchronized int read(byte b[], int off, int len){
if (b == null)
throw new NullPointerException();
else if(off < 0 || len < 0 || len > b.length - off)
throw new IndexOutOfBoundsException();
if (pos >= count)
return -1;
if(pos + len > count)
len=count - pos;
if (len <= 0)
return 0;
System.arraycopy(buf, pos, b, off, len);
pos += len;
return len;
}
public synchronized long skip(long n){
if(pos +n>count){
n=count - pos;
}
if(n<0)
return 0;
pos +=n;
return n;
}
public synchronized int avaiable(){
return count - pos;
}
public boolean markSupported(){
return true;
}
public synchronized void mark(int readAheadLimit){
mark=pos;
}
public synchronized void reset(){
pos=mark;
}
public void close() throws IOException{
}
}
{code}
{code:title=Pattern.java|borderStyle=solid}
abstract class Pattern {
abstract int length();
abstract int get(int pos);
abstract int skip(int value);
abstract Pattern swap();
}
{code:title=PatternList.java|borderStyle=solid}
public class PatternList {
private ReentrantReadWriteLock lock=new ReentrantReadWriteLock();
private Lock readLock=lock.readLock();
private Lock writeLock=lock.writeLock();
private PatternEntry head=new PatternEntry(null,-1,null,null,null);
public PatternList(){
head.next=head.previous=head;
}
public void add(byte[] bBegin,byte[] bEnd,PatternReplacer handler, int maxOccurence){
add(new BeginPattern(new PatternTable(bBegin),new PatternTable(bEnd)),maxOccurence,handler);
}
public void add(byte[] p,PatternReplacer handler, int maxOccurence){
add(new SinglePattern(new PatternTable(p)),maxOccurence,handler);
}
PatternListIterator iterator(){
return new PatternListIterator(head.previous);
}
private void add(Pattern pattern,int maxOccurence,PatternReplacer handler){
writeLock.lock();
try{
for(PatternEntry tmp=head;;tmp=tmp.next)
if(tmp.pattern==null||tmp.compareTo(pattern)<=0){
PatternEntry newEntry=new PatternEntry(handler,maxOccurence,pattern,tmp,tmp.previous);
newEntry.previous.next=newEntry;
newEntry.next.previous=newEntry;
if(tmp==head)
head=newEntry;
break;
}
}finally{
writeLock.unlock();
}
}
class PatternListIterator{
private PatternEntry copyHead;
PatternListIterator(PatternEntry head){
this.copyHead=head;
}
public PatternEntry next(){
copyHead=copyHead.next;
return copyHead;
}
public PatternEntry previous(){
copyHead=copyHead.previous;
return copyHead;
}
public boolean hasNext(){
return copyHead.next.pattern!=null;
}
void readLock(){
readLock.lock();
}
void readUnlock(){
readLock.unlock();
}
}
static class PatternEntry implements Comparable<Pattern>{
Pattern pattern;
PatternEntry next;
PatternEntry previous;
PatternReplacer handler;
int maxOccurence;
private PatternEntry(PatternReplacer handler,int maxOccurence,Pattern element,PatternEntry next, PatternEntry previous){
this.handler=handler;
this.pattern=element;
this.next=next;
this.previous=previous;
this.maxOccurence=maxOccurence;
}
byte[] replace(int pos,byte[] match){
return handler.replace(pos,match);
}
int getMaxOccurence() {
return maxOccurence;
}
Pattern getPattern(){
return pattern;
}
public int compareTo(Pattern other) {
if(this.pattern instanceof SinglePattern && other instanceof BeginPattern)
return -1;
else if(this.pattern instanceof BeginPattern && other instanceof SinglePattern)
return 1;
else{
return ((ComparablePattern)this.pattern).size-((ComparablePattern)other).size;
}
}
boolean isSingle(){
return pattern instanceof SinglePattern;
}
}
private static abstract class ComparablePattern extends Pattern{
int size;
ComparablePattern(int size){
this.size=size;
}
}
private static class SinglePattern extends ComparablePattern {
PatternTable pattern;
private SinglePattern(PatternTable pattern){
super(pattern.length());
this.pattern=pattern;
}
@Override
int length() {
return pattern.length();
}
@Override
int get(int pos) {
return pattern.get(pos);
}
@Override
int skip(int value) {
return pattern.skip(value);
}
@Override
protected Pattern swap() {
throw new UnsupportedOperationException("single pattern cannot be swapped.");
}
}
private static class EndPattern extends Pattern {
PatternTable end;
Pattern nextMatcher;
private EndPattern(PatternTable end,Pattern nextMatcher){
this.end=end;
this.nextMatcher=nextMatcher;
}
@Override
int length() {
return end.length();
}
@Override
int get(int pos) {
return end.get(pos);
}
@Override
int skip(int value) {
return end.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class BeginPattern extends ComparablePattern{
PatternTable begin;
Pattern nextMatcher;
private BeginPattern(PatternTable begin,PatternTable end){
super(begin.length()+end.length());
this.begin=begin;
nextMatcher=new EndPattern(end,this);
}
@Override
int length() {
return begin.length();
}
@Override
int get(int pos) {
return begin.get(pos);
}
@Override
int skip(int value) {
return begin.skip(value);
}
@Override
protected Pattern swap() {
return nextMatcher;
}
}
private static class PatternTable {
private final int[] pattern;
private int[] skip;
PatternTable(byte[] target){
pattern=new int[target.length];
for(int i=0;i<target.length;i++)
pattern[i]=target[i] & 0xff;
this.skip=getSkipArray(target);
}
int length(){
return pattern.length;
}
int get(int i){
return pattern[i];
}
int skip(int i){
return skip[i];
}
private static int[] getSkipArray(byte[] pattern){
int[] skip=new int[256];
int i;
for(i=0; i<skip.length;i++)
skip[i]=pattern.length+1;
for(i=0; i<pattern.length;i++)
skip[pattern[i] & 0xff]=pattern.length -i;
return skip;
}
}
}
{code}
{code:title=PatternReplacer.java|borderStyle=solid}
public interface PatternReplacer {
byte[] replace(int pos,final byte[] matched);
}
{code}
{code:title=StringPatternReplacer.java|borderStyle=solid}
public abstract class StringPatternReplacer implements PatternReplacer{
private String charset;
protected StringPatternReplacer(String charset){
this.charset=charset;
}
public final byte[] replace(int pos, byte[] matched) {
String replaced;
try {
replaced = replace(pos,new String(matched,charset));
if(replaced!=null)
return replaced.getBytes(charset);
} catch (UnsupportedEncodingException e) {
}
return null;
}
protected abstract String replace(int pos,String matched);
}
{code}
{code:title=SrcHrefReplacer.java|borderStyle=solid}
public class SrcHrefReplacer extends StringPatternReplacer {
public SrcHrefReplacer(String charset){
super(charset);
}
public String replace(int pos, String matched) {
if(matched.endsWith(".jpg")||matched.endsWith(".gif"))
return matched;
if(matched.contains("somedomain.com"))
return matched;
return matched.replaceAll("somedomain.com","mydomain.com");
}
}
{code}
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IO-218) Introduce new filter input
stream with replacement facilities
Posted by "haruhiko nishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753307#action_12753307 ]
haruhiko nishi edited comment on IO-218 at 9/9/09 3:31 PM:
-----------------------------------------------------------
There is no problem at all.
It's just some other way to solve search replace stream. Mine just needs to buffer up everything before the parse occurs.
I was just wondering if you know how not to buffer everything and filter the stream in my code, I appreciate it very much.
I'll try using yours. Does yours support surround match?
P.S.
I'll just delete the source code.
was (Author: hanishi):
There is no problem at all.
It's just some other way to solve search replace stream. Mine just needs to buffer up everything before the parse occurs.
I was just wondering if you know how not to buffer everything and filter the stream in my code, I appreciate it very much.
I'll try using yours. Does yours support surround match?
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, } -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IO-218) Introduce new filter input stream with
replacement facilities
Posted by "Denis Zhdanov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denis Zhdanov updated IO-218:
-----------------------------
Description:
It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
For example we may want to configure the following replacements:
{noformat}
{1, 2} -> {7, 8}
{1} -> {9}
{3, 2} -> {}
{noformat}
and apply them to the input like
{noformat}
{4, 3, 2, 1, 2, 1, 3}
{noformat}
in order to get a result like
{noformat}
{4, 7, 8, 9, 3}
{noformat}
I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
was:
It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
For example we may want to configure the following replacements:
{noformat}
{1, } -> {7, 8}
{1} -> {9}
{3, 2} -> {}
{noformat}
and apply them to the input like
{noformat}
{4, 3, 2, 1, 2, 1, 3}
{noformat}
in order to get a result like
{noformat}
{4, 7, 8, 9, 3}
{noformat}
I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
> Introduce new filter input stream with replacement facilities
> -------------------------------------------------------------
>
> Key: IO-218
> URL: https://issues.apache.org/jira/browse/IO-218
> Project: Commons IO
> Issue Type: Improvement
> Components: Filters
> Affects Versions: 1.4
> Environment: all environments
> Reporter: Denis Zhdanov
> Fix For: 1.4, 2.0
>
> Attachments: ReplaceFilterInputStream.java, ReplaceFilterInputStreamTest.java
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> It seems convenient to have a FilterInputStream that allows to apply predefined repalcement rules against the read data.
> For example we may want to configure the following replacements:
> {noformat}
> {1, 2} -> {7, 8}
> {1} -> {9}
> {3, 2} -> {}
> {noformat}
> and apply them to the input like
> {noformat}
> {4, 3, 2, 1, 2, 1, 3}
> {noformat}
> in order to get a result like
> {noformat}
> {4, 7, 8, 9, 3}
> {noformat}
> I created the class that allows to do that and attached it to this ticket. Unit test class at junit4 format is attached as well.
> So, the task is to review the provided classes, consider if it's worth to add them to commons-io distribution and perform the inclusion in the case of possible result.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.