You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Ray Chang (JIRA)" <ji...@apache.org> on 2015/06/29 09:10:04 UTC
[jira] [Comment Edited] (FLUME-2431) Add simple regex
search-and-replace interceptor
[ https://issues.apache.org/jira/browse/FLUME-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605220#comment-14605220 ]
Ray Chang edited comment on FLUME-2431 at 6/29/15 7:09 AM:
-----------------------------------------------------------
Hello, Mark
I tried to replace "carriage return" with empty string in my spooldir source event body using search-and-replace interceptor to convert DOS newlines (CRLF) format to Linux format. So, I configured some relevant properties in flume.conf like below.
{noformat}
agent1.sources.src-1.interceptors = search-replace
agent1.sources.src-1.interceptors.search-replace.type = search_replace
agent1.sources.src-1.interceptors.search-replace.searchPattern = [\r]
agent1.sources.src-1.interceptors.search-replace.replaceString =
{noformat}
An Example configuration in flume user guide, it implies leave replaceString property blank does not matter, but I encounted NullpointerException like below.
{noformat}
java.lang.NullPointerException: Must supply a replacement string replaceString (empty is ok)
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.flume.interceptor.SearchAndReplaceInterceptor$Builder.configure(SearchAndReplaceInterceptor.java:108)
at org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:110)
at org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:80)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:348)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{noformat}
I think null checking for replaceString property in configure() function needs to be modified to use null pointer exception handling.
I attach a patch for your review. Please remind me if there is anything amiss(about review request process, etc).
Thanks
Ray
was (Author: ray chang):
Hello, Mark
I tried to replace "carriage return" with empty string in my spooldir source event body using search-and-replace interceptor to convert DOS newlines (CRLF) format to Linux format. So, I configured some relevant properties in flume.conf like below.
agent1.sources.src-1.interceptors = search-replace
agent1.sources.src-1.interceptors.search-replace.type = search_replace
agent1.sources.src-1.interceptors.search-replace.searchPattern = [\r]
agent1.sources.src-1.interceptors.search-replace.replaceString =
An Example configuration in flume user guide, it implies leave replaceString property blank does not matter, but I encounted NullpointerException like below.
java.lang.NullPointerException: Must supply a replacement string replaceString (empty is ok)
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.flume.interceptor.SearchAndReplaceInterceptor$Builder.configure(SearchAndReplaceInterceptor.java:108)
at org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:110)
at org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:80)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:348)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
I think null checking for replaceString property in configure() function needs to be modified to use null pointer exception handling.
I attach a patch for your review. Please remind me if there is anything amiss(about review request process, etc).
Thanks
Ray
> Add simple regex search-and-replace interceptor
> -----------------------------------------------
>
> Key: FLUME-2431
> URL: https://issues.apache.org/jira/browse/FLUME-2431
> Project: Flume
> Issue Type: New Feature
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.6.0
>
> Attachments: FLUME-2431-1.patch, FLUME-2431-2.patch, FLUME-2431-3.patch, FLUME-2431-4.patch
>
>
> It would be nice to have a simple interceptor that would allow for running regular expressions inline on the event body contents, assuming that event body is a string.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)