You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Ryan Withers <wi...@gmail.com> on 2019/04/19 19:21:03 UTC
NIFI-1693 Maxmind Database Files and File Expansion with Expression
Language Enabled
Hello all,
The basis for the story / jira I've been working on is around properly
expanding the users home directory when the shortcut ~ is provided as a
value for any PropertyDescriptor pointing to a file. So I've been working
through all the processors and implementing this.
I have two questions as I'm nearing completion with this.
1. The class I'm looking at is TestGeoEnrichIP.java. It has a number of
test cases that look at the base directory of the project for a maxmind
database file. An example of this is:
testRunner.setProperty(GeoEnrichIP.GEO_DATABASE_FILE, "./");
I've looked through the project for *.mmdb files as described here:
https://github.com/maxmind/MaxMind-DB/tree/master/test-data, I didn't find
any references to files like that. I know I need to spend some additional
time with this but I wanted to ask the group to get some guidance and
additional understanding.
2. My second question is related to file expansion and expression
language. Specifically given the requirement to take ~/ and map it to
/home/rwithers/ should this same behavior be enabled if the property
descriptor has expression language enabled or only if expression language
is disabled. I've recently run into an example where expression language
is enabled on a property descriptor and it interfered with the path
expansion. So naturally I started wondering if path expansion should work
when expression language is also enabled.
Thanks in advance,
--
Ryan Withers
Senior Software Developer / Analyst
http://www.linkedin.com/in/ryanwithers
Re: NIFI-1693 Maxmind Database Files and File Expansion with
Expression Language Enabled
Posted by Ryan Withers <wi...@gmail.com>.
Bryan,
Thanks for your response.
1) Thank you for the feedback on the TestableGeoEnrichIP class. I see all
that in the @Before setup() method of the TestGeoEnrichIP now. I was stuck
looking at the setProperty() method and the value that was being set "./"
wondering why can't I find the file. Anyway this cleared it up for me.
2) So I've been thinking about how to generalize the solution even further
than I currently have it. Before I do that I'm going to finalize using my
current approach making sure the tests are all in place. Then as a
refactor and as you suggest I have been thinking about how to add to the
PropertyDescriptor to make it a bit easier to interrogate to determine if
the value represents a file. My hesitancy to put a boolean that has to be
set is that the developer has to remember to set it so you could end up
with a situation where it isn't set when it needs to be or it gets set when
it shouldn't be. Thanks again for the suggestions these have definitely
opened up some additional paths to go down.
Ryan
On Mon, Apr 22, 2019 at 9:19 AM Bryan Bende <bb...@gmail.com> wrote:
> Hello,
>
> 1) In that test, there is a class TestableGeoEnrichIP which is a
> subclass of the processor that overrides where the database reader
> comes from, and replaces it with a mock reader. So the property for
> the file location is not actually used for anything but does need to
> be set for validation to pass.
>
> 2) I haven't fully thought this through, or looked at any of the work
> being done, but in order to support this concept it seems like the
> framework would have to know that a given PropertyDescriptor
> represents a file or directory. I would think we need an option on the
> PropertyDescriptor builder like ".identifiesFileSystemResource(true)"
> then whenever a processor calls context.getProperty(foo).getValue() it
> would know if it needs to do the expansion. If expression language is
> involved, and a processor calls
> context.getProperty(foo).evaluateAttributeExpressions().getValue()
> then the framework would need to do the expansion after evaluating the
> expressions.
>
> -Bryan
>
> On Fri, Apr 19, 2019 at 3:21 PM Ryan Withers <wi...@gmail.com>
> wrote:
> >
> > Hello all,
> >
> > The basis for the story / jira I've been working on is around properly
> > expanding the users home directory when the shortcut ~ is provided as a
> > value for any PropertyDescriptor pointing to a file. So I've been
> working
> > through all the processors and implementing this.
> >
> > I have two questions as I'm nearing completion with this.
> >
> > 1. The class I'm looking at is TestGeoEnrichIP.java. It has a number of
> > test cases that look at the base directory of the project for a maxmind
> > database file. An example of this is:
> >
> > testRunner.setProperty(GeoEnrichIP.GEO_DATABASE_FILE, "./");
> >
> > I've looked through the project for *.mmdb files as described here:
> > https://github.com/maxmind/MaxMind-DB/tree/master/test-data, I didn't
> find
> > any references to files like that. I know I need to spend some
> additional
> > time with this but I wanted to ask the group to get some guidance and
> > additional understanding.
> >
> > 2. My second question is related to file expansion and expression
> > language. Specifically given the requirement to take ~/ and map it to
> > /home/rwithers/ should this same behavior be enabled if the property
> > descriptor has expression language enabled or only if expression language
> > is disabled. I've recently run into an example where expression language
> > is enabled on a property descriptor and it interfered with the path
> > expansion. So naturally I started wondering if path expansion should
> work
> > when expression language is also enabled.
> >
> > Thanks in advance,
> >
> > --
> > Ryan Withers
> > Senior Software Developer / Analyst
> >
> > http://www.linkedin.com/in/ryanwithers
>
--
Ryan Withers
Senior Software Developer / Analyst
http://www.linkedin.com/in/ryanwithers
Re: NIFI-1693 Maxmind Database Files and File Expansion with
Expression Language Enabled
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
1) In that test, there is a class TestableGeoEnrichIP which is a
subclass of the processor that overrides where the database reader
comes from, and replaces it with a mock reader. So the property for
the file location is not actually used for anything but does need to
be set for validation to pass.
2) I haven't fully thought this through, or looked at any of the work
being done, but in order to support this concept it seems like the
framework would have to know that a given PropertyDescriptor
represents a file or directory. I would think we need an option on the
PropertyDescriptor builder like ".identifiesFileSystemResource(true)"
then whenever a processor calls context.getProperty(foo).getValue() it
would know if it needs to do the expansion. If expression language is
involved, and a processor calls
context.getProperty(foo).evaluateAttributeExpressions().getValue()
then the framework would need to do the expansion after evaluating the
expressions.
-Bryan
On Fri, Apr 19, 2019 at 3:21 PM Ryan Withers <wi...@gmail.com> wrote:
>
> Hello all,
>
> The basis for the story / jira I've been working on is around properly
> expanding the users home directory when the shortcut ~ is provided as a
> value for any PropertyDescriptor pointing to a file. So I've been working
> through all the processors and implementing this.
>
> I have two questions as I'm nearing completion with this.
>
> 1. The class I'm looking at is TestGeoEnrichIP.java. It has a number of
> test cases that look at the base directory of the project for a maxmind
> database file. An example of this is:
>
> testRunner.setProperty(GeoEnrichIP.GEO_DATABASE_FILE, "./");
>
> I've looked through the project for *.mmdb files as described here:
> https://github.com/maxmind/MaxMind-DB/tree/master/test-data, I didn't find
> any references to files like that. I know I need to spend some additional
> time with this but I wanted to ask the group to get some guidance and
> additional understanding.
>
> 2. My second question is related to file expansion and expression
> language. Specifically given the requirement to take ~/ and map it to
> /home/rwithers/ should this same behavior be enabled if the property
> descriptor has expression language enabled or only if expression language
> is disabled. I've recently run into an example where expression language
> is enabled on a property descriptor and it interfered with the path
> expansion. So naturally I started wondering if path expansion should work
> when expression language is also enabled.
>
> Thanks in advance,
>
> --
> Ryan Withers
> Senior Software Developer / Analyst
>
> http://www.linkedin.com/in/ryanwithers