You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Satish Mittal <sa...@apache.org> on 2014/03/20 05:47:13 UTC
Review Request 19449: Add new coord EL function to get input partitions
value string
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/
-----------------------------------------------------------
Review request for oozie and Ryota Egashira.
Repository: oozie-git
Description
-------
Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
(minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
In this case, Hive export fails with error:
FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
We need to add a new EL function that returns partition value in the above format.
Diffs
-----
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
core/src/main/resources/oozie-default.xml 34362aa
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
Diff: https://reviews.apache.org/r/19449/diff/
Testing
-------
- Manually verified the new EL function on my local setup.
- Added new test cases for the same.
Thanks,
Satish Mittal
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
> On March 20, 2014, 4:57 p.m., Rohini Palaniswamy wrote:
> > What is the expected string to be returned from the function, when the data-in has current(0) to current(-23)? Can you give some examples where more than one instance of the dataset is involved?
Hi, This function is intended to be used when there is only one data-set instance. E.g. for current(0), a sample output is:
(year='2014',month='03',day='11',hour='09',minute='00')
I haven't added a check for multiple instances scenario. Will do that in the updated patch.
- Satish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review37909
-----------------------------------------------------------
On March 20, 2014, 6 a.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated March 20, 2014, 6 a.m.)
>
>
> Review request for oozie and Ryota Egashira.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 34362aa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review37909
-----------------------------------------------------------
What is the expected string to be returned from the function, when the data-in has current(0) to current(-23)? Can you give some examples where more than one instance of the dataset is involved?
- Rohini Palaniswamy
On March 20, 2014, 6 a.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated March 20, 2014, 6 a.m.)
>
>
> Review request for oozie and Ryota Egashira.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 34362aa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Rohini Palaniswamy <ro...@gmail.com>.
> On March 28, 2014, 6:54 p.m., Rohini Palaniswamy wrote:
> > We should call it dataInPartition as it only works if there is one partition.
> >
> > I like the idea of the generalization, but I can't seem to be able to think of another use case where the current generalization will actually help. Can you list out some usecases so that we are sure that this generalization is actually useful and can be applied elsewhere?
>
> Satish Mittal wrote:
> Hi Rohini, the reason I named it dataInPartitions is so that it is consistent with dataOutPartitions. Please let me know if you still think we should rename it.
>
> Generalizing the separators and equals string to create partition value will prevent unnecessarily introduce new EL functions just to create a partition value which only differs in a particular aspect, e.g. a) suppose the desired partition value for a particular feature in Pig/Hive/New action type is ';'/'AND/OR' separated instead of current ','. Or b) suppose the expected equality string is '==' instead of current '='. Or c) It could be a combination of a) and b).
> We may not be having such a scenario immediately, but it is not hard to imagine one in future, given the fact that partition value/filter formats are not standardized even today across Pig/Hive. In such a case, we don't want to have an overloaded dataInPartition EL for each new format and it is good to be future-proof.
Since you are throwing error on more than one partition, it has to be dataInPartition. If another EL has to be added to get all partitions, this should not clash with it.
My question is, is it really future proof and has it considered all possibilities? Current EL can only address (col1="value1",col2="value2"), (col1="value1" AND col2="value2") and (col1=="value1" AND col2=="value2"). It is not easy to change the EL Functions or overload it (according to Shwetha in OOZIE-1757) and a new one has to be created if this one does not work. For eg: What about ' vs "". What if the argument expected was something like ("minute=00,month=03,year=2014,hour=09") which is used by hcat (OOZIE-1757) for output currently? This function cannot be applied there. Even if future possibilities cannot be thought, look at all existing usages in pig and hive and enhance this function.
- Rohini
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review38930
-----------------------------------------------------------
On March 28, 2014, 12:38 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated March 28, 2014, 12:38 p.m.)
>
>
> Review request for oozie and Ryota Egashira.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 34362aa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 44a3d54
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
> On March 28, 2014, 6:54 p.m., Rohini Palaniswamy wrote:
> > We should call it dataInPartition as it only works if there is one partition.
> >
> > I like the idea of the generalization, but I can't seem to be able to think of another use case where the current generalization will actually help. Can you list out some usecases so that we are sure that this generalization is actually useful and can be applied elsewhere?
>
> Satish Mittal wrote:
> Hi Rohini, the reason I named it dataInPartitions is so that it is consistent with dataOutPartitions. Please let me know if you still think we should rename it.
>
> Generalizing the separators and equals string to create partition value will prevent unnecessarily introduce new EL functions just to create a partition value which only differs in a particular aspect, e.g. a) suppose the desired partition value for a particular feature in Pig/Hive/New action type is ';'/'AND/OR' separated instead of current ','. Or b) suppose the expected equality string is '==' instead of current '='. Or c) It could be a combination of a) and b).
> We may not be having such a scenario immediately, but it is not hard to imagine one in future, given the fact that partition value/filter formats are not standardized even today across Pig/Hive. In such a case, we don't want to have an overloaded dataInPartition EL for each new format and it is good to be future-proof.
>
> Rohini Palaniswamy wrote:
> Since you are throwing error on more than one partition, it has to be dataInPartition. If another EL has to be added to get all partitions, this should not clash with it.
>
> My question is, is it really future proof and has it considered all possibilities? Current EL can only address (col1="value1",col2="value2"), (col1="value1" AND col2="value2") and (col1=="value1" AND col2=="value2"). It is not easy to change the EL Functions or overload it (according to Shwetha in OOZIE-1757) and a new one has to be created if this one does not work. For eg: What about ' vs "". What if the argument expected was something like ("minute=00,month=03,year=2014,hour=09") which is used by hcat (OOZIE-1757) for output currently? This function cannot be applied there. Even if future possibilities cannot be thought, look at all existing usages in pig and hive and enhance this function.
>
> Rohini Palaniswamy wrote:
> Actually you can keep it dataInPartitions, by adding a separator for different partitions/instances so that the same EL will handle one or more partition/instance.
Discussed with Shwetha on OOZIE-1757. Decided to go ahead with type (instead of separator) in both jiras. Also named the method as dataInPartitions to be consistent with dataOutPartitions (both handle single partition).
- Satish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review38930
-----------------------------------------------------------
On March 28, 2014, 12:38 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated March 28, 2014, 12:38 p.m.)
>
>
> Review request for oozie and Ryota Egashira.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 34362aa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 44a3d54
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
> On March 28, 2014, 6:54 p.m., Rohini Palaniswamy wrote:
> > We should call it dataInPartition as it only works if there is one partition.
> >
> > I like the idea of the generalization, but I can't seem to be able to think of another use case where the current generalization will actually help. Can you list out some usecases so that we are sure that this generalization is actually useful and can be applied elsewhere?
Hi Rohini, the reason I named it dataInPartitions is so that it is consistent with dataOutPartitions. Please let me know if you still think we should rename it.
Generalizing the separators and equals string to create partition value will prevent unnecessarily introduce new EL functions just to create a partition value which only differs in a particular aspect, e.g. a) suppose the desired partition value for a particular feature in Pig/Hive/New action type is ';'/'AND/OR' separated instead of current ','. Or b) suppose the expected equality string is '==' instead of current '='. Or c) It could be a combination of a) and b).
We may not be having such a scenario immediately, but it is not hard to imagine one in future, given the fact that partition value/filter formats are not standardized even today across Pig/Hive. In such a case, we don't want to have an overloaded dataInPartition EL for each new format and it is good to be future-proof.
- Satish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review38930
-----------------------------------------------------------
On March 28, 2014, 12:38 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated March 28, 2014, 12:38 p.m.)
>
>
> Review request for oozie and Ryota Egashira.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 34362aa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 44a3d54
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Rohini Palaniswamy <ro...@gmail.com>.
> On March 28, 2014, 6:54 p.m., Rohini Palaniswamy wrote:
> > We should call it dataInPartition as it only works if there is one partition.
> >
> > I like the idea of the generalization, but I can't seem to be able to think of another use case where the current generalization will actually help. Can you list out some usecases so that we are sure that this generalization is actually useful and can be applied elsewhere?
>
> Satish Mittal wrote:
> Hi Rohini, the reason I named it dataInPartitions is so that it is consistent with dataOutPartitions. Please let me know if you still think we should rename it.
>
> Generalizing the separators and equals string to create partition value will prevent unnecessarily introduce new EL functions just to create a partition value which only differs in a particular aspect, e.g. a) suppose the desired partition value for a particular feature in Pig/Hive/New action type is ';'/'AND/OR' separated instead of current ','. Or b) suppose the expected equality string is '==' instead of current '='. Or c) It could be a combination of a) and b).
> We may not be having such a scenario immediately, but it is not hard to imagine one in future, given the fact that partition value/filter formats are not standardized even today across Pig/Hive. In such a case, we don't want to have an overloaded dataInPartition EL for each new format and it is good to be future-proof.
>
> Rohini Palaniswamy wrote:
> Since you are throwing error on more than one partition, it has to be dataInPartition. If another EL has to be added to get all partitions, this should not clash with it.
>
> My question is, is it really future proof and has it considered all possibilities? Current EL can only address (col1="value1",col2="value2"), (col1="value1" AND col2="value2") and (col1=="value1" AND col2=="value2"). It is not easy to change the EL Functions or overload it (according to Shwetha in OOZIE-1757) and a new one has to be created if this one does not work. For eg: What about ' vs "". What if the argument expected was something like ("minute=00,month=03,year=2014,hour=09") which is used by hcat (OOZIE-1757) for output currently? This function cannot be applied there. Even if future possibilities cannot be thought, look at all existing usages in pig and hive and enhance this function.
Actually you can keep it dataInPartitions, by adding a separator for different partitions/instances so that the same EL will handle one or more partition/instance.
- Rohini
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review38930
-----------------------------------------------------------
On March 28, 2014, 12:38 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated March 28, 2014, 12:38 p.m.)
>
>
> Review request for oozie and Ryota Egashira.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 34362aa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 44a3d54
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review38930
-----------------------------------------------------------
We should call it dataInPartition as it only works if there is one partition.
I like the idea of the generalization, but I can't seem to be able to think of another use case where the current generalization will actually help. Can you list out some usecases so that we are sure that this generalization is actually useful and can be applied elsewhere?
- Rohini Palaniswamy
On March 28, 2014, 12:38 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated March 28, 2014, 12:38 p.m.)
>
>
> Review request for oozie and Ryota Egashira.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 34362aa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 44a3d54
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
> On June 17, 2014, 6:37 a.m., shwethags wrote:
> > core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, line 303
> > <https://reviews.apache.org/r/19449/diff/5/?file=610674#file610674line303>
> >
> > There should be validation on type in the beginning of the function
Added validation for any other type than hive-export.
> On June 17, 2014, 6:37 a.m., shwethags wrote:
> > core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java, line 292
> > <https://reviews.apache.org/r/19449/diff/5/?file=610676#file610676line292>
> >
> > why do you need assertEquals here?
-ve test case; was written similar to other tests in this class. Have removed assertEquals now.
> On June 17, 2014, 6:37 a.m., shwethags wrote:
> > core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java, line 301
> > <https://reviews.apache.org/r/19449/diff/5/?file=610676#file610676line301>
> >
> > why do you need assertEquals here?
-ve test case; was written similar to other tests in this class. Have removed assertEquals now.
> On June 17, 2014, 6:37 a.m., shwethags wrote:
> > core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java, line 305
> > <https://reviews.apache.org/r/19449/diff/5/?file=610676#file610676line305>
> >
> > Can you add another test with different type (not hive-export)
Added a -ve test scenario in testDataInPartitions() to execute EL with a different type than hive-export.
> On June 17, 2014, 6:37 a.m., shwethags wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 2918
> > <https://reviews.apache.org/r/19449/diff/5/?file=610677#file610677line2918>
> >
> > Uses dataInPartitions for output
Have renamed various property names in example from IN/OUT prefixes to EXPORT/IMPORT prefixes for better clarity.
- Satish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review45914
-----------------------------------------------------------
On June 17, 2014, 11:01 a.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 17, 2014, 11:01 a.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml b944d3d
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 961377e
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review45914
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment81040>
Can you specify dataInName in the exceptions. When there are multiple data ins for a coord, its difficult to figure out for which data in the error is for
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment81039>
There should be validation on type in the beginning of the function
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment81041>
why do you need assertEquals here?
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment81042>
why do you need assertEquals here?
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment81043>
Can you add another test with different type (not hive-export)
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/19449/#comment81044>
Uses dataInPartitions for output
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/19449/#comment81046>
extra)
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java
<https://reviews.apache.org/r/19449/#comment81045>
throw unhandled type
- shwethags
On June 16, 2014, 1:09 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 16, 2014, 1:09 p.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml c9d9591
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review46036
-----------------------------------------------------------
Ship it!
Ship It!
- shwethags
On June 17, 2014, 11:01 a.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 17, 2014, 11:01 a.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml b944d3d
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 961377e
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/
-----------------------------------------------------------
(Updated June 17, 2014, 11:01 a.m.)
Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
Repository: oozie-git
Description
-------
Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
(minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
In this case, Hive export fails with error:
FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
We need to add a new EL function that returns partition value in the above format.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
core/src/main/resources/oozie-default.xml b944d3d
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 961377e
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
Diff: https://reviews.apache.org/r/19449/diff/
Testing
-------
- Manually verified the new EL function on my local setup.
- Added new test cases for the same.
Thanks,
Satish Mittal
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/
-----------------------------------------------------------
(Updated June 16, 2014, 1:09 p.m.)
Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
Changes
-------
Updated patch after review comments.
Repository: oozie-git
Description
-------
Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
(minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
In this case, Hive export fails with error:
FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
We need to add a new EL function that returns partition value in the above format.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
core/src/main/resources/oozie-default.xml c9d9591
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
Diff: https://reviews.apache.org/r/19449/diff/
Testing
-------
- Manually verified the new EL function on my local setup.
- Added new test cases for the same.
Thanks,
Satish Mittal
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review45732
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment80701>
pass e as argument. Not in message (so that stactrace is available)
- shwethags
On June 13, 2014, 5:40 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 13, 2014, 5:40 p.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 63a91fa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
> On June 16, 2014, 2:02 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, lines 293-298
> > <https://reviews.apache.org/r/19449/diff/4/?file=609416#file609416line293>
> >
> > Can you put this code in if (type equals hive-export) block?
Done.
> On June 16, 2014, 2:02 a.m., Rohini Palaniswamy wrote:
> > core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java, line 301
> > <https://reviews.apache.org/r/19449/diff/4/?file=609418#file609418line301>
> >
> > Aren't there 2 parameters here? Why would it fail?
The expected string has 2 parameters, but the actual one (expr = "${coord:dataInPartitions('ABCD')}") has only one param. You have probably missed that.
> On June 16, 2014, 2:02 a.m., Rohini Palaniswamy wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 2845
> > <https://reviews.apache.org/r/19449/diff/4/?file=609419#file609419line2845>
> >
> > This needs to be changed to type
Done
> On June 16, 2014, 2:02 a.m., Rohini Palaniswamy wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 2848
> > <https://reviews.apache.org/r/19449/diff/4/?file=609419#file609419line2848>
> >
> > Can you say that currently the only type supported is hive-export and type hive-export can be used in Hive action to create ....
> >
> > Also mention that there can be only one partition instance
Done
> On June 16, 2014, 2:02 a.m., Rohini Palaniswamy wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 3007
> > <https://reviews.apache.org/r/19449/diff/4/?file=609419#file609419line3007>
> >
> > We should make the EL function return the value without the enclosing braces( ) as most of the others do not. i.e
> >
> > export table ${sourceDatabase}.${sourceTable} partition (${sourcePartition}) to '${sourceStagingDir}';
Done
- Satish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review45715
-----------------------------------------------------------
On June 16, 2014, 1:09 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 16, 2014, 1:09 p.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml c9d9591
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review45715
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment80686>
Can you put this code in if (type equals hive-export) block?
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment80687>
Aren't there 2 parameters here? Why would it fail?
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/19449/#comment80688>
This needs to be changed to type
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/19449/#comment80689>
Can you say that currently the only type supported is hive-export and type hive-export can be used in Hive action to create ....
Also mention that there can be only one partition instance
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/19449/#comment80691>
We should make the EL function return the value without the enclosing braces( ) as most of the others do not. i.e
export table ${sourceDatabase}.${sourceTable} partition (${sourcePartition}) to '${sourceStagingDir}';
- Rohini Palaniswamy
On June 13, 2014, 5:40 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 13, 2014, 5:40 p.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 63a91fa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/
-----------------------------------------------------------
(Updated June 13, 2014, 5:40 p.m.)
Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
Changes
-------
Attaching revised patch with dataInPartitions() and type as hive-export.
Repository: oozie-git
Description
-------
Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
(minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
In this case, Hive export fails with error:
FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
We need to add a new EL function that returns partition value in the above format.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
core/src/main/resources/oozie-default.xml 63a91fa
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
Diff: https://reviews.apache.org/r/19449/diff/
Testing
-------
- Manually verified the new EL function on my local setup.
- Added new test cases for the same.
Thanks,
Satish Mittal
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/
-----------------------------------------------------------
(Updated April 8, 2014, 6:38 p.m.)
Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
Repository: oozie-git
Description
-------
Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
(minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
In this case, Hive export fails with error:
FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
We need to add a new EL function that returns partition value in the above format.
Diffs
-----
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
core/src/main/resources/oozie-default.xml 0198230
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java a689cb1
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 44a3d54
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
Diff: https://reviews.apache.org/r/19449/diff/
Testing
-------
- Manually verified the new EL function on my local setup.
- Added new test cases for the same.
Thanks,
Satish Mittal
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/
-----------------------------------------------------------
(Updated April 3, 2014, 7:02 p.m.)
Review request for oozie and Ryota Egashira.
Repository: oozie-git
Description
-------
Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
(minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
In this case, Hive export fails with error:
FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
We need to add a new EL function that returns partition value in the above format.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
core/src/main/resources/oozie-default.xml 0198230
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java a689cb1
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 44a3d54
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
Diff: https://reviews.apache.org/r/19449/diff/
Testing
-------
- Manually verified the new EL function on my local setup.
- Added new test cases for the same.
Thanks,
Satish Mittal
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/
-----------------------------------------------------------
(Updated March 28, 2014, 12:38 p.m.)
Review request for oozie and Ryota Egashira.
Changes
-------
Attaching updated patch that incorporates the review comments.
Repository: oozie-git
Description
-------
Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
(minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
In this case, Hive export fails with error:
FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
We need to add a new EL function that returns partition value in the above format.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
core/src/main/resources/oozie-default.xml 34362aa
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 44a3d54
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
Diff: https://reviews.apache.org/r/19449/diff/
Testing
-------
- Manually verified the new EL function on my local setup.
- Added new test cases for the same.
Thanks,
Satish Mittal
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
> On March 24, 2014, 7:11 p.m., Ryota Egashira wrote:
> > in addition to rohini's comment.
> > how about making it more generic EL funct, where user can specify separator(e.,g ',' 'AND' 'OR') and equal symbol (e.g, '=', '==', etc) in argument?, since there could be new requirement in future more than "," "=" pair for hive export/import.
> > current dataInPartitionFilter somehow trying to hide the detail, only asking user to specify action type to make it easy. but also good to provide new configurable EL funct for those who know what they are doing.
Hi Ryota,
I like your idea of generalizing various aspects of new EL function (separator, equal symbol) so that it can handle newer use-cases to generate single partition value in future, without the need to add a new function for each of them. Will do that.
- Satish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review38343
-----------------------------------------------------------
On March 20, 2014, 6 a.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated March 20, 2014, 6 a.m.)
>
>
> Review request for oozie and Ryota Egashira.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 34362aa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Ryota Egashira <eg...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review38343
-----------------------------------------------------------
in addition to rohini's comment.
how about making it more generic EL funct, where user can specify separator(e.,g ',' 'AND' 'OR') and equal symbol (e.g, '=', '==', etc) in argument?, since there could be new requirement in future more than "," "=" pair for hive export/import.
current dataInPartitionFilter somehow trying to hide the detail, only asking user to specify action type to make it easy. but also good to provide new configurable EL funct for those who know what they are doing.
- Ryota Egashira
On March 20, 2014, 6 a.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated March 20, 2014, 6 a.m.)
>
>
> Review request for oozie and Ryota Egashira.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 34362aa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/
-----------------------------------------------------------
(Updated March 20, 2014, 6 a.m.)
Review request for oozie and Ryota Egashira.
Summary (updated)
-----------------
OOZIE-1741: Add new coord EL function to get input partitions value string
Repository: oozie-git
Description
-------
Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
(minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
In this case, Hive export fails with error:
FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
We need to add a new EL function that returns partition value in the above format.
Diffs
-----
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
core/src/main/resources/oozie-default.xml 34362aa
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f46f1ec
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
Diff: https://reviews.apache.org/r/19449/diff/
Testing
-------
- Manually verified the new EL function on my local setup.
- Added new test cases for the same.
Thanks,
Satish Mittal