You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Clemens Valiente (JIRA)" <ji...@apache.org> on 2019/05/03 16:45:00 UTC
[jira] [Updated] (KUDU-2814) improve the kudu-mapreduce handling of
KuduTableOutputFormat
[ https://issues.apache.org/jira/browse/KUDU-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Clemens Valiente updated KUDU-2814:
-----------------------------------
Description:
[https://github.com/apache/kudu/blob/master/java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableOutputFormat.java#L134-L135]
* MULTITON only allows one instance per thread, a second Unit Test fails (That seems like it would also cause problems when MR's JVM reuse feature is enabled)
* whole point is so that we can have a static `KuduTableOutputFormat.getKuduTable` to have this `getTableFromContext` method in the KuduTableMapReduceUtil
* `getTableFromContext` is used by two sample mapper implementations in kudu-client-tools, the importcsvmapper and importparquetmapper:
** Insert insert = this.table.newInsert();
PartialRow row = insert.getRow();
* the whole point of having an OutputFormat is to not have to worry how your data gets written when you work on the mapper/reducer
* You write to the OutputFormat not get it’s internals and write yourself
To Do: remove those static methods and think about how importparquetmapper and csvmapper can properly use a KuduTableOutputFormat without having to rely on its internals
remove the Multiton if it's not needed any more.
was:
[https://github.com/apache/kudu/blob/master/java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableOutputFormat.java#L134-L135]
* MULTITON only allows one instance per thread, a second Unit Test fails
* whole point is so that we can have a static `KuduTableOutputFormat.getKuduTable` to have this `getTableFromContext` method in the KuduTableMapReduceUtil
* `getTableFromContext` is used by two sample mapper implementations in kudu-client-tools, the importcsvmapper and importparquetmapper:
** Insert insert = this.table.newInsert();
PartialRow row = insert.getRow();
* the whole point of having an OutputFormat is to not have to worry how your data gets written when you work on the mapper/reducer
* You write to the OutputFormat not get it’s internals and write yourself
To Do: remove those static methods and think about how importparquetmapper and csvmapper can properly use a KuduTableOutputFormat without having to rely on its internals
remove the Multiton if it's not needed any more.
> improve the kudu-mapreduce handling of KuduTableOutputFormat
> ------------------------------------------------------------
>
> Key: KUDU-2814
> URL: https://issues.apache.org/jira/browse/KUDU-2814
> Project: Kudu
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.9.0
> Reporter: Clemens Valiente
> Priority: Major
>
> [https://github.com/apache/kudu/blob/master/java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableOutputFormat.java#L134-L135]
>
> * MULTITON only allows one instance per thread, a second Unit Test fails (That seems like it would also cause problems when MR's JVM reuse feature is enabled)
> * whole point is so that we can have a static `KuduTableOutputFormat.getKuduTable` to have this `getTableFromContext` method in the KuduTableMapReduceUtil
> * `getTableFromContext` is used by two sample mapper implementations in kudu-client-tools, the importcsvmapper and importparquetmapper:
> ** Insert insert = this.table.newInsert();
> PartialRow row = insert.getRow();
> * the whole point of having an OutputFormat is to not have to worry how your data gets written when you work on the mapper/reducer
> * You write to the OutputFormat not get it’s internals and write yourself
>
> To Do: remove those static methods and think about how importparquetmapper and csvmapper can properly use a KuduTableOutputFormat without having to rely on its internals
> remove the Multiton if it's not needed any more.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)