You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Reinier Kip (JIRA)" <ji...@apache.org> on 2018/03/12 16:24:00 UTC
[jira] [Updated] (HIVE-18931) Race condition when ensuring a
partition exists often causes AlreadyExistsException for the partition
[ https://issues.apache.org/jira/browse/HIVE-18931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reinier Kip updated HIVE-18931:
-------------------------------
Description:
Hiya! I'm using Apache Beam's HCatalogIO to store data in Hive. As part of HCatOutputFormatWriter#commit(), partitions are registered in FileOutputCommitterContainer#registerPartitions(). Here, it checks for the existence of the partitions, and, if so needed, creates them. For parallel processes, this fails a lot of the time, because, as far as I understand, another process creates the partition in the meantime. This causes an AlreadyExistsException to be created for the partition, which bubbles up to the API consumer.
Relevant logic: [https://github.com/apache/hive/blob/release-1.2.1/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java#L898-L920]. Logic is also present on master.
Am I missing something, and if not, is it an acceptable solution to swallow the AlreadyExistsException when adding the partitions?
was:
Hiya! I'm using Apache Beam's HCatalogIO to store data in Hive. As part of HCatOutputFormatWriter#commit(), partitions are registered in FileOutputCommitterContainer#registerPartitions(). Here, it checks for the existence of the partitions, and, if so needed, creates them. For parallel processes, this fails a lot of the time, because, as far as I understand, another process creates the partition in the meantime. This causes an AlreadyExistsException to be created for the partition.
Relevant logic: [https://github.com/apache/hive/blob/release-1.2.1/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java#L898-L920]. Logic is also present on master.
Am I missing something, and if not, is it an acceptable solution to swallow the AlreadyExistsException when adding the partitions?
> Race condition when ensuring a partition exists often causes AlreadyExistsException for the partition
> -----------------------------------------------------------------------------------------------------
>
> Key: HIVE-18931
> URL: https://issues.apache.org/jira/browse/HIVE-18931
> Project: Hive
> Issue Type: Bug
> Components: Clients, HCatalog
> Affects Versions: 1.2.1
> Reporter: Reinier Kip
> Priority: Major
>
> Hiya! I'm using Apache Beam's HCatalogIO to store data in Hive. As part of HCatOutputFormatWriter#commit(), partitions are registered in FileOutputCommitterContainer#registerPartitions(). Here, it checks for the existence of the partitions, and, if so needed, creates them. For parallel processes, this fails a lot of the time, because, as far as I understand, another process creates the partition in the meantime. This causes an AlreadyExistsException to be created for the partition, which bubbles up to the API consumer.
> Relevant logic: [https://github.com/apache/hive/blob/release-1.2.1/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java#L898-L920]. Logic is also present on master.
> Am I missing something, and if not, is it an acceptable solution to swallow the AlreadyExistsException when adding the partitions?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)