You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ms1111 (via GitHub)" <gi...@apache.org> on 2024/04/19 02:30:33 UTC
[I] Support for writing Parquet files from the Iceberg Java API without the Hadoop Configuration class [iceberg]
ms1111 opened a new issue, #10180:
URL: https://github.com/apache/iceberg/issues/10180
### Feature Request / Improvement
If the hadoop-common library is not present, trying to write a Parquet file:
```java
DataWriter<Record> dataWriter =
Parquet.writeData(file)
.schema(schema)
.createWriterFunc(GenericParquetWriter::buildWriter)
.overwrite()
.withSpec(partitionSpec)
.build();
```
... will fail with:
```
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at org.apache.iceberg.parquet.Parquet$WriteBuilder.<init>(Parquet.java:164)
at org.apache.iceberg.parquet.Parquet$WriteBuilder.<init>(Parquet.java:143)
at org.apache.iceberg.parquet.Parquet.write(Parquet.java:129)
at org.apache.iceberg.parquet.Parquet$DataWriteBuilder.<init>(Parquet.java:646)
at org.apache.iceberg.parquet.Parquet$DataWriteBuilder.<init>(Parquet.java:637)
at org.apache.iceberg.parquet.Parquet.writeData(Parquet.java:623)
```
In org.apache.iceberg.parquet.Parquet, an empty Configuration is created:
```java
private WriteBuilder(OutputFile file) {
this.file = file;
if (file instanceof HadoopOutputFile) {
this.conf = new Configuration(((HadoopOutputFile) file).getConf());
} else {
this.conf = new Configuration();
}
}
```
ParquetWriter eventually passes this to ParquetIO.file(), which ignores it if the file is not a HadoopOutputFile.
hadoop-common is a heavy dependency with many transitive dependencies, would be nice to avoid it.
Similar to Iceberg Flink issues - #3117 / #4183
### Query engine
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support for writing Parquet files from the Iceberg Java API without the Hadoop Configuration class [iceberg]
Posted by "ysn2233 (via GitHub)" <gi...@apache.org>.
ysn2233 commented on issue #10180:
URL: https://github.com/apache/iceberg/issues/10180#issuecomment-3615910758
Thank you very much! I applied your method in our scenario testing, and it works properly after additionally removing the dependencies on woodstox and commons-logging. Others who encounter similar issues can refer to this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support for writing Parquet files from the Iceberg Java API without the Hadoop Configuration class [iceberg]
Posted by "ms1111 (via GitHub)" <gi...@apache.org>.
ms1111 commented on issue #10180:
URL: https://github.com/apache/iceberg/issues/10180#issuecomment-3615053652
@ysn2233 We were using Flink 1.19 or 1.20 at the time, and Iceberg 1.9.x. This may help:
```java
/**
* Overrides Iceberg's FlinkCatalogFactory to avoid a dependency on HdfsConfiguration,
* a subclass of Configuration.
*
* For Flink to discover this class, an entry is needed in
* META-INF/services/org.apache.flink.table.factories.Factory.
*/
public class NoHDFSFlinkCatalogFactory extends FlinkCatalogFactory {
// This is an old, deprecated entrypoint, but it's the one available in Iceberg's FlinkCatalogFactory at the moment
@Override
@Deprecated
public Catalog createCatalog(String name, Map<String, String> properties) {
return createCatalog(name, properties, new org.apache.hadoop.conf.Configuration());
}
// This is the new entrypoint
@Override
public Catalog createCatalog(Context context) {
return createCatalog(context.getName(), context.getOptions());
}
@Override
public String factoryIdentifier() {
return "iceberg-nohdfs";
}
}
```
and src/main/resources/META-INF/org.apache.flink.table.factories.Factory:
```
# This is needed so StreamTableEnvironment.createCatalog() can find our catalog factory
your.package.NoHDFSFlinkCatalogFactory
```
To use it:
```java
Catalog catalogFactory = new NoHDFSFlinkCatalogFactory().createCatalog(catalogSettings.getCatalogName(), options);
Map<String, String> options = catalogFactory.requiredContext();
options.put("catalog-impl", "org.apache.iceberg.rest.RESTCatalog");
options.put(...);
Configuration catalogConfig = Configuration.fromMap(options);
// We already have the catalog factory, but Flink wants to find it by type.
// There was a streamTableEnvironment.registerCatalog() method but it was deprecated.
// Have to do it indirectly now.
catalogConfig.set(CommonCatalogOptions.CATALOG_TYPE, catalogFactory.factoryIdentifier());
CatalogDescriptor catalogDescriptor = CatalogDescriptor.of("<catalog name>", catalogConfig);
streamTableEnvironment.createCatalog("<catalog name>", catalogDescriptor);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support for writing Parquet files from the Iceberg Java API without the Hadoop Configuration class [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10180:
URL: https://github.com/apache/iceberg/issues/10180#issuecomment-2477675152
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support for writing Parquet files from the Iceberg Java API without the Hadoop Configuration class [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10180: Support for writing Parquet files from the Iceberg Java API without the Hadoop Configuration class
URL: https://github.com/apache/iceberg/issues/10180
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support for writing Parquet files from the Iceberg Java API without the Hadoop Configuration class [iceberg]
Posted by "ms1111 (via GitHub)" <gi...@apache.org>.
ms1111 commented on issue #10180:
URL: https://github.com/apache/iceberg/issues/10180#issuecomment-2790585440
It's possible to use Flink with Iceberg with two Hadoop jars only, hadoop-common and hadoop-mapreduce-client-core, and exclude all their transitive dependencies. The Parquet read/write path depends on a FileInputFormat class from hadoop-mapreduce-client-core, so it's not as simple as removing the dependency on the Configuration class.
```xml
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
</exclusion>
<exclusion>
<groupId>xmlenc</groupId>
<artifactId>xmlenc</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</exclusion>
<exclusion>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</exclusion>
<exclusion>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
</exclusion>
<exclusion>
<groupId>commons-net</groupId>
<artifactId>commons-net</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
</exclusion>
<exclusion>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-util</artifactId>
</exclusion>
<exclusion>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
</exclusion>
<exclusion>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-webapp</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-epoll</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet.jsp</groupId>
<artifactId>jsp-api</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-core</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-json</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-server</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-servlet</artifactId>
</exclusion>
<exclusion>
<groupId>com.github.pjfanning</groupId>
<artifactId>jersey-json</artifactId>
</exclusion>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
<exclusion>
<groupId>net.java.dev.jets3t</groupId>
<artifactId>jets3t</artifactId>
</exclusion>
<exclusion>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
</exclusion>
<exclusion>
<groupId>commons-configuration</groupId>
<artifactId>commons-configuration</artifactId>
</exclusion>
<exclusion>
<groupId>commons-digester</groupId>
<artifactId>commons-digester</artifactId>
</exclusion>
<exclusion>
<groupId>commons-beanutils</groupId>
<artifactId>commons-beanutils</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-reload4j</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.reload4j</groupId>
<artifactId>reload4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.ant</groupId>
<artifactId>ant</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-auth</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.kerby</groupId>
<artifactId>kerb-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk18on</artifactId>
</exclusion>
<exclusion>
<groupId>com.jcraft</groupId>
<artifactId>jsch</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.curator</groupId>
<artifactId>curator-client</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.htrace</groupId>
<artifactId>htrace-core4</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.woodstox</groupId>
<artifactId>stax2-api</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.woodstox</groupId>
<artifactId>woodstox-core</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- Brings in org.apache.hadoop.mapreduce.lib.input.FileInputFormat, used when reading parquet files -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-client</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-common</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs-client</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
<!-- These ones aren't even in the mapreduce-client-core pom, but are coming in somehow -->
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-reload4j</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.inject.extensions</groupId>
<artifactId>guice-servlet</artifactId>
</exclusion>
</exclusions>
</dependency>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support for writing Parquet files from the Iceberg Java API without the Hadoop Configuration class [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10180:
URL: https://github.com/apache/iceberg/issues/10180#issuecomment-2448719170
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support for writing Parquet files from the Iceberg Java API without the Hadoop Configuration class [iceberg]
Posted by "ysn2233 (via GitHub)" <gi...@apache.org>.
ysn2233 commented on issue #10180:
URL: https://github.com/apache/iceberg/issues/10180#issuecomment-3615022837
May I ask which Flink version you are using? Under Flink 1.18, I packaged the minimal Hadoop as you suggested and placed it in the /lib directory of the image, but it still seems to throw a NoClassDefFoundError: org.apache.hadoop.hdfs.HdfsConfiguration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org