You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Brandon Dahler (Jira)" <ji...@apache.org> on 2024/02/07 03:25:00 UTC

[jira] [Updated] (HUDI-7390) [Regression] HoodieStreamer no longer works without --props being supplied

     [ https://issues.apache.org/jira/browse/HUDI-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brandon Dahler updated HUDI-7390:
---------------------------------
    Description: 
h2. Problem

When attempting to run HoodieStreamer without a props file, specifying all required extra configuration via {{--hoodie-conf}} parameters, the execution fails and an exception is thrown:
{code:java}
24/02/06 22:15:13 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.hudi.exception.HoodieIOException: Cannot read properties from dfs from file file:/private/tmp/hudi-props-repro/src/test/resources/streamer-config/dfs-source.properties
        at org.apache.hudi.common.config.DFSPropertiesConfiguration.addPropsFromFile(DFSPropertiesConfiguration.java:166)
        at org.apache.hudi.common.config.DFSPropertiesConfiguration.<init>(DFSPropertiesConfiguration.java:85)
        at org.apache.hudi.utilities.UtilHelpers.readConfig(UtilHelpers.java:232)
        at org.apache.hudi.utilities.streamer.HoodieStreamer$Config.getProps(HoodieStreamer.java:437)
        at org.apache.hudi.utilities.streamer.StreamSync.getDeducedSchemaProvider(StreamSync.java:656)
        at org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource(StreamSync.java:632)
        at org.apache.hudi.utilities.streamer.StreamSync.fetchFromSourceAndPrepareRecords(StreamSync.java:525)
        at org.apache.hudi.utilities.streamer.StreamSync.readFromSource(StreamSync.java:498)
        at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:404)
        at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:850)
        at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
        at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
        at org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:207)
        at org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:592)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.FileNotFoundException: File file:/private/tmp/hudi-props-repro/src/test/resources/streamer-config/dfs-source.properties does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:779)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1100)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:769)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:160)
        at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:372)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:976)
        at org.apache.hudi.common.config.DFSPropertiesConfiguration.addPropsFromFile(DFSPropertiesConfiguration.java:161)
        ... 25 more {code}
h2. Reproduction Steps

1. Setup clean spark install

 
{code:java}
mkdir /tmp/hudi-props-repro 
cd /tmp/hudi-props-repro 
tar -xvzf ~/spark-3.4.2-bin-hadoop3.tgz{code}
2. Copy the schema file from the docker demo

 
{code:java}
wget https://raw.githubusercontent.com/apache/hudi/release-0.14.1/docker/demo/config/schema.avsc {code}
3. Copy data file from the docker demo

 
{code:java}
mkdir data
cd data
wget https://raw.githubusercontent.com/apache/hudi/release-0.14.1/docker/demo/data/batch_1.json 
cd .. {code}
4. Run HoodieStreamer
{code:java}
spark-3.4.2-bin-hadoop3/bin/spark-submit \
   --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:0.14.1,org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.1 \
   --conf spark.kryoserializer.buffer.max=200m \
   --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
   --class org.apache.hudi.utilities.streamer.HoodieStreamer \
   spark-3.4.2-bin-hadoop3/examples/jars/spark-examples_2.12-3.4.2.jar \
   --table-type COPY_ON_WRITE \
   --source-class org.apache.hudi.utilities.sources.JsonDFSSource \
   --target-base-path /tmp/hudi-props-repro/table \
   --target-table table \
   --hoodie-conf hoodie.datasource.write.recordkey.field=key \
   --hoodie-conf hoodie.datasource.write.partitionpath.field=date \
   --hoodie-conf hoodie.table.recordkey.fields=key \
   --hoodie-conf hoodie.table.partition.fields=date \
   --hoodie-conf hoodie.streamer.schemaprovider.source.schema.file=/tmp/hudi-props-repro/schema.avsc \
   --hoodie-conf hoodie.streamer.schemaprovider.target.schema.file=/tmp/hudi-props-repro/schema.avsc \
   --hoodie-conf hoodie.streamer.source.dfs.root=/tmp/hudi-props-repro/data \
   --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
{code}
h3. Expected Results

Command runs successfully, data is ingested successfully into /{{{}tmp/hudi-decimal-repro/table{}}}, some files exist under {{{}/tmp/hudi-decimal-repro/table/2018/08/31/{}}}.
h3. Actual Results

Command fails with exception, no data is ingsted into the table.  The table's directory is initialized but no commits exist.

Logs of the attempted run are attached as spark.log
h2. Additional Information

This issue does not appear to exist in versions 0.12.2 through 0.14.0 based on my own testing.  It does affect both the 0.14.1 and 1.0.0-beta1 releases.

 

 

  was:
h2. Problem

When attempting to run HoodieStreamer without a props file, specifying all required extra configuration via {{--hoodie-conf}} parameters, the execution fails and an exception is thrown:

 
{code:java}
24/02/06 22:15:13 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.hudi.exception.HoodieIOException: Cannot read properties from dfs from file file:/private/tmp/hudi-props-repro/src/test/resources/streamer-config/dfs-source.properties
        at org.apache.hudi.common.config.DFSPropertiesConfiguration.addPropsFromFile(DFSPropertiesConfiguration.java:166)
        at org.apache.hudi.common.config.DFSPropertiesConfiguration.<init>(DFSPropertiesConfiguration.java:85)
        at org.apache.hudi.utilities.UtilHelpers.readConfig(UtilHelpers.java:232)
        at org.apache.hudi.utilities.streamer.HoodieStreamer$Config.getProps(HoodieStreamer.java:437)
        at org.apache.hudi.utilities.streamer.StreamSync.getDeducedSchemaProvider(StreamSync.java:656)
        at org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource(StreamSync.java:632)
        at org.apache.hudi.utilities.streamer.StreamSync.fetchFromSourceAndPrepareRecords(StreamSync.java:525)
        at org.apache.hudi.utilities.streamer.StreamSync.readFromSource(StreamSync.java:498)
        at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:404)
        at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:850)
        at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
        at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
        at org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:207)
        at org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:592)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.FileNotFoundException: File file:/private/tmp/hudi-props-repro/src/test/resources/streamer-config/dfs-source.properties does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:779)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1100)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:769)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:160)
        at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:372)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:976)
        at org.apache.hudi.common.config.DFSPropertiesConfiguration.addPropsFromFile(DFSPropertiesConfiguration.java:161)
        ... 25 more {code}
h2. Reproduction Steps

1. Setup clean spark install

 
{code:java}
mkdir /tmp/hudi-props-repro 
cd /tmp/hudi-props-repro 
tar -xvzf ~/spark-3.4.2-bin-hadoop3.tgz{code}
2. Copy the schema file from the docker demo

 
{code:java}
wget https://raw.githubusercontent.com/apache/hudi/release-0.14.1/docker/demo/config/schema.avsc {code}
3. Copy data file from the docker demo

 
{code:java}
mkdir data
cd data
wget https://raw.githubusercontent.com/apache/hudi/release-0.14.1/docker/demo/data/batch_1.json 
cd .. {code}
4. Run HoodieStreamer
{code:java}
spark-3.4.2-bin-hadoop3/bin/spark-submit \
   --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:0.14.1,org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.1 \
   --conf spark.kryoserializer.buffer.max=200m \
   --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
   --class org.apache.hudi.utilities.streamer.HoodieStreamer \
   spark-3.4.2-bin-hadoop3/examples/jars/spark-examples_2.12-3.4.2.jar \
   --table-type COPY_ON_WRITE \
   --source-class org.apache.hudi.utilities.sources.JsonDFSSource \
   --target-base-path /tmp/hudi-props-repro/table \
   --target-table table \
   --hoodie-conf hoodie.datasource.write.recordkey.field=key \
   --hoodie-conf hoodie.datasource.write.partitionpath.field=date \
   --hoodie-conf hoodie.table.recordkey.fields=key \
   --hoodie-conf hoodie.table.partition.fields=date \
   --hoodie-conf hoodie.streamer.schemaprovider.source.schema.file=/tmp/hudi-props-repro/schema.avsc \
   --hoodie-conf hoodie.streamer.schemaprovider.target.schema.file=/tmp/hudi-props-repro/schema.avsc \
   --hoodie-conf hoodie.streamer.source.dfs.root=/tmp/hudi-props-repro/data \
   --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
{code}
h3. Expected Results

Command runs successfully, data is ingested successfully into /{{{}tmp/hudi-decimal-repro/table{}}}, some files exist under {{{}/tmp/hudi-decimal-repro/table/2018/08/31/{}}}.
h3. Actual Results

Command fails with exception, no data is ingsted into the table.  The table's directory is initialized but no commits exist.

Logs of the attempted run are attached as spark.log
h2. Additional Information

This issue does not appear to exist in versions 0.12.2 through 0.14.0 based on my own testing.  It does affect both the 0.14.1 and 1.0.0-beta1 releases.

 

 


> [Regression] HoodieStreamer no longer works without --props being supplied
> --------------------------------------------------------------------------
>
>                 Key: HUDI-7390
>                 URL: https://issues.apache.org/jira/browse/HUDI-7390
>             Project: Apache Hudi
>          Issue Type: Bug
>    Affects Versions: 1.0.0-beta1, 0.14.1
>            Reporter: Brandon Dahler
>            Priority: Major
>         Attachments: spark.log
>
>
> h2. Problem
> When attempting to run HoodieStreamer without a props file, specifying all required extra configuration via {{--hoodie-conf}} parameters, the execution fails and an exception is thrown:
> {code:java}
> 24/02/06 22:15:13 INFO SparkContext: Successfully stopped SparkContext
> Exception in thread "main" org.apache.hudi.exception.HoodieIOException: Cannot read properties from dfs from file file:/private/tmp/hudi-props-repro/src/test/resources/streamer-config/dfs-source.properties
>         at org.apache.hudi.common.config.DFSPropertiesConfiguration.addPropsFromFile(DFSPropertiesConfiguration.java:166)
>         at org.apache.hudi.common.config.DFSPropertiesConfiguration.<init>(DFSPropertiesConfiguration.java:85)
>         at org.apache.hudi.utilities.UtilHelpers.readConfig(UtilHelpers.java:232)
>         at org.apache.hudi.utilities.streamer.HoodieStreamer$Config.getProps(HoodieStreamer.java:437)
>         at org.apache.hudi.utilities.streamer.StreamSync.getDeducedSchemaProvider(StreamSync.java:656)
>         at org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource(StreamSync.java:632)
>         at org.apache.hudi.utilities.streamer.StreamSync.fetchFromSourceAndPrepareRecords(StreamSync.java:525)
>         at org.apache.hudi.utilities.streamer.StreamSync.readFromSource(StreamSync.java:498)
>         at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:404)
>         at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:850)
>         at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
>         at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
>         at org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:207)
>         at org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:592)
>         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
>         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.base/java.lang.reflect.Method.invoke(Method.java:568)
>         at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>         at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
>         at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
>         at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
>         at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
>         at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.io.FileNotFoundException: File file:/private/tmp/hudi-props-repro/src/test/resources/streamer-config/dfs-source.properties does not exist
>         at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:779)
>         at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1100)
>         at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:769)
>         at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462)
>         at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:160)
>         at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:372)
>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:976)
>         at org.apache.hudi.common.config.DFSPropertiesConfiguration.addPropsFromFile(DFSPropertiesConfiguration.java:161)
>         ... 25 more {code}
> h2. Reproduction Steps
> 1. Setup clean spark install
>  
> {code:java}
> mkdir /tmp/hudi-props-repro 
> cd /tmp/hudi-props-repro 
> tar -xvzf ~/spark-3.4.2-bin-hadoop3.tgz{code}
> 2. Copy the schema file from the docker demo
>  
> {code:java}
> wget https://raw.githubusercontent.com/apache/hudi/release-0.14.1/docker/demo/config/schema.avsc {code}
> 3. Copy data file from the docker demo
>  
> {code:java}
> mkdir data
> cd data
> wget https://raw.githubusercontent.com/apache/hudi/release-0.14.1/docker/demo/data/batch_1.json 
> cd .. {code}
> 4. Run HoodieStreamer
> {code:java}
> spark-3.4.2-bin-hadoop3/bin/spark-submit \
>    --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:0.14.1,org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.1 \
>    --conf spark.kryoserializer.buffer.max=200m \
>    --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
>    --class org.apache.hudi.utilities.streamer.HoodieStreamer \
>    spark-3.4.2-bin-hadoop3/examples/jars/spark-examples_2.12-3.4.2.jar \
>    --table-type COPY_ON_WRITE \
>    --source-class org.apache.hudi.utilities.sources.JsonDFSSource \
>    --target-base-path /tmp/hudi-props-repro/table \
>    --target-table table \
>    --hoodie-conf hoodie.datasource.write.recordkey.field=key \
>    --hoodie-conf hoodie.datasource.write.partitionpath.field=date \
>    --hoodie-conf hoodie.table.recordkey.fields=key \
>    --hoodie-conf hoodie.table.partition.fields=date \
>    --hoodie-conf hoodie.streamer.schemaprovider.source.schema.file=/tmp/hudi-props-repro/schema.avsc \
>    --hoodie-conf hoodie.streamer.schemaprovider.target.schema.file=/tmp/hudi-props-repro/schema.avsc \
>    --hoodie-conf hoodie.streamer.source.dfs.root=/tmp/hudi-props-repro/data \
>    --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
> {code}
> h3. Expected Results
> Command runs successfully, data is ingested successfully into /{{{}tmp/hudi-decimal-repro/table{}}}, some files exist under {{{}/tmp/hudi-decimal-repro/table/2018/08/31/{}}}.
> h3. Actual Results
> Command fails with exception, no data is ingsted into the table.  The table's directory is initialized but no commits exist.
> Logs of the attempted run are attached as spark.log
> h2. Additional Information
> This issue does not appear to exist in versions 0.12.2 through 0.14.0 based on my own testing.  It does affect both the 0.14.1 and 1.0.0-beta1 releases.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)