You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/30 07:48:22 UTC
[GitHub] [hudi] yyar opened a new issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
yyar opened a new issue #4162:
URL: https://github.com/apache/hudi/issues/4162
**Describe the problem you faced**
I could check only the last appended data, not the first writen data when I execute hudi schema evolution example code in [this URL](https://hudi.apache.org/docs/schema_evolution/).
**To Reproduce**
Steps to reproduce the behavior:
1. Run hudi schema evolution example using spark-shell or sbt test command or something.
2. Check result data.
**Expected behavior**
Row count of result table should be 5. All saved data should remain well.
**Environment Description**
* Hudi version : 0.8.0, 0.9.0
* Spark version : 3.0.1
* Hive version : -
* Hadoop version : 2.7.4
* Storage (HDFS/S3/GCS..) : local storage
* Running on Docker? (yes/no) : no
* Local Environment: macOS Big Sur 11.6.1
**Additional context**
This problem does not happen when I executed in HDFS. When I execute using IntelliJ test case, it`s OK, but when I execute using spark-shell or sbt test to save local storage, it happens.
**Execution Log**
```
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.2.0
/_/
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 11.0.12)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apache.hudi.QuickstartUtils._
import org.apache.hudi.QuickstartUtils._
scala> import scala.collection.JavaConversions._
import scala.collection.JavaConversions._
scala> import org.apache.spark.sql.SaveMode._
import org.apache.spark.sql.SaveMode._
scala> import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceReadOptions._
scala> import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.DataSourceWriteOptions._
scala> import org.apache.hudi.config.HoodieWriteConfig._
import org.apache.hudi.config.HoodieWriteConfig._
scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._
scala> import org.apache.spark.sql.Row
import org.apache.spark.sql.Row
scala> val tableName = "hudi_trips_mor"
tableName: String = hudi_trips_mor
scala> val basePath = "file:///tmp/hudi_trips_mor"
basePath: String = file:///tmp/hudi_trips_mor
scala> val schema = StructType( Array(
| StructField("rowId", StringType,true),
| StructField("partitionId", StringType,true),
| StructField("preComb", LongType,true),
| StructField("name", StringType,true),
| StructField("versionId", StringType,true),
| StructField("intToLong", IntegerType,true)
| ))
schema: org.apache.spark.sql.types.StructType = StructType(StructField(rowId,StringType,true), StructField(partitionId,StringType,true), StructField(preComb,LongType,true), StructField(name,StringType,true), StructField(versionId,StringType,true), StructField(intToLong,IntegerType,true))
scala> val data1 = Seq(Row("row_1", "part_0", 0L, "bob", "v_0", 0),
| Row("row_2", "part_0", 0L, "john", "v_0", 0),
| Row("row_3", "part_0", 0L, "tom", "v_0", 0))
data1: Seq[org.apache.spark.sql.Row] = List([row_1,part_0,0,bob,v_0,0], [row_2,part_0,0,john,v_0,0], [row_3,part_0,0,tom,v_0,0])
scala> var dfFromData1 = spark.createDataFrame(data1, schema)
warning: one deprecation (since 2.12.0); for details, enable `:setting -deprecation' or `:replay -deprecation'
dfFromData1: org.apache.spark.sql.DataFrame = [rowId: string, partitionId: string ... 4 more fields]
scala> dfFromData1.write.format("hudi").
| options(getQuickstartWriteConfigs).
| option(PRECOMBINE_FIELD.key(), "preComb").
| option(RECORDKEY_FIELD.key(), "rowId").
| option(PARTITIONPATH_FIELD.key(), "partitionId").
| option("hoodie.index.type","SIMPLE").
| option(TBL_NAME.key(), tableName).
| option(TABLE_TYPE.key(), "MERGE_ON_READ").
| mode(Overwrite).
| save(basePath)
21/11/30 16:38:23 WARN HoodieSparkSqlWriter$: hoodie table at file:/tmp/hudi_trips_mor already exists. Deleting existing data & overwriting with new data.
scala> var tripsSnapshotDF1 = spark.read.format("hudi").load(basePath + "/*/*")
tripsSnapshotDF1: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string, _hoodie_commit_seqno: string ... 9 more fields]
scala> tripsSnapshotDF1.show(false)
+-------------------+--------------------+------------------+----------------------+----------------------------------------------------------------------+-----+-----------+-------+----+---------+---------+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |rowId|partitionId|preComb|name|versionId|intToLong|
+-------------------+--------------------+------------------+----------------------+----------------------------------------------------------------------+-----+-----------+-------+----+---------+---------+
|20211130163823 |20211130163823_0_1 |row_3 |part_0 |341703d8-672f-4651-8e96-59f4d98247ee-0_0-14-211_20211130163823.parquet|row_3|part_0 |0 |tom |v_0 |0 |
|20211130163823 |20211130163823_0_2 |row_2 |part_0 |341703d8-672f-4651-8e96-59f4d98247ee-0_0-14-211_20211130163823.parquet|row_2|part_0 |0 |john|v_0 |0 |
|20211130163823 |20211130163823_0_3 |row_1 |part_0 |341703d8-672f-4651-8e96-59f4d98247ee-0_0-14-211_20211130163823.parquet|row_1|part_0 |0 |bob |v_0 |0 |
+-------------------+--------------------+------------------+----------------------+----------------------------------------------------------------------+-----+-----------+-------+----+---------+---------+
scala> val newSchema = StructType( Array(
| StructField("rowId", StringType,true),
| StructField("partitionId", StringType,true),
| StructField("preComb", LongType,true),
| StructField("name", StringType,true),
| StructField("versionId", StringType,true),
| StructField("intToLong", LongType,true),
| StructField("newField", StringType,true)
| ))
newSchema: org.apache.spark.sql.types.StructType = StructType(StructField(rowId,StringType,true), StructField(partitionId,StringType,true), StructField(preComb,LongType,true), StructField(name,StringType,true), StructField(versionId,StringType,true), StructField(intToLong,LongType,true), StructField(newField,StringType,true))
scala> val data2 = Seq(Row("row_2", "part_0", 5L, "john", "v_3", 3L, "newField_1"),
| Row("row_5", "part_0", 5L, "maroon", "v_2", 2L, "newField_1"),
| Row("row_9", "part_0", 5L, "michael", "v_2", 2L, "newField_1"))
data2: Seq[org.apache.spark.sql.Row] = List([row_2,part_0,5,john,v_3,3,newField_1], [row_5,part_0,5,maroon,v_2,2,newField_1], [row_9,part_0,5,michael,v_2,2,newField_1])
scala> var dfFromData2 = spark.createDataFrame(data2, newSchema)
warning: one deprecation (since 2.12.0); for details, enable `:setting -deprecation' or `:replay -deprecation'
dfFromData2: org.apache.spark.sql.DataFrame = [rowId: string, partitionId: string ... 5 more fields]
scala> dfFromData2.write.format("hudi").
| options(getQuickstartWriteConfigs).
| option(PRECOMBINE_FIELD.key(), "preComb").
| option(RECORDKEY_FIELD.key(), "rowId").
| option(PARTITIONPATH_FIELD.key(), "partitionId").
| option("hoodie.index.type","SIMPLE").
| option(TBL_NAME.key(), tableName).
| option(TABLE_TYPE.key(), "MERGE_ON_READ").
| mode(Append).
| save(basePath)
scala> var tripsSnapshotDF2 = spark.read.format("hudi").load(basePath + "/*/*")
tripsSnapshotDF2: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string, _hoodie_commit_seqno: string ... 10 more fields]
scala> tripsSnapshotDF2.show(false)
+-------------------+--------------------+------------------+----------------------+----------------------------------------------------------------------+-----+-----------+-------+-------+---------+---------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |rowId|partitionId|preComb|name |versionId|intToLong|newField |
+-------------------+--------------------+------------------+----------------------+----------------------------------------------------------------------+-----+-----------+-------+-------+---------+---------+----------+
|20211130163841 |20211130163841_0_4 |row_2 |part_0 |341703d8-672f-4651-8e96-59f4d98247ee-0_0-37-427_20211130163841.parquet|row_2|part_0 |5 |john |v_3 |3 |newField_1|
|20211130163841 |20211130163841_0_5 |row_5 |part_0 |341703d8-672f-4651-8e96-59f4d98247ee-0_0-37-427_20211130163841.parquet|row_5|part_0 |5 |maroon |v_2 |2 |newField_1|
|20211130163841 |20211130163841_0_6 |row_9 |part_0 |341703d8-672f-4651-8e96-59f4d98247ee-0_0-37-427_20211130163841.parquet|row_9|part_0 |5 |michael|v_2 |2 |newField_1|
+-------------------+--------------------+------------------+----------------------+----------------------------------------------------------------------+-----+-----------+-------+-------+---------+---------+----------+
scala> tripsSnapshotDF2.count()
res4: Long = 3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan edited a comment on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
xushiyan edited a comment on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-991838841
> @xushiyan I found a cause of this problem when I'm using sbt. [The embedded-kafka library](https://github.com/embeddedkafka/embedded-kafka) for testing kafka streaming had been caused this problem. I removed this line in my test dependency and my test had been changed to be ok.
>
> ```scala
> "io.github.embeddedkafka" %% "embedded-kafka" % "2.4.1" % "test",
> ```
>
> I think dependency libraries in embedded-kafka can cause this problem and same library may been used in spark 3.2. Can you please explain this problem occurs when using spark 3.2 at code level?
>
> (added) After excluding avro library in embedded-kafka, my unit test had been ok which contains embedded-kafka library. Could a specific version of avro library cause this problem?
@yyar actually i don't quite understand your scenario. Are you saying you have a sbt project with hudi dependency and there is dependency conflict? you can always exclude transitive dependency. you can also activate `spark-shade-unbundle-avro` maven profile to build a hudi jar without spark-avro. is this the avro library you're talking about?
I see the original issue with regards to spark versions has been resolved. Can we close this? The further question is about embedded kafka lib, does not look like hudi-related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yyar commented on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
yyar commented on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-993036667
@xushiyan
I initially posted a question that both the spark version and sbt happen. Looking at my sbt problem, I assume that avro version (probably upper than 1.8.2) will have a problem in hudi. Last time I asked code level question, I wanted to know why this happens when I reference that avro library.
If it is not appropriate to ask a question here, you may close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-986327002
@yyar the spark shell environment shows it's spark 3.2, probably causing different behavior from your intellij environment where spark 3.0.x was used. in 0.9 and 0.8, spark 3.2 wasn't supported. can you try fixing the spark shell version and see if it persists?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-991838374
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-997117091
It's fine. Good to get it clarified. will close out the issue since you get past the original issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan edited a comment on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
xushiyan edited a comment on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-991838841
> @xushiyan I found a cause of this problem when I'm using sbt. [The embedded-kafka library](https://github.com/embeddedkafka/embedded-kafka) for testing kafka streaming had been caused this problem. I removed this line in my test dependency and my test had been changed to be ok.
>
> ```scala
> "io.github.embeddedkafka" %% "embedded-kafka" % "2.4.1" % "test",
> ```
>
> I think dependency libraries in embedded-kafka can cause this problem and same library may been used in spark 3.2. Can you please explain this problem occurs when using spark 3.2 at code level?
>
> (added) After excluding avro library in embedded-kafka, my unit test had been ok which contains embedded-kafka library. Could a specific version of avro library cause this problem?
@yyar actually i don't quite understand your scenario. Are you saying you have a sbt project with hudi dependency and there is dependency conflict? you can always exclude transitive dependency. you can also activate `spark-shade-unbundle-avro` maven profile to build a hudi jar without spark-avro. is this the avro library you're talking about?
I see the original issue with regards to spark versions has been resolved. Can we close this? The further question is about embedded kafka lib, does not look like hudi-related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-991831448
@xushiyan : with 0.10.0, we have made default compile version for spark3 as 3.2.0 right? is this going to cause any serious issue then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-991846329
gotcha, my bad. mistook 3.1.2 for 3.2. thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yyar commented on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
yyar commented on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-986498066
@xushiyan You're right. When I ran spark-shell in 3.0.1, this behavior doesn't appear and it happens in 3.2.0. One strange point is when I ran using sbt. I will check running in sbt and reply again. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-991831448
@xushiyan : with 0.10.0, we have made default compile version for spark3 as 3.2.0 right? is this going to cause any serious issue then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-991838374
> @xushiyan : with 0.10.0, we have made default compile version for spark3 as 3.2.0 right? is this going to cause any serious issue then?
in 0.10.0 it's built against spark 3.1.2, not 3.2. Don't think this would cause any serious issue. People can still build with maven profile spark3.0.x for earlier version they use, whenever they decide to upgrade the jar. The release notes are clear about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-991838841
> @xushiyan I found a cause of this problem when I'm using sbt. [The embedded-kafka library](https://github.com/embeddedkafka/embedded-kafka) for testing kafka streaming had been caused this problem. I removed this line in my test dependency and my test had been changed to be ok.
>
> ```scala
> "io.github.embeddedkafka" %% "embedded-kafka" % "2.4.1" % "test",
> ```
>
> I think dependency libraries in embedded-kafka can cause this problem and same library may been used in spark 3.2. Can you please explain this problem occurs when using spark 3.2 at code level?
>
> (added) After excluding avro library in embedded-kafka, my unit test had been ok which contains embedded-kafka library. Could a specific version of avro library cause this problem?
@yyar actually i don't quite understand your scenario. Are you saying you have a sbt project with hudi dependency and there is dependency conflict? you can always exclude transitive dependency. you can also activate `spark-shade-unbundle-avro` maven profile to build a hudi jar without spark-avro. is this the avro library you're talking about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yyar commented on issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
yyar commented on issue #4162:
URL: https://github.com/apache/hudi/issues/4162#issuecomment-989664124
@xushiyan
I found a cause of this problem when I'm using sbt. [The embedded-kafka library](https://github.com/embeddedkafka/embedded-kafka) for testing kafka streaming had been caused this problem. I removed this line in my test dependency and my test had been changed to be ok.
```sbt
"io.github.embeddedkafka" %% "embedded-kafka" % "2.4.1" % "test",
```
I think dependency libraries in embedded-kafka can cause this problem and same library may been used in spark 3.2. Can you please explain this problem occurs when using spark 3.2 at code level?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #4162: [SUPPORT] First written data disappears when I append second data for schema evolution in example code.
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4162:
URL: https://github.com/apache/hudi/issues/4162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org